ENHANCED TESTING AUTOMATION PROOF OF CONCEPT FOR FREENEST, Exercises of Computer Science

Download ENHANCED TESTING AUTOMATION PROOF OF CONCEPT FOR FREENEST and more Exercises Computer Science in PDF only on Docsity!

ENHANCED TESTING AUTOMATION PROOF

OF CONCEPT FOR FREENEST

Niko Korhonen

Bachelor's thesis

November 2012

Software Engineering

The School of Technology

OPINNÄYTETYÖN KUVAILULEHTI Tekijä(t) KORHONEN, Niko Julkaisun laji Opinnäytetyö Päivämäärä 20.11. Sivumäärä 75 Julkaisun kieli Englanti Verkkojulkaisulupa myönnetty ( X ) Työn nimi PARANNELLUN TESTAUSAUTOMAATION ESIMERKKI FREENESTILLE Koulutusohjelma Ohjelmistotekniikka Työn ohjaaja(t) PIETIKÄINEN, Kalevi Toimeksiantaja(t) RINTAMÄKI, Marko Tiivistelmä Työssä oli tarkoituksena luoda toimiva testausautomaation esimerkki käyttämällä TestLink, Robot Framework ja Git -työkaluja. Testausautomaatin on tarkoitus ajaa TestLinkistä valitut testit automaattisesti pilviympäristössä käyttämällä Robot Frameworkia ja raportoida tulokset takaisin TestLinkille. Testien uusimmat versiot haetaan Git-repositoriosta ajon aikana. Työn teoriaosassa on esitelty ohjelmiston testausta, pilvipalveluita, sekä projektissa käytettyjä työkaluja. Käytännön osassa on esitelty työssä tehdyn testausautomaatin elementtejä, arkkitehtuuria ja komponentteja, sekä itse automaation toimintaa. Asennusohjeet ovat työssä mukana liitteenä. Testausautomaation perustoiminnallisuus saatiin rakennettua ajallaan siten, että sitä pystyi käyttämään FreeNestin testaamiseen. Osa suunnitelluista lisäominaisuuksista jäi kuitenkin keskeneräisiksi tai puuttumaan kokonaan ajan puutteen vuoksi, mutta niiden puuttumisesta ei kuitenkaan koitunut huomattavaa haittaa testausautomaation toiminnalle. Avainsanat (asiasanat) FreeNest, SkyNest, Robot Framework, Git, TestLink, Selenium, testausautomaatio Muut tiedot

CONTENTS

• TERMINOLOGY
• 1INTRODUCTION
  • 1.1 FreeNest
  • 1.2 Cloud Software Finland
  • 1.3 Objective
• 2 MAIN CONCEPTS.
  • 2.1 Cloud computing
    • 2.1.1 Software as a Service
    • 2.1.2 Platform as a Service.
    • 2.1.3 Infrastructure as a Service.
  • 2.2 Software testing.
    • 2.2.1 Testing in general
    • 2.2.2 Testing approaches.
    • 2.2.3 Testing levels.
    • 2.2.4 Automated testing.
• 3 TOOLS
  • 3.1 TestLink.
  • 3.2 Robot Framework
  • 3.3 Selenium
  • 3.4 Git
  • 3.5 OpenStack
• 4 OBJECTIVES
• 5 ARCHITECTURE OF FNTC
  • 5.1 Development.
  • 5.2 Environment elements.
  • 5.3 Components.
    • 5.3.1 Core.
    • 5.3.2 Testlink API
    • 5.3.3 Git wrapper
    • 5.3.4 Plug-in system.
    • 5.3.5 Robot engine.
    • 5.3.6 Grid engine
  • 5.4 Installation.
  • 5.5 Usage.
• 6 RESULTS.
  • 6.1 Current state
  • 6.2 Future improvements.
• REFERENCES
• FIGURE 1. Cloud computing layers FIGURES
• FIGURE 2. Major software testing levels
• FIGURE 3. A screenshot of TestLink 1.9.4 demo
• FIGURE 4. An example of a passed test case
• FIGURE 5. Robot Framework's high-level architecture
• FIGURE 6. A screenshot of RIDE's interface
• FIGURE 7. Failed and passed reports from Robot Framework
• FIGURE 8. Simplified Selenium architecture diagram
• FIGURE 9. Distributed version control diagram
• FIGURE 10. OpenStack operating system diagram
• FIGURE 11. The original testing automation architecture plan
• FIGURE 12. The required structure of the Engine -class
• FIGURE 13. The class diagram of FNTC.
• FIGURE 14. The sequence diagram of FNTC

TestLink A web based test management tool. Git A code version control system. Python An object-oriented, high-level programming language that emphasizes code readability. (What is Python? Executive summary, Python.org, 3.10.2012) PHP Hypertext Preprocessor, A general-purpose server-side scripting language used for dynamic web pages. (PHP, Wikipedia, 3.10.2012) HTML HyperText Markup Language, mainly used for displaying web pages and documents. (HTML, Wikipedia, 3.10.2012) OpenStack An open source cloud computing platform. JunkCloud Projects' own OpenStack cloud built from scrap computers. Selenium A tool suite for automating web browsers. Selenium Grid Hub The central point of the grid, distributes all the tests for nodes to run. Selenium Grid Node A slave machine in the grid, does all the hard work and reports the results to the hub.

1 INTRODUCTION

1.1 FreeNest

FreeNest is an open source application life cycle management environment developed in the SkyNest -project at the School of Technology at JAMK. Since the project is hosted at JAMK, most of the workers in the project are students either performing their internships or working on their bachelor's thesis projects. SkyNest is part of the Cloud Software Finland -program and it is funded by Tekes, the Finnish Funding Agency for Technology and Innovation. (Projects, Jyväskylä University of Applied Sciences) FreeNest is meant to be used for managing the project life cycle from the initial idea to the development of the application, to support after the release. It integrates several widely used open source tools and acts like glue between them, making it possible for the tools to communicate between each other. It is built in a way that one FreeNest should be used in one project and when the project ends and a new one starts, the environment gets taken down and a new one is launched for the new project. If there are several projects, there have to be several FreeNests. (FreeNest, 2012) FreeNest includes more or less all the tools that are needed in a software project and it can be configured for users' own needs. The current version at the time of writing this thesis, 1.3, is available as a virtual machine image, but 1.4 will be available as Debian packages, making the installation more flexible. The user can then decide which tools to install and the environment is ready for use in a short time. (FreeNest, 2012)

2 MAIN CONCEPTS

2.1 Cloud computing

A cloud can be considered to be a large pool of resources, like a cluster of computers connected to each other, acting as one very powerful and flexible computer. All the data and power is spread on multiple nodes, so if some nodes break, their tasks are moved on to other nodes and the cloud is still working without any data loss. The computing resources can be distributed as services in the form of applications, servers, virtual machines or platforms over the Internet where ever they are needed and they can be scaled up or down depending on demand. The user does not have to buy or maintain new hardware since all the hardware is located in large data centers maintained by the service provider. (What is cloud computing?, IBM; Cloud Computing, Wikipedia) There are several types of cloud services, the most notable ones being SaaS (Software as a Service), PaaS (Platform as a Service) and IaaS (Infrastructure as a Service)(See figure 1).

FIGURE 1. Cloud computing layers (Cloud Computing, Wikipedia)

2.1.1 Software as a Service

Software as a Service means a single application accessible via a client interface, such as a web browser. The application can be anything from project management tools like Trac to email services like Gmail. The users do not have to install or maintain anything on any of their computers since the software is installed in the cloud and maintained by the service provider. The user only needs to log in to the service from any computer and start using the application. (Cloud Computing, InfoWorld; What is cloud computing?, IBM; Cloud Computing, Wikipedia)

software and should be handled throughout the software's development. It is a verification and validation process, making sure the product behaves the way it is wanted to behave and that the product is built according to customer requirements. (CodeProject, March 2012) Manual software testing is performed by a human sitting in front of the computer, going through the application, trying different usage and input combinations and comparing the results to the expected results. Testers follow a test plan and execute the tests according to the detailed test cases assigned to them. The results are then gathered from all testers and compiled into a test report. (SmartBear; Manual testing, Wikipedia)

2.2.2 Testing approaches

White-box testing There are three approaches for testing software: white-box, black-box and gray-box testing. White-box testing, also known as glass box or structural testing, means that the tester has full understanding about what happens inside the program and test cases are designed using that knowledge and programming skills. The tester chooses his inputs in a way that he can check all the paths through the code and determine appropriate results. (CodeProject, March 2012; Software testing, Wikipedia) Black-box testing Black box testing, also known as functional testing treats the program like a black-box. The tester has no knowledge about the internal

functions of the program; he only knows what the program is supposed to do and not how it does it. The tester takes the role of an end user; tries different inputs and checks that the program returns appropriate outputs. (CodeProject, March 2012; Software testing, Wikipedia) Gray-box testing Gray-box testing is a combination of white-box and black-box testing. The tester can have access to the internal code of the program and has some knowledge about the functions of the program. The test cases are designed using that knowledge; however all testing is still done in black-box level. The testers can for example manipulate the data inside a database, which would not be possible in black-box level, or use reverse engineering to determine boundary values or error messages and then test the program like they would normally do in black-box testing. (Software testing, Wikipedia; Software Testing Fundamentals)

2.2.3 Testing Levels

Software testing can be categorized into testing levels based on the depth, scale and subject of the tests. There are three main levels of software testing: unit, integration and system testing as shown in figure 2. Acceptance testing is the last level of testing where the functionality of the software is compared to the business requirements and while it is an important part of testing, this thesis will not go into specifics or acceptance testing. (Software testing Fundamentals; Software Testing, Wikipedia)

testing approach. (CodeProject, March 2012; CodeThinked, June 2009; Unit Testing, Wikipedia) Integration testing Integration testing focuses on testing the interaction between a group of units of the system. While unit testing makes sure that individual units work as they should, integration testing makes sure they also work well together. If the software and hardware have any relation, that is also tested in integration testing. The testing is usually handled using the unit interfaces, simulating different scenarios by using different inputs, therefore integration testing usually falls under the black-box testing approach. (CodeProject; Software testing Tutor; Integration Testing, Wikipedia) There are three approaches how to carry out integration testing: Top- down, Bottom-up and Big bang. In top-down approach the integration starts from the graphical user interface and follows the architectural structure to the bottom level, while in bottom-up approach the integration starts from the bottom level. In big bang approach all the developed modules are integrated together to form a complete system. This can save time compared to other approaches, but if the tests are not designed and recorded correctly, the testing process becomes more complicated and might even prevent testing the system properly. (Software testing Tutor; Integration Testing, Wikipedia) System testing The purpose of system testing is to verify that the software works as expected as a whole, fully integrated system. The software should be fully installed on different platforms with different configurations to

make sure that all scenarios are covered. Testing that the system installs correctly and uninstalls without leaving any traces are also covered in system testing. Test case designs are based on the original requirements from the end user's perspective, thus system testing falls under black-box testing approach. (Guru99; System Testing, Wikipedia) System testing includes over 50 types of testing from usability and interface testing to stress, load and hardware/software testing, which means testing the interactions between the hardware and software. In most cases it is impossible to use all testing types for system testing due to limited resources and huge number of testing types, thus the tester needs to decide which parts are important enough for full testing to be necessary. (Guru99; System Testing, Wikipedia) Regression testing Regression testing, also known as verification testing is a special type of testing executed on all testing levels. It means repetitive tests that are run every time the application source code has changed. Its purpose is to verify that recent code changes have not affected the existing features of the application, created new bugs or resurfaced old bugs. Regression testing should cover features that undergo frequent changes and are most visible for users, and a good amount of test cases, including but not limited to all integration and complex test cases and the test that verify the core features of the application. Due to the repetitive nature of regression testing it can take up huge amount of resources depending on the frequency of the changes, so testing automation would be a valid solution to save testing time. (Guru 99; What is regression testing, Webopedia, 2012)

(SmartBear, 2012; Testing automation, Wikipedia) Downsides While test automation can make a huge difference in bigger projects, it cannot be used efficiently in some cases and it can be more harmful than beneficial. Setting up the tool chain in the middle of the project can take a great deal of time and resources, and testing scripts need to be written for all the tests that need to be automated. This can be a serious problem if the project has been going on for a while and hundreds of tests need to be converted into automated tests. That is why automation must be considered at the beginning of the project, tools can be set up at the same time with the development environments and scripts are faster to create while developing new features. 3 TOOLS

3.1 TestLink

TestLink is a web based test management system developed by the TestLink community. It is designed to provide support for test specification and execution, and monitor test activities. It helps the testers to keep track of which tests are assigned to which testers, which tests are supposed to be run in the build and the total progress of the tests. In general it keeps the testing process under control. A screenshot of TestLink's interface can be seen in figure 3. (TestLink user Manual, March 2012)

FIGURE 3. A screenshot of TestLink 1.9.4 demo. Test projects A test project is the base of the whole testing process in TestLink, without it there is nothing to be tested. It can mean a product or solution to be tested and it contains the requirements documentation, test specification, test plans and project specific user rights. Test projects do not share data, so each project should be used for only one product and by only one testing team if possible. Creating a new test project is usually the first thing that needs to be done after logging in with administrator's rights for the first time. This can be done by clicking “Test Project Management” -link in the front page after logging