Hi there, in February Nemanja Talijan and Rade Dragosavac held a Meetup at Symphony Belgrade where they talked about everything we need and don’t need to know about software quality assurance. As always, we made sure to give a proper technical overview to our guests, as well as ensuring that everybody takes something valuable home.
The Meetup itself was dedicated to quality assurance, its processes, valuable inputs and main aspects which actually lead to “quality software”. Our aim with this Meetup was to bring closer what exactly one QA engineer is working on during his engagement and what are some of the processes of his/her job that you perhaps didn’t know. We also shared the benefits and downsides of this process and a brief overview of testing processes in large systems with the help of the Test Pyramid.
History
But, we have to go way, way back first. In 1947 team of engineers, found a moth in Panel F, Relay #70 of the Harvard Mark II system.
The computer was running a test of its multiplier and adder when the engineers noticed something was wrong. Later on, the moth was trapped, removed and taped into the computer's logbook with the words: "first actual case of a bug being found.” The book Grace Hopper and her associates worked on can now be found in the Smithsonian Museum.
Fast forward to today, the annual cost of bugs to US economy is 59.5 billion dollars.
The Cost of Bugs
Of course, it’s very important to emphasize that the cost of the bug defect largely depends on the software development stage in which it was found. These costs can even be lower if the bug is found in the very concept or design of the software solution.
Bugs cannot be avoided, but we can control them. The price of bug repair grows as time passes and fixing it can cost up to 7k USD. Why is that?
If you find a bug a few minutes after it’s made, you are still in the focus and you can easily fix it.
If you have just created a bug in the code, this code still has no dependency and repair is more expensive as dependences accumulate and more time is needed to fix the problem.
If a bug is known for some time and the paths around the defect have been created, and if you fix the defect just now, it may happen that it does not work anymore, which is even more expensive.
If the defect is found in the production environment, it is highly probable that the user has found it, which, in addition to creating a bad image.
As soon as the defect is fixed, it is less likely to make a problem, hurt or maybe even kill someone, and start WW3. Just think about it.
CONCLUSION: Repair the bug as soon as possible, okay?
Automated vs Manual Testing Approach
The big question comes here: manual vs automation testing. We have to admit it, manual testing is a great gain for the project because as soon as the SQA engineer starts to test the product the value can be seen immediately. On the other hand, it’s very time consuming especially if you need to test one feature over and over again. When we talk about automation testing, first of all, we have to see that it’s an investment because we cannot see the full value at the beginning. On the other hand, execution is very fast, it saves time and it can be run at any time. But, how to pick the right automation tool?
First of all, this tool needs to be easy to learn in order for everyone to get to know the programming language fast. You don’t need a powerful and fast programming language, we need a language that has support for the Selenium (UI testing) and libraries for backend testing. An ideal language for this is Python!
White, Black or Gray Testing Methods
We recognize three kinds of testing methods: White Box, Black Box and Grey Box. White Box testing method is a software testing method in which the internal structure/design/implementation of the item being tested is known to the tester. The main focus of this method is code structure (classes, methods, loops, arrays) and it can be implemented on Unit testing. Black Box testing method is a method in which the internal structure/design/implementation of the item being tested is not known to the tester. In this phase, QA engineer is observing software like a black box - knows what software should do, but don’t know how exactly. These tests can be functional or non-functional, though usually functional. Grey Box testing method is a combination of the two aforementioned and is the most frequently used method because QA engineer usually has some access to the source code (API documentation, database access, etc.).
We also talked about the timeline of one sprint and its benefits. When the tasks are created and distributed among development team members, it starts in parallel with the development of software and writing automation test scripts. The benefit of this method of parallel work is that immediately after completing the development of the software, functionality is checked by running automation tests, thereby shortening the time for any repairs, increasing productivity and eliminates confusion in relation to the client's request.
SQA and Agile Working Together
The goal of all agile approaches to software development is to deliver product increment in controlled conditions using good practice and a range of actions that will ensure the smooth operation of development teams.
One of the important moments in the process is the moment where the so-called “Code Freeze” happens when all development stops. That also gives an additional time for QA engineers to launch regression tests - tests that contain all the automation scripts developed up to that point and which check how the software functions as a whole. After that, the regression tests are run on the staging environment, where the interaction of our module with modules developed by other teams is tested. After every sprint, we have few important meetings such as Sprint Review and Sprint Retrospective. During the first one, we talk about how we did our tasks and during the latter one we give answers to questions such as “what we did?”, “what was good?”, “what was bad?” and “what we need to change?”. Now, where do the SQA engineers fit here? First of all, their responsibilities are incorporated in the whole software development process and they are the ones who should know the product the best. They work closely with stakeholders and their mission is to always look ahead. With software testing the goal is to:
- Ensure you did things right - Ensure you are doing things right - Ensure you will do things right - Ensure you understand things right - Ensure you did the right things
Pyramid, but without Egyptians
One of the most important lessons we made sure to talk about in depth was the Software Quality Assurance Tests Pyramid.
Its goal is to help us understand how to achieve sustainable, high-quality product during five testing levels. The first one is Process Chain Testing at the top of the pyramid which is very user-centric and it’s very complex to test with nearly infinite potential failure scenarios. The good news is that if these tests work they are a guarantee that all subsystems and steps worked. System Tests have a more technical approach and they are more system specific. Their duration is 5-15 minutes and they provide more in-depth testing. Integration Tests are intended to check I/O of a system’s components. They are much smaller than System Tests and their duration is 50ms-few seconds. Feature Tests are fast to execute and they test the interaction between multiple methods and/or classes that create together some feature. Their duration is as little as 20ms. Unit Tests are something we call the bread and butter of software development and their main purpose is to ensure that the code is working even when it’s refactored. They are easily written and used. The main question here is: Is there any reason to test the higher levels of the pyramid? No! Why? Because it’s too expensive, slow and unreliable.
Testing Framework As It Should Be
In the end, we talked a bit about automation testing and best practices which ensure us to easier write and maintain tests. The code can be reused and tests provide maximum coverage. The cost of maintenance is very low once you make a good automation framework and they actually decrease manual intervention. The reporting is easier as it can be connected to TestRail or Jira. Page Object model gives a framework all these benefits where every page/screen has its own class which makes code reusable and reduce insufficient lines of code. Now, let’s just touch base on what makes good automation framework?
- Base(base classes) - Page(page classes) - Test(test classes) - Utilities(utility classes) - Configuration files - Screenshots
Thanks for reading through our overview of the latest Symphony Meetup held in Belgrade by Nemanja Talijan and Rade Dragosavac. More is yet to come!
