To mock or not to mock - why is this discussion even on the table? Somehow it’s a complete no-brainer that the answer is a straightforward ‘yes’ since developers need to remove all dependencies and use mocking frameworks to write reliable, repeatable tests yielding consistent outcomes. However, the purpose of this article is not to delve into the nitty-gritty details of covering unit tests using mocks and corresponding libraries or to provide a cookie-cutter solution applicable to any system at hand. This article offers different angles and approaches which could come in handy to QA Engineers interested in overcoming mocking obstacles and in laying out the risks and dangers mocking can pose to evaluating the general application health.
So who’s it for?
- Mainly QAs struggling with testing integrations with third-party systems
- Developers who are curious about different possible approaches used to overcome testing problems and making testing dependencies easier in the E2E approach
- Anyone interested in cost-benefit analysis of mock usage within the system
When do we need mocking?
There could be a number of reasons why we would be reduced to using mock implementation instead of consuming either production or test services. One of the most common ones are:
- Test service is not available at all
- Test services are available but not reliable or tremendously slow (a service can temporarily go down/under maintenance or intermittently start returning different outcomes resulting in flaky tests); Additionally, slowness of external services could adversely affect the execution time of automation scripts which we would like to avoid at all costs
- Binding test cases with data provided by the test services could leave a sour taste in your mouth once the dataset has changed. Admittedly, no external dependencies are consistently stable enough, and you could find out that the external provider has done a database dump out of your control, leaving you with a bunch of failing tests. You need to be in control of your data if your aim is to have stable pre-deterministic test runs, therefore using mocks will save you a significant amount of time since you will not have to deal with compromised results caused by external factors
- Each call to the production service could potentially be chargeable depending on the contract signed between the client and provider, thus increasing the cost to the client;
- Calling the production service with real data could significantly have an impact on a person in real life (e.g., repeatedly performing credit checks could seriously affect a real person’s credit check score)
- Mocks can help us overcome the challenge of rigid GDPR test data compliance
- Mocks can help us simulate some behaviors that cannot be easily triggered (e.g. network error), and the list goes on
Overall, mocks help you test code more efficiently and increase test coverage by incorporating more edge cases into your test suite.
How to mock - Possible ways to mock in a QA-friendly manner?
The real-life examples below might not necessarily depict a 100% correct mocking approach, but this is the solution we opted for and came off with flying colors in overcoming testing obstacles within a given project.
Case 1
Let’s say an application X should result in providing a credit check decision (approved, rejected, back office handling needed, …). In order for the credit check to be carried out, four different credit check providers should be called, and based on the response returned by the services, the outcome should vary. As a tester, in order to meet business requirements, you would need to do combinatorial testing and cover all possible cases and variations, but how can you sync the data from all four different services in order to build your test case - e.g., we would need a person named John Doe to be fully approved in credit check provider 1, 2 and 3 and should fail on credit check provider 4 for a reason (e.g., bad credit marks). Services 1, 2, and 3 return soap responses, whereas service 4 returns JSON responses. How can we possibly achieve that?
Action items applied included the following:
- Obtain a production copy of the object (call production endpoint, copy the SOAP/JSON response), and make sure that all sensitive and personal data are anonymized, e.g., a person named John Doe with ID 12345 should now be mocked across all 4 services and confirm that once the services are called, a newly mocked response is returned
- Have a session with the development team (or you can read it from the code if you have access to the backend repository) to determine which properties are consumed by the credit check method and change the specific values needed for your specific test case
- The development team could set up the mock server where these mocks would be run 247 (in our case SoapUI mock server was used for defining SOAP requests and responses, and JSON responses were read from JSON files stored within the backend repository if the flag to consume mocks was turned on). Alternatively, you now have the opportunity to easily set up a mock server via Postman, allowing you to test both REST and SOAP clients (for further info, you can refer to this article).
In this way, you can make all the data directly under your control, helping you produce and verify the desired credit check outcomes consistently. Note that coming up with a reliable setup for testing purposes was not all plain sailing. Data anonymization and investigation of the inner workings of external services was a pretty time-consuming process, and extra caution was needed in object modification in order not to corrupt data in any way.
Unless you have a strict business requirement to cover all the test cases in the E2E fashion, it would be worthwhile to consider offloading most of these test cases to either unit or integration level, should you ever bump into a similar case on your future project.
Case 2
Let’s say an application X is calling a service to get a person's details and is supposed to save all values from the response in the corresponding person database table within our system. The person service is working inconsistently, sometimes throwing errors; however, it represents a critical step in the application under test - all the flows start by validating a person, and should the service return an error, the next step is out of reach.How can we possibly overcome this?
Although copying the production object and faking it to meet your needs could also be possible, due to the different architectural setup, a different approach was applied. We switched to using mocks fully during the test phase and let the code read from the mock database tables as the main source. Wherever we needed a person, we would insert statements in the corresponding tables so that the person could have been validated and the application flows could have been finalized.
If using a database for testing purposes is a big NO within your organization, testers should go through the API documentation and try to utilize application APIs for testing purposes (if available). If not, preferably, helper APIs should be created so that testing can be done with predefined data (at least CRD operations - for proper setup and data clean-up). This will open up an opportunity to create any person on the fly that is necessary for a test case depending on business logic (e.g., under 18, deceased, of a certain nationality).
The problem with our database approach is that the tester needs to be deeply ingrained with the internal workings of the code and needs to know exactly which tables need to be filled in in order to use proper insert statements (assumption: tester must have an access to the database and basic SQL knowledge). Also, it comes with a cost of script maintenance once the external response changes. Using helper APIs should be pretty straightforward for testers, but developing these adds additional complexity to the whole application and involves allocating more development time. Last but not least, tests cannot be written in advance or in parallel until APIs have been defined.
Case 3
The main purpose of application X is to display dashboard KPI metrics calculated and imported by the daily jobs. The test data dynamics is to be tested so that the first day of the month displays the initial revenue gradually increasing up until the month’s end. Special note for this one: no database usage is allowed in the test code.
When code makes correct calculations from the external systems, we run integration tests in an isolated environment where the copy of the external system database would be set up and data modified to our needs. The calculation method would then make a database call, calculate revenue from it and make a POST request with correct values to the fake server. To create a server, we opted for the Wiremock library (make sure you check this one out), stubbed the needed endpoints, and verified the exposed requests. Unlike the usual API verification, if the server returns the correct response, this one entails if the correct number of requests with proper body reaches the server.
To test data dynamics, you can have a “ticking” mechanism where the ticker could send API requests at a predefined interval to simulate the job. This should eventually materialize within the system by displaying the expected value, e.g., revenue to be $1000 for the first day in the month, gradually increasing up until the month’s end. The contract between the development team and the QA should be made on the internal logic of the way calculations will work so that expected results can be predefined.
Pros:
- Deterministic behavior and predictable data dynamics can be achieved
- Tests can be written in parallel or in advance once the backend-QA contract has been defined
Cons:
- Additional time is needed to set up an isolated environment
- Additional time is needed to develop the tick tasks to meet the testing purpose; two versions of the test system are created, sometimes causing confusion while debugging issues - when a problem arises, you need to determine if the real implementation code is malfunctioning or the ticker is acting up
- Maintenance with API changes
Pitfalls
Many eyebrow-raising questions tend to pop up when using mocks instead of real objects, and the concerns are quite justified. How accurate is mock implementation, and who is responsible for it anyway? Are the created mocks equal to their production counterparts, that is - how can we get realistically mocked objects? If we cannot achieve this, mock objects could be invalid, and we could create a bunch of test cases and implement irrelevant assertions that are never going to occur in the production environment, thus wasting valuable time.
How will we know on time that the mock implementation has not become obsolete and that the external system has not changed something on their end (unfortunately, quite often, external teams do not communicate the changes they make and just promote them silently). How can we guarantee that everything working under the mock will work the same way when we switch to production endpoints? Sticking with mocks will not give you peace of mind, but verifying the code implementation on a staging environment with real integrations would significantly decrease the uncertainty and give you a clear sign if you are on the right track before the release.
On top of that, as with the examples and overall mocking process, it can be quite a time-consuming process, additionally requesting frequent maintenance if the code undergoes some changes, which is bound to happen at some point in the future. So, if you have the luxury to leave mocking out, just do it.
Conclusion
Mocking, wielded correctly, can be a genuinely powerful tool for testing that can help you facilitate the testing process and catch bugs early on in the development phase. But It does come at a cost. Quite often, it can be confusing what to mock and how to do it realistically. Additionally, creating and maintaining mock data can be a dauntingly complex, time-consuming, and error-prone process.
So do we really need to fake it until we make it fully functional? The answer will greatly depend on your project needs and application type. If you are dealing with an integration-heavy application, the answer will likely be a resounding ‘Yes’ if no alternatives are available to meet your testing needs. However, only you and your team should unanimously agree upon which direction is the right one for your unique project and purposes and choose carefully when the mock benefits outweigh the costs and when they don’t.
Preferably, in an ideal setup, we should always have the pre-release environment where all production services are in place, and smoke tests can be run so that we can validate all the assumptions made during the development process and gain more confidence that the quality will not be compromised once the release takes place. If that is not the case, make sure that all parties are aware of the calculated risks and silence your inner perfectionist by coming to terms with the fact that some issues will only become detectable in production and will be acted upon accordingly as hot-fixes, regardless of the additional costs these unfavorable situations can incur.
About the author
Aleksandar Djordjevic is a QA Engineer with over five years of experience working at our Belgrade engineering hub.
Aleksandar is a detail-oriented and skilled QA Engineer proficient in manual testing skills and possesses hands-on experience designing, implementing, and executing automated functional Web and API tests. Skilled in test automation, web application testing, databases, Java, C#, and many more. However, most comfortable and experienced using Selenium library with Java/TestNG/Cucumber for automating frontend system components and Rest Assured library for backend automation.
