Gray box is the way to go for automation testing

When we finished the design of our objects for setting up the preconditions of an automated test case we showed it to one of our developers, and his first reaction was “it is the same with the app’s domain model.” This comment made me think that knowing the application internals would be extremely useful in developing correct test scripts. Knowing this had me thinking that the gray box testing would be more appropriate in our automation tests than the black box, thus a switch to that direction is worth considering.

Why is gray box more appropriate in automation and not manual testing? Because any automation tester will have the technical skills to understand the internals of a software application and to prepare the appropriate testing code to interface with available hooks or services, something that a manual (non-programming-savvy) tester would not have.

As a case study, I used one of the projects I was working for. In this project, there was the front-end playing the role of an emulator of an external system. The front-end app is retrieving data from a database using an MVC model. Black box testing ignores where the front-end app gets its data as long as the displayed data is correct. While this is acceptable for any functional test case, the big picture is revealing shady areas of operations. For example for a large amount of data we need pagination (more effort and man-hours out of the developers), in parallel testing, there are significant delays due to concurrency issues (front-end was not designed initially for heavy-duty traffic), and in some cases, there was data loss misleading the testing team into opening false defects.

Switching in gray box testing to identify the internals (understanding how data is written to the database and how it is queried by the front-end) was simple with the usage of some Spring connectors to read the database. The advantages were the following:

• No more maintenance for the front end emulator

• Parallel testing speed up

• Single point of failure (database not emulator)

Switching to gray box testing made our scripts more robust and lifted the maintenance effort from the development team therefore becoming more suitable for our needs. Expanding the gray box testing technique allowed us to start using services like JMX to communicate with the app in a more accurate and direct way, leading to testing various paths inside the app’s code that were not exposed to us initially. This assisted us in revealing not functional but bottleneck-related bugs of the software.

Gray box testing is the way to go for any automation tester because it can reveal potential bottlenecks inside the code before the performance test does and because it simplifies the way test scripts are created. Gray box testing pushes you to learn more tools to hook into the app thus expanding your software development skills thus becoming a better automation engineer.