Real-Time Reporting

One of the most critical aspects of regression testing is the test results reporting. The most popular reporters are based on listeners writing to files upon event triggering and translating the files to HTML to display the results. The aforementioned reporting technique is well suited for test scripts with a small number of quick test steps, but not sufficient for a larger scale of tests because of the waiting time before a report could be generated.

The regression suite we have at Persado falls into the large scale tests with lots of test steps category. Each test step has a long execution time mainly because of the preconditions setup and time-consuming processes of the application under test. Report for each test is generated at the end of the test execution thus wasting time in discovering issues that arise at the first test steps.

The solution to the above problem is to create a real-time reporter displaying the results as happen without waiting for the end of the test execution. The reporter for the test framework we use Anax is based on the AnaxTestReporter interface that is compatible with JUnit 4.

public interface AnaxTestReporter {

    void startOutput(String reportDirectory, String suiteName) throws FileNotFoundException;

    void setSystemOutput(String out);

    void setSystemError(String out);

    void startTestSuite(Suite suite) throws ReportException;

    boolean endTestSuite(Suite suite) throws ReportException;

    void startAnaxTest(Test test);

    void endAnaxTest(Test test);

    void startTest(Test test, TestMethod testMethod);

    void endTest(Test test, TestMethod testMethod);

    void addFailure(Test test, TestMethod method, Throwable t);

    void addSkipped(Test test, TestMethod method, String skipReason);

    void addError(Test test, TestMethod method, Throwable t);
}

The implementation of the above interface is a custom reporter module embedded in Anax

that instead of writing the events to a file posts them as an external service.

@Slf4j
@Component("customTestReporter")
@Controller
public class CustomReporter implements AnaxTestReporter {

    private String reporterBaseURL = "http://localhost:8081/websocket";
    WebClient webClient = WebClient.create(reporterBaseURL);

    @Override
    public void startTestSuite(Suite suite) {
    webClient.post()
                .uri("/app/testSuite/Start")
                .body(Mono.just(suite), Suite.class)
                .exchange()
                .flatMap(response -> response.toEntity(String.class))
                .block();

    log.info("Websocket Reporting TestSuite Start... ");
    }

    @Override
    public void startAnaxTest(Test test) {
    webClient.post()
            .uri("/app/test/Start")
            .body(Mono.just(test), Test.class)
            .exchange()
            .flatMap(response -> response.toEntity(String.class))
            .block();

    log.info("Websocket Reporting... Start Anax Test");
    }
}

The calls from the Anax custom reporter are received to a spring boot service that serves as a backend to an angular frontend. The spring boot app after receiving the triggering event stores the data to a DB and transmits a command to the front-end via a WebSocket.

@RequestMapping("/websocket/app/testSuite/Start")
public void sendTestSuiteStartData(@RequestBody Suite suite) throws JSONException {
    log.info("We Got The Test Suite Start from the reporter ...");
    Execution executionSchema = Execution.builder()
            .testSuiteName(suite.getName())
            .testSuiteStart(new Date().getTime())
            .tests(getTestSuiteTests(suite.getTests())).build();
    executionsRepository.save(executionSchema);
    Execution currentExecution = executionsRepository.findTopByOrderByIdDesc();

    JSONObject json = new JSONObject();
    json.put("command",ExecutionStates.TEST_SUITE_START);
    json.put("id",currentExecution.getId());

    this.template.convertAndSend("/topic/greetings", json.toString());
}

The front-end implemented in angular receives the command

connect(){
  const ws = new SockJS(this.webSocketEndPoint);
  this.stompClient = Stomp.over(ws);
  return this.stompClient;
}

onMessage(stompClient: any) {
  this.stompClient.connect({}, frame => {
      this.stompClient.subscribe('/topic/greetings', (message) => {
        if (message.body) {
          this.callComponent(message.body);
        }
      });
    }
  );
}

callComponent(value) {
  const message = JSON.parse(value);
  if (ExecutionStates[(message.command)] as unknown as ExecutionStates === this.registerExecutionStates.TEST_SUITE_START ||
    ExecutionStates[(message.command)] as unknown as ExecutionStates === this.registerExecutionStates.TEST_SUITE_END){
    this.invokeTestSuiteEvent.next(value);
  }
  else{
    this.invokeTestEvent.next(value);
  }
}

and based on received command retrieves data from the DB and displays accordingly

executions: Suite [];

handleAction(action){
  const message = JSON.parse(action);
  switch (ExecutionStates[(message.command)]) {
    case this.registerExecutionStates.TEST_SUITE_START as unknown as string: {      this.testSuiteService.getExecutionById(message.id).subscribe((data: Suite) => {
        this.executions.push(data);
        this.refresh();
      });
      break;
    }
    case ExecutionStates.TEST_SUITE_END as unknown as string: {
 this.testSuiteService.getExecutionById(message.id).subscribe((data: Suite) => {
        this.executions.pop();
        this.executions.push(data);
        this.refresh();
      });
      break;
    }
    default: {
      break;
    }
  }
}

The image below demonstrates a test execution displaying real-time results displaying the test process where the test preconditions have been executed while test step 1 is currently executing

Real time reporting after test step 1 is finished and test step 2 is currently executed.

Real-time reporting after test step 1 is finished with status pass, test steps 2 and 4 have failed, test step 3 is skipped and test step 5 is currently executing.