Continuous Integration Testing Using Postman Collections With Jenkins

A little bit of history:

Some of the biggest issues a QA automation team faces are flaky UI tests and long regression execution times. Flaky tests usually are introduced either by UI loading times and/or continuously changing element locators. The long regression executions are the result of including into regression suite validation cases and negative tests. In order to eliminate both issues, we decided to pass the validation tests and the negative tests to the Manual QA team while the flaky tests could be minimized by avoiding using the UI.

In previous article, this strategy was described in further detail.

A universal solution in the aforementioned issues was to introduce a semi-automation suite to cover the validations and negative tests driven by the manual testing team and avoid the web interface by utilizing REST requests to the backend service in the same way that a form is submitted to the frontend. An example of a test would be the validation of an upload form, in the case of a user uploading an incorrect file format, we would submit the invalid file to the backend and assert the response.

With the responsibility transferred to our Manual QA Engineers and the methodology validated, the next step was "how to create an automation pipeline without writing code"?.

Problems to Solve:

1. Find a way to create full API scenarios and not just individual API calls.

2. Find a way to validate the response status, message, HTTP code.

3. Automate API tests which will run under a continuous automating testing pipeline.

Solution:

So, we needed to figure out a way to help our Manual QA engineers to automate the process, to implement a continuous integration solution without the need for coding but by implementing, probably more suitable to be called, a semi-automation continuous testing process that will execute several API calls, then determine how the backend respond to cases like negative ones or not.

Tooling:

To implement this semi-automation approach we needed to find the appropriate tools to support our needs, such as:

1. A tool that would provide us with the capability to create tests, most likely API ones organized on collections, and at the same time support us with a library that will validate the responses. The tool that suits our needs is postman with its capability to assert responses.

2. Secondly, a tool that would execute the exported test collections from the Command line. The tool that covers it, is newman. The ability of command-line execution actually gave us the opportunity to run our postman tests on our continuous integration tools like Jenkins.

Our tests, exported as postman collections are being stored on a GitLab repository where our manual QA engineers pushed their tests.

Example:

Assume we need to run tests to validate the case of uploading a file with faults and to test how the system responds to situations like incorrect formats, missing columns, or even incorrect column names.

Step1: Prepare your Postman tests, export your Postman collection, then push it to your GitLab repository.

Example of postman exported test validating wrong date column format

{
  "name": "RA Upload File - Wrong Date Format",
  "event": [{
    "listen": "test",
    "script": {
        "exec": [
        "pm.test(\"Response error code\", function () {",        
        "pm.response.to.have.status(422);",
        "pm.test(\"Invalid dates error present\", function () {",
        "var jsonData = pm.response.json();",
        "pm.expect(jsonData.parse_errors[0]).to.equal(",
           "Message results file contains invalid date format:         
           please use d/m/yyyy or d-month-yy\");
     ],
     "type": "text/javascript"
         

Example of exported POST request of file with wrong date column format

"request": {
     "method": "POST",
     "header": [{
	  "key": "X-User-Email",
	  "value": "{{user-email}}",
	  "type": "text"
         },
	 {
	   "key": "X-User-Token",
	   "value": "{{user-token}}",
	   "type": "text"
	 },
	 {
	   "key": "Content-Type",
	   "name": "Content-Type",
	   "value": "multipart/form-data;",
	   "type": "text"
	},
	"body": {
	  "mode": "formdata",
	  "formdata": [{
	      "key": "payload",
	      "type": "file",
	      "src": "files/ra_wrong_date_format.csv"
	   },
	"url": {
	   "raw": "{{my-url}}/message_results_files/upload",
	   "host": [
	      "{{my-url}}"
	   ],
	}
   },
   "response": []
 }		

Example of postman test with wrong date column format

pm.test("Response error code", function () {
    pm.response.to.have.status(422);
});

pm.test("Invalid dates error present", function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.parse_errors[3]).to.equal
    ("Message results file contains invalid date format: 
    please use d/m/yyyy or d-month-yy");
});

Step2: Execute Collections from Command Line Interface to debug or even test locally

newman run <collection name> -e validations_environment.json 

validations_environment.json: is the file that stores execution variables like URL, token, username, or even passwords.

Step 3: Integrate Postman Collection with Jenkins

Create a job with a pipeline to run the Postman collection:

stage('Run collection'){
  steps{
    script{
      parallel(
        "Project":{ 
        sh 'newman run postman_collection_release.json  
             -e validations_environment.json 
             --suppress-exit-code 1 
             -- reporters cli,junit,htmlextra 
             --reporter-junit export "newman/report.xml" 
             --reporter-htmlextra-export                 
             "newman/report.html"'
          }
        )
      }
    }
  }
}

The pipeline will run postman, include JUnit or Html reporters, specify the path for the exported newman report.xml, and then publish the Html report as a post-action.

post {
   always {
        archiveArtifacts artifacts: '**/*', followSymlinks: false
        junit allowEmptyResults: true, keepLongStdio: true,      
        testResults: 'newman/report.xml'
        publishHTML([allowMissing: false, alwaysLinkToLastBuild:         
        true, keepAll: true, reportDir: '<PROJECT_NAME>/newman/', 
        reportFiles: 'report.html', reportName: 'HTML Report', 
        reportTitles: ''])
   }
}

Do not forget to install the Html Publisher plugin on Jenkins

Finally, once you've finished building your project, you'll get a nice, easy-to-read report like: