TestVagrant

Generating JUnit Reports for Flutter Integration Tests with Gitlab Integration

Generating JUnit Reports for Flutter Integration Tests with Gitlab Integration
  • Blog

Blog

Generating JUnit Reports for Flutter Integration Tests with Gitlab Integration

Flutter is an open source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. It provides smooth integration for integration testing also. To do a setup for integration testing, You can refer Flutter Integration Testing Doc.

Prerequisite:

  1. A repository that has some integration tests for the flutter app.

If You need a reference, you can make a clone of my repository. Flutter Integration Test Setup Project Reference.

2. You should be able to run the test on Your local.

JUnit Report Integration On Your Local Machine:

  1. Dart provides one package for JUnit Report, JUnit Package
  2. We will create a shell file to run our Test script. Let’s Say the File name of the shell script is — testRunner.sh
  3. We will install the Junit package using flutter and will activate it for reporting.
  4. We will write our command to run the flutter integration test and store the results in JUnit XML report format.
				
					flutter pub get junitreport
export PATH="$PATH":"$HOME/.pub-cache/bin"
flutter pub global activate junitreport 
junitReportFile="./junit-report.xml" # My Report file path, You can give any path suitable to you
flutter test --machine integration_test/tests/tests/sanity.test.dart -d "$DEVICE" | tojunit --output $junitReportFile

Update your Runner shell file with the above content and run this Shell file. Once your execution will be completed, the test result will be stored in the ./junit-report.xml file.

Running Test on GitLab Runner and Generating JUnit Report:

We will create a simple gitlab.yml file that will run the above shell script. You can enhance the same based on your requirement and configurations.

				
					
stages:
  - test

test:
  stage: test
  script:
    - ./testRunner.sh  ## This is your shell file path which has test execution command
    
  tags:
    - testRunnerMachine
  
  artifacts:   ## Stroing the xml report as artifacts
    when: always
    reports:
      junit:
        - junit-report.xml

Yeah, In the above steps we created a simple gitlab.yml file that will execute our shell script. As per our above shell script, it will store all execution data in Junit Report File i.e ‘./junit-report.xml‘.

we will store the XML file as an artifact for each pipeline execution, that would be helpful to get more logs and failure results. We can generate the trend also with the same.

Till now we are good with integrating the test execution with GitLab and generating readable reports and storing previous reports as artifacts.

One thing you will notice with your execution is, In all cases whether tests are passed or failed, Your pipeline’s status will come as PASSED. This is due to the case that, on flutter test execution, if some test cases failed, it gives only errors instead of showing as failure in the JUnit report. Due to this reason, pipeline status always comes as PASSED.

The above drawback could be misleading in some situations. For example, You have integrated slack/email notification on pipeline failure. But due to the above issue, your notification will always indicate that pipeline is passed.

To achieve the correct status of the pipeline, we will update our above shell script file.

				
					flutter pub get junitreport
export PATH="$PATH":"$HOME/.pub-cache/bin"
flutter pub global activate junitreport 
junitReportFile="./junit-report.xml" # My Report file path, You can give any path suitable to you
flutter test --machine integration_test/tests/tests/sanity.test.dart -d "$DEVICE" | tojunit --output $junitReportFile

## We will parse the Junit Reporter and update the exit status based on pass and error counts

errors=$(echo 'cat //testsuites/testsuite/@errors' | xmllint --shell $junitReportFile | awk -F'[="]' '!/>/{print $(NF-1)}') ## we are getting total error count from testSuit tag
failures=$(echo 'cat //testsuites/testsuite/@failures' | xmllint --shell $junitReportFile | awk -F'[="]' '!/>/{print $(NF-1)}') ## we are getting total failure count from testSuit tag

echo "Total Tests With Error : $errors"
echo "Total Tests With Failure: $failures"

## setting up exit status. Exit ocde 0 if No error else 1
if [[ "$errors" == "0" && "$failures" == "0" ]]; then
    echo " All Tests Passed "
    STATUS=0
else
    echo " Some Test Failed "
    STATUS=1
fi

## Based on above status Code, we will exit 
exit $STATUS

Now we are all set . If Any test will fail, You will get FAILED status and if all Test passed, You will get PASSED status.

References –

Unit test reports | GitLab

junitreport | Dart Package (pub.dev)

Happy Learning..

Share This Article

Other Related Articles

See More Articles
See More Articles
Scroll to Top