TestVagrant

Flutter Blog Series – Blog 2

By /

Flutter Integration Test Vs Appium

appium vs flutter
  • Article

Article

Appium Vs Flutter Integration Test

Hello, and thank you for joining us for the second blog of our series about test automation for Flutter applications. In case you haven’t done so already, we strongly recommend you read the first blog post for a better understanding.

Selecting the right test automation tool/framework

Test automation is a powerful process yielding benefits, including but not limited to: Faster feedback, Increased testing efficiency, Cost reduction, Improved accuracy, Enhanced test coverage and Time-saving.

The very first step and the most important one is selecting the right test automation tool/framework. These tools and frameworks always come with trade-offs that need to be carefully considered and ideally balanced. The choices we make come with costs and risks. Risks inherent in the concepts behind it, in its design and implementation, in our ability to use it, in using it and in our understanding of what it achieves. We need to be diligent about strengths and weaknesses, and aware of potential biases while making the choice.

For the scope of this blog, we would attempt to compare two prominent and matured tools available — Appium & Flutter Integration Test.

Appium is a widely recognized open-source mobile test automation tool, whereas Flutter Integration Test is a built-in testing framework provided by Flutter itself.

Parameters for Comparison

  1. Test design approach
  2. Language Bindings
  3. Support for platforms
  4. Locator Strategy
  5. WebView and Native View Support
  6. Performance monitoring & Network calls
  7. Continuous Integration & Device Cloud
  8. Reporting
  9. Engineering mindset and taste

1. Test design approach

[Appium] Appium test automation is a black-box testing approach. In black box testing, the tester does not have access to the internal workings of the application being tested and focuses solely on the external behaviour and outputs.

[Flutter Integration Test] Flutter Integration Test can be considered a grey-box testing approach. In grey-box testing, the tester has limited access to the internal workings of the application being tested and combines knowledge of its internal architecture and behaviour with external observations of its behaviour and outputs.

2. Language Bindings

[Appium] Appium supports language bindings for a variety of programming languages, including:

  • Java
  • Python
  • Ruby
  • JavaScript (Node.js)
  • C#
  • PHP
  • Swift
  • Kotlin

These language bindings allow engineers to write Appium tests in their preferred programming language, making it easier to incorporate Appium into their existing workflows and toolchains.

[Flutter Integration Test] Flutter Integration Test is written in Dart, the same programming language used to develop Flutter apps. Therefore, engineers can write integration tests for Flutter apps using the same language and tools as their app’s codebase.

This means there is a learning curve for test automation engineers / SDETs in order to learn the nuances of dart language

However, the fact that the app’s codebase and tests are written in the same language, enables the entire engineering team to participate in test automation and reap the benefits of the same. This is particularly suitable for agile teams where there is no clear separation or boundaries between development and test engineer responsibilities.

3. Support for Platforms

It is typical for modern businesses to have web applications, Android and iOS mobile apps, and occasionally desktop applications as well including but not limited to Windows, Mac & Linux.

[Appium] Appium supports automating tests for Android, iOS and Windows desktop applications. However, to effectively automate testing across these platforms, it is necessary to incorporate platform-specific intelligence into the framework, which requires a significant level of effort. Examples: Locator strategy, Driver initialisation and management etc… Failure to incorporate intelligence into the framework may result in code duplication and a cumbersome framework.

[Flutter Integration Test] Flutter Integration Test supports automating tests for Android, iOS, Web, and Desktop applications across Windows, Mac and Linux platforms. Flutter Integration tests enable seamless execution of the same tests across all supported platforms. This results in a lean framework and code. Having said that some level of framework-level intelligence needs to be built to handle platform-specific application differences.

4. Locator Strategy

Locators in UI automation play an important role in the life of an automation engineer. They provide a path to access web elements that are essential to automate specific actions like click, type, checkboxes, etc. Poor locator strategies can result in false positives, leading to reduced confidence and trust in the test automation, which can have negative implications for the team.

[Appium] Appium is in the beginning stage to support Flutter applications. At the moment it is not robust to recognize all the widgets. In some cases, the flutter container element hides the contained elements making accessibility impossible.

[Flutter Integration Test] Easy to locate elements and to add missing locators. Given test code is part of the application’s code, we have the flexibility to add locators to widgets. This significantly improves testability. With this ability to add concrete locators, the false positives can be reduced and the test script’s stability increases. This leads to increased confidence in test automation and positively impacts the team. 

5. WebView and Native View Support

A WebView enables an app to present web-based content, without the need for a conventional web browser such as Safari or Chrome. Essentially, it functions as a browser that can be incorporated into an app and displays dynamic web pages, while omitting the typical features like the address bar and tabs found in regular browsers.

[Appium] Appium supports WebViews and Native Views seamlessly.

[Flutter Integration Test] Flutter Integration Test supports only Flutter views and does not support either Native Views or WebViews. However, The patrol package has the potential to address these limitations. However, its effectiveness may differ from one application to another. If you use patrol, then it has custom finders that simplify the process of identifying elements with complex locators. Read more here

6. Performance monitoring & Network calls

[Appium] Does not have an inbuilt performance monitoring capability to monitor memory, battery, CPU, Network calls, etc… We need to use a different and separate technique to capture the same.

[Flutter Integration Test] Flutter Dev Tools enable us with performance monitoring capability along with network calls, logs and more

Flutter DevTools — Widget Inspector
Flutter DevTools — Widget Inspector
Performance Metric View
Network Tab
Network Tab

7. Continuous Integration & Device Cloud

This needs to be carefully chosen & crafted based on the below parameters:

  • Your application’s supported platforms
  • Build tool & Available runners
  • Execute tests on Emulator/Simulator or
  • Execute tests on Cloud Device Lab

[Appium] It is easy to configure and work with various build tools and device labs such as BrowserStack, LambdaTest and others

[Flutter Integration Test] You need to discover the perfect combination based on the parameters called out above. Currently, the most effective combination for Flutter development is Google Firebase Test Lab & Codemagic, which we have already used with successful results.

8. Reporting

[Appium] It is easy to integrate various standard reporting tools with Appium, including but not limited to Allure, Extent, TestNG, ReportNG, Maven Surefire and more

[Flutter Integration Test] Does not have good reporting options. If you are using Google firebase Test Lab, then we have good HTML reports with logs and videos for each test execution

9. Engineering Mindset and Taste

Certain organisations believe and follow “Quality is Engineering Team’s responsibility”. Here developers and test engineers play overlapping roles and there are no hard boundaries. The primary goal and mindset is to shift-left.

In some organisations, test engineers own test automation in its entirety and there is clear segregation between the developer’s responsibilities and the tester’s responsibilities.

[Flutter Integration Test] is more suited for organisations that believe and follow “Quality is Engineering Team’s responsibility” due to the below characteristics

  • Test code resides within the development code. This enables developers to execute tests after making changes so that they can get the fastest feedback possible to their code.
  • Tests are written at the unit, widget and integration layers. This enables a test pyramid whose benefits are called out in the previous blog
  • Developers could participate in authoring integration tests as both dev code and test code are authored in the same language. No additional learning is needed.
  • The same build & CI tools and process could be leveraged for test execution

[Appium] Traditionally we have seen only test engineers authoring, executing and managing tests.

After analysing the above parameters in detail, you can make an informed decision about which tool to select for automating Flutter applications, depending on your requirements.

Read the next blog in the series as we explore the steps involved in writing the first end-to-end test with Flutter Integration Test.

Other Related Articles

See More Articles
See More Articles

Share This Article

Scroll to Top