AgileWay Test Automation Formula, Part 2

Continue from Part 1.

Automation Frameworks

The framework that drives the application.

1. Selenium WebDriver for testing web apps

WebDriver is a W3C standard, just like all web technologies. So, it is a no-brainer to choose Selenium WebDriver for web testing. Don't believe some hyped Protractor, TestCafe, Cypress or Playwright. I have seen many attempts of using each of them, all shared the same outcome: failure (see the Definition of End-to-End Test Automation Success). To add another fact, all browser vendors (Google, Microsoft, Apple, Mozilla) support only one automation framework: WebDriver.

Facebook replaces "UI tier" in its Testing Pyramid with "WebDriver".

Seeing it believing:

Video clip from "Continuous Integration at Facebook"

"Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released." - DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Microsoft deprecated Coded UI Test and recommend Selenium WebDriver in 2018.

The above shall be convincing enough, right? I guess some might still think "That's because they are tech giants, they have resources to get Selenium working". Wrong, raw Selenium WebDriver is much easier than Cypress or Playwright, if under proper guidance.

Here is a regression run of the WhenWise suite, consisting of 551 user-story Selenium tests.

A run of 555 WhenWise suites, I have been developing/maintaining them for over 4 years.WhenWise is just one of several apps that I developed/maintained in my spare time.

Related readings:

• Why Do Most UI Test Automation Fail? (Part 1: Wrong Automation Framework)
• Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework
• Web Automation Framework Trend ⇒ Selenium WebDriver
• Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver
• False 'Selenium WebDriver Cons"​

2. Appium + WinAppDriver for testing desktop apps

Appium is also based on W3C's WebDriver standard.

I have developed and maintained 307 Appium tests for my Desktop App: TestWise.

3. Ruby for testing API/Webservices/Microservices

I don't use SoapUI, Postman, or REST Assured for API testing. In fact, I have a long history (since 2006) of doing API testing by using the same technology, Ruby.

Related reading:

• eBook: API Testing Recipes in Ruby

4. Appium + iOSDriver/AndroidDriver for testing mobile apps

Though Appium is more well-known for testing mobile apps than desktop apps, I haven't done real mobile testing (hello-world level tests do not count). The simple reason is that I did not have my own mobile app. I did take a contract role with mobile testing once. It turned out that the mobile app is a wrapper of a web app. This is the area I am planning to explore more in the future.

Based on my experience with Selenium WebDriver and Appium with WinAppDriver, I think Appium + iOSDriver/AndroidDriver will fit well with this formula.

Scripting Language: Ruby

Besides Ruby, I am proficient in developing Selenium tests in other languages as well.

However, Ruby is proven far better than others.

Related readings:

Q & A

• What do you think of using Java with Selenium?

Java (and C#) is a compiled language, and I know Java very well (programmed professionally between 1997–2010). However, test scripts shall be written in a scripting language such as Ruby and Python.

• How about JavaScript?

JavaScript is not a good language for scripting automated tests. Please read Why JavaScript Is Not a Suitable Language for Real Web Test Automation?

• How about Python?

If all your team members (including business analysts) are comfortable with Python indentation, it is OK.

The original article was published on my Medium Blog on 2021-03-31