Test Automation and Continuous Testing Competition Week. Part 1: How?

Most software companies (or IT divisions) have no clue how to do automated functional testing or execute these tests in CI/CD. Not a believer? Check the end-to-end test reports in your CI server. Without executing automated end-to-end tests, there will be no ‘Delivery’ (of CD), right?

I am talking about the really useful automated testing that provides the teams with useful and real feedback which will ultimately enable the software to “Ship early, ship often”. For those companies who have claimed that they are doing automated functional testing, what they have in common is that there are dozens of different non-working test automation frameworks/tools existing within the projects and the company. Subsequently, the outcome is the same: the automated testing is either failed already or is about to fail.

I still remember a meeting that was called by a newly-on-board test architect in a large organization. There were about 70 testers in the room. The test architect asked a representative from each team to shout out their test automation framework or tool. He was surprised that there were so many different frameworks and tools that were currently in use within the organisation. He was even more surprised to learn that the testing was still mostly done manually, while with all these test automation frameworks and tools!

It was obviously a mess and a big waste, but it is very common. How will you solve it? In this article, I will suggest a practical (and fun) way to create successful test automation and Continuous Testing (CT) Formula for your company, in a matter of days. I call it “Test Automation and CT Competition Week”.

With the executives’ support, it is actually quite easy and cost-effective. Here is how: set up a competition event (like some companies’ innovation day) and invite all interested engineers to participate in groups/teams.

1. Nominate a set of acceptance tests for your main app. 25 is a good starting number.

Usually, a company has already defined a suite of acceptance tests (for manual regression testing) for its main application. Therefore, it shall be easily done. My suggestions for test selections:

• Cover a broad range of business functions
• Web features in testing perspective, such as AJAX, verifying data in a complex table, verifying generated PDF, file upload, frames, multiple browser tabs, multi-domains, drag-n-drop, pages with dynamically-generated IDs,…, etc
• A good mixture of simple, medium, and complicated test cases

The number of tests must not be too many or too few. Too many, people tend to give up; Too few, besides lacking the coverage, will be inadequate to verify the CT process.

2. Announce Test Automation and Continuous Testing Competition (3-5 days)

Some companies might call it “innovation day or week”. The movie “The Internship (2013)” had this kind of competition among the intern teams at Google. The ideal duration of the competition is 4 days (leaving some time for introduction and reviewing). It should definitely not exceed one week.

The purpose: each team will be given exactly the same task to complete a regression suite

A team consists of 1 business analyst or manual tester and 3 software engineers in Test or Software Engineers. The business analyst or manual tester in the team provides business knowledge of the test cases.

The rules of competitions shall be simple:

• No limitation of technologies (important!)
• At least 3 new functionally correct automated tests are added to the CI build daily. If failed to meet this goal for 2 days, the team will be disqualified.

Here I would like to highlight the importance of “No limitation of technologies”. Executives don’t usually know a sad fact: many good ideas were shot down by the tech leads and mid-level managers due to their fear (of something new or they don’t understand).

3. Prepare the environment and support

Before the competition starts, ensure the teams get all the infrastructure support they need, such as

• the testing server can handle the load
• dedicated test user logins
• testing tools (trial licenses are fine)
• virtual machines for parallel testing
• all access privileges are sorted out

This is also a test for your infrastructure team.

4. About halfway into the competition, introduce some simple application changes that would affect some tests in the suite

The purpose is to check the team’s capability to maintain automated test scripts. I suggest the following changes:

• Changes to some HTML element IDs and names
• Addition of new mandatory elements, e.g. text field, on some pages.
• Removal of web elements
• Changes to some buttons and link texts
• Extend some AJAX operations
• A brand new form is added to the business process

The software updates (typically, check out a tagged revision from the Git repository) shall be rolled out gradually. Taking a 5-day competition as an example, there should be releases on the early mornings of Day #3, #4, and #5.

5. Introduce some defects to the application

This is to verify the quality of regression tests and how quickly the teams can detect them. Simulate real working situations.

The major challenge of E2E Test Automation is not creation, but rather Ongoing Maintenance. Check out my other article, Test Creation Only Accounts for ~10% of Web Test Automation Efforts

--

In Part 2, I will talk about how to assess the results of this competition and possible outcomes.