AgileWay Test Automation Formula, Part 3

Test Syntax Framework and Continuous Testing Server

By Zhimin ZhanPublished 2 years ago • 3 min read

Continue from Part 1 and Part 2.

Test Syntax Framework: RSpec

A test syntax framework defines the structure in test scripts and provides assertion syntax.

RSpec

RSpec is the most popular BDD framework in Ruby. There are v3.8.0 over 194 million downloads alone. While RSpec is mostly used for integration and unit tests, it is very suitable for end-to-end tests as well.

Testing Tools

A testing tool is a tool that develops, runs, refines and debugs test scripts. A good tool enhances professionals' productivity without adding constraints (i.e., vendor locking).

TestWise Functional Testing IDE

TestWise is a next-generation functional (dedicated) testing tool that supports Maintainable Automated Test Design and Functional Test Refactoring.

Also, Aqua, JetBrains' upcoming testing IDE, is worth a look.

Q & A

Can I not use TestWise for developing/executing Selenium Ruby tests?

Of course, that's the beauty of freedom (using Selenium). Choose a testing tool that your team members are more productive with.

It is worth remembering that the audiences of automated end-to-end test scripts are not just programmers. It's the whole team.

Continuous Testing Server

With a large automated test suite, the only practical way to manage the test executions of the whole suite is in a Continuous Testing Server. Please note, I don't use CI servers such as Jenkins as they are only suitable for executing unit tests and integration tests, not UI tests. (As mentioned in My Continuous Testing Journey, I have customized a CT server with some CT features; It is far easier to just use CT Server directly).

1. BuildWise CT Server

For the AgileWay test lab, I set up a BuildWise server on a Mac computer (Mac Mini).

2. Parallel Testing with BuildWise Agents on macOS, Linux, and Win10 VMs

I prefer Linux as it is free and fast, while Windows 10 is slow and expensive!

Related readings:

My Continuous Testing Journey

Q & A

How does the CT server go with unit test execution?

CT servers are OK to execute unit tests as well. At AgileWay, we are light on unit testing. Only for complex business logic, we will practice TDD. Maintaining executions of the large UI tests is our main focus. Comparatively, the unit testing execution is too easy.

How about integration/API test execution?

Yes, we include them in the same way as end-to-end tests, with test script files all starting with api_XXXX_spec.rb.

Do you separate the tests into sub-suites and run them separately?

No, absolutely not. This will cause a number of management issues and confusion. What will you do if 7 out of the 9 suites pass?

So, we always run all the tests. If you want to reduce the total execution time, add more build agents. If executions of some test scripts are not reliable, make them reliable. Sorting them into a low-priority suite is not the answer.

Will BuildWise (and Agents) work in AWS or Azure?

Yes, absolutely. The BuildWise server and Agents do not require high-spec VMs. In fact, the lowest-spec (4GB ram) will do.

Test Automation and Continuous Testing are both highly hands-on and practical. If the tech leads and managers of a company spent more than 1 day deciding the formula, their test automation attempts all failed from my observation. The reason is simple: they don't have the knowledge of test automation and the formula. More discussions or meetings will only lead to a compromise. In the world of test automation and CT, there is only 0 or 1, with little room for compromise.

In Part 3, Test Syntax Framework and Continuous Testing Server