Comments on the claims of “7 ways Cypress is different”. All False, Wrong or Lie: Part 2

Continue from Part 1.

Claim 1: “does not use Selenium”, WRONG (they should have)

The wrong statement “Selenium or Selenium-based is problematic” (see this article for reasons and proofs), we can understand, Selenium was seen by Cypress as a competitor, these are marketing words for a commercial company.

However, when I posted a post on LinkedIn (not in any group, a personal update).

Gleb (Distinguished Engineer at Cypress.io, an authority on Cypress) reshared with a comment, calling “everything I wrote (you are reading or have read) is a joke”. That’s unprofessional at least, right?

I wrote my article on my personal blogger and posted “It is featured in a reputable software testing newsletter” (i.e. was selected by a reputable curator who at least consented with professionalism). Even that, I received this personal attack. Suddenly, I could relate to the “Cypress Soviet propaganda” phrase in that Reddit post. My article refers to both Cypress and Playwright, also, I have made accurate predictions regarding the failures of Phantom.JS, Protractor.js, Cucumber, and several other tools. These opinions are purely technical in nature and should not be subjected to personal attacks, regardless of their accuracy. I have never received such verbal abuse from others. See, there is a difference.

I share the same feeling: ‘Angry’ as Lisa Crispin (author of the classic ‘Agile Testing’ book) when people throw evil mudslingings to Selenium for commercial reasons (Selenium is 100% free and open-source), over the years.

All real successful (dozens of) web test automation, I witnessed and heard (trusty source), were implemented in Selenium WebDriver.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Check out my articles for more. As a comparison, every Cypress test automation attempt (I witnessed) was a complete failure.

Having said that, Cypress as a tool (not as an automation framework), some of its features do have value. If Cypress reinvents itself to become a Selenium WebDriver tool (like the upcoming JetBrains’ Aqua), that will be a welcoming change, and may survive and make a positive impact.

Cypress testers: If Cypress is a tool for Selenium WebDriver, and hearing the rumours that Cypress.io (the company) might face the bankruptcy, you will be less concerned, right?

Claim 2: “Cypress focuses on doing end-to-end testing REALLY well”, FALSE

Of course, it is not. If it is really well, why do more and more JS testers dump it (so quickly) and migrate to Playwright? Remember, the target, the web, has not changed for nearly two decades. Cypress had enough long time to prove this claim, and it failed.

Again, every Cypress test automation I witnessed was a complete failure.

Claim 3: “Cypress works on any front-end framework or website”, LIE

Any website? NO! I don’t even need to quote references from others (“My Thoughts on Cypress.io” by Alister Scott on 2019–07–16, comment section).

“There are technical limitations to the way Cypress loads and runs the tests, which are covered in https://www.cypress.io/how-it-works Some of the limitations will be removed (iframes, shadow DOM support) as we continue, and some limits will probably be there for many years.” - Gleb Bahmutov, distinguished engineer at cypress

and the reply from Alister Scott:

I did hear this line “if it’s on the web, Cypress can test it” which I still have a problem with as my recent apps I have worked on have both used iFrames can Cypress can’t test them — including WordPress.com which is a super high volume web application. - Alister Scott, Excellence Wrangler at Automattic (WordPress)

Please, say no defending words. A web framework/tool is released without supporting iFrames, this is objectively a joke? And claim “any website”, …

Selenium WebDriver supports all websites well from DAY 1. If Selenium has only one limitation (in contrast to the extensive list of limitations of Cypress), it is not hard to imagine what the Cypress website might have to say about it.

Claim 4: “Cypress tests are only written in JavaScript”, BAD

This is a bad thing, surely, right? Selenium supports 5 official languages: Ruby, Java, C#, JavaScript and Python. Playwright supports JavaScript, Java, .NET and Python. Supporting multiple languages, from a software architecture’s point of view, indicates a good and mature platform.

I know what Cypress means by that, “JavaScript, in their view, is the best. Only one means no noises”. Playwright, with the majority of users in the JS community, disagrees.

Is JS really good for test automation? History told me that it was bad, very bad.

Let me point out another obvious thing. The prime use of JavaScript is coding, i.e., developer-focused. With any further change to JavaScript, few will consider its testing use.

According to Douglas Crockford, a world-known authority on JavaScript, the future does not look promising.

• The good parts have nothing to do with testing
• The bad parts make testing harder and more confusing

Claim 5: “Cypress is all in one”, Not Good

From my experience, it is not a good thing. Remember, Rational Unified Process (RUP), IBM later marketed the same way: “All in one”.

Test automation means two frameworks by nature:

• Automation Framework, e.g. Selenium, Appium. Drives the app.
• Test Syntax Framework, e.g. JUnit, RSpec, Cucumber. Provides structure and assertions.

A good combination, such as Selenium WebDriver + RSpec, offers flexibility. For example, using a common test syntax framework, such as XUnit, RSpec or Mocha, make it very easy to integrate with any CI or CT servers.

If Cypress.io faces bankruptcy in the coming months, what will do with your existing test scripts? During migrating to Playwright or others, you would wish you at least have chosen a common syntax framework, such as Mocha, right?

Claim 6: “Cypress is for developers and QA engineers” Not Really

I don’t quite understand this statement and description. If it means “Cypress is good for both developers and QA engineers to work on test automation together”, I disagree, mostly.

From my experience, developers and QA engineers usually don’t get along well (might be OK but can be quite bad though). The logic behind this is simple: QA engineers come to a developer (or report directly to the manager) about his defects, i.e., an act of telling “Your code sucks”.

This is especially true when the coding language and test scripting language are the same, like in Cypress's case. It never worked well for the teams I led. Why? Developers tend to criticize QA Testers for the wrong use of tools or language. Later, I solved the problem largely by making the call to switch test scripts to Ruby (from Java), and the conflicts were a lot less thereafter.

Claim 7: “Cypress runs much, much faster.” LIE

Giovanni Rago did a thorough test comparing the performance of tools under different scenarios: Cypress vs Selenium vs Playwright vs Puppeteer speed comparison. The test result: not much difference in terms of speed for the four frameworks, however, Cypress was the slowest in most categories. This matches my observations too.

Speed-wise, it is now well accepted that Cypress is slower than Playwright.

“Playwright tests generally run faster than Cypress tests.

Example: This team saw 3.87x increase in test execution speed after migrating from Cypress to Playwright.” — from Luc Guan’s article

Please note, Cypress’s claim was “much, much faster”, by a conventional understanding, it should be about 2 or 3 times faster. In fact, it was the slowest 😨 from the recent benchmarks.

Anyway, if a testing company quotes ‘much, much faster than others’, shouldn’t publish a test suite and its test results? It is a very simple job, but there never was one on Cypress’s claim page.

The Missing Claim 8: “Cypress has severe limitations.”

There should be another item on Cypres’s “How it works” page. According to Gleb’s words (in 2019), it should be there.

“There are technical limitations to the way Cypress loads and runs the tests, which are covered in https://www.cypress.io/how-it-works Some of the limitations will be removed (iframes, shadow DOM support) as we continue, and some limits will probably be there for many years.” - Gleb Bahmutov, distinguished engineer at cypress

I remember seeing that, in text form (not highlighted as the above 7 claims), such as not supporting frames, don’t support two browser windows/tabs, and others. Now the part is gone.

Maybe all issues are resolved or with a workaround? But if so, it shall be stated there too, right?

Anyhow, this Claim 8 should have been there with the other 7, back in 2019, don’t you agree?

---

In JS testing word, web automation frameworks come and go quickly, especially for stuff like Cypress, only good for demonstration.

As a test automation engineer, I am not concerned about all hypes, because I do real useful test automation, i.e. daily automated end-to-end (via UI) regression testing => Daily Production Releases, using raw Selenium WebDriver + RSpec, unchanged since 2011.

Here is a recent test report of the regression test suite (consisting of 569 raw Selenium + RSpec Tests) of my WhenWise app.

--

This article was originally published on my Medium blog on 2023-07-08.