The testing pyramid should look more like a crab
it's time to fully embrace end-to-end testing
This is a fancified excerpt from Gleb Bahmutov on JS Party #148. Gleb is VP of Engineering at Cypress.io. To get the full experience you should listen while you read.
đ§ Click here to listen along while you read. It's better! đ§
The Testing Pyramid
Typically, people say that testing is like a pyramid. Imagine a pyramid like in Egypt.
unit tests
At the bottom you have a very wide layer of unit tests. And unit tests test the smallest piece of code.
(Imagine you write a function that adds two numbers; so you write a test. If I call that function with arguments 2 and 3, do I get 5?)
Every language under the sun has a unit testing framework, because itâs so easy. Just load a piece of code, run it, check the result that you get, make sure itâs what you expect. So thatâs why the bottom of the pyramid is usually very wide, because itâs easy to just write hundreds of tests to exercise all your little components.
integration tests
And when you move higher up in the pyramid and now youâre trying to put units of code together.
Maybe youâre trying to use a Todo
class, but represent something and you do some other pieces of code. Youâre now mostly trying to see if a couple of units of code work together; how they integrate. And thatâs where you discover parts where the backend team and the frontend team actually did not communicate very well. So my module doesnât work very well with another module.
end-to-end tests
Then at the very top of the pyramid you have end-to-end tests.
An end-to-end test is when youâre trying to run the whole thing as the end user would. For example, you open a website in your browser and you navigate and you work with your web application and you check if it updates the page correctly, if it calls the backend correctly.
The top of the pyramid is usually very sharp. Thatâs because youâre not supposed to write many end-to-end tests. I think this is obsolete thinking nowadays, because why was it hard to write end-to-end test?
It was hard to install the end-to-end test runner. It was finicky. It was flaky. The tests were flaky and didnât give you much confidence, so you actually spent more time maintaining most end-to-end tests than you would actually spend time writing your web application.
Many people say:
Write many unit tests, write many integration tests, but just a few end-to-end tests (maybe just as a sanity).
And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests. You wanna make the pyramid almost like a rectangle or maybe as a pizza slice, where you have a lot of end-to-end tests and a few unit tests.
The Testing Pizza
It goes back to efficiency. If you test a small, single function that adds two numbers⌠Yeah, the test is easy, itâs fast, but it really only hits that particular function. But your web app is large and potential sources of errors are not just logical errors in your functions:
- Itâs integration
- Itâs assumptions
- Itâs your bundler
- Itâs your transpiler
- Itâs your code deployment
- Itâs your backend server
- Itâs your DNS configuration
- itâs all the environment variables
That you have to set correctly in the backend and the frontend and all the little internet stuff in between. And the modern browser (which is an awfully, awfully complicated machine) where your assumptions that it will execute this add âtwo numbers togetherâ completely is different from what the end thing will do.
So when you think about whatâs the effectiveness, or how much do you actually exercise? How many potential errors can you find?
Well, the unit tests can find you a few logical errors, which is great. I write unit tests for that all the time. But all possible sources of error are discovered by end-to-end tests.
If you can test the site you just deployed and go through the main user story âjust like a human user would do later onâ if that works⌠the chances are when the real user goes through the same thing, it will be successful.
So for us, the end-to-end test should be the primary focus.
So if we flip the pyramid and we make the top wider and wider, and we will write more end-to-end tests, because theyâre effective and we make it almost like a pizza slice, where we write more end-to-end tests, or if we start with end-to-end tests, it makes sense.
The Testing Crab
And what happens recently? Well, a functional tester or test runner like Cypress finds a text, clicks on the button, does all those things, but it only verifies that the application works.
It doesnât verify that the applications looks good. And weâre all humans. We like pretty things, so we like styles. Some people even add CSS to their apps. I donât know why, but itâs crazy. đ
Once they add CSS and they do the styling, they want to make sure the app looks the same, and they donât accidentally break it.
If we select elements, and you work with them, and you check the number of items, the CSS can still change. And then the application will look like crap, and users will be unhappy, and nobody is gonna buy anything from your website.
Cypress is just a functional test runner. It doesnât care about CSS. And itâs very hard to write all the assertions saying:
The color of this element should be blue, and the border radius should be 2âŚ
Itâs impossible.
visual tests
So instead, what people do âbecause itâs a real browserâ is you can take and generate a screenshot of your page (or a part of it) and then you can do visual testing.
So you save a screenshot and it becomes a baseline or a master image. The next time you run the test you take another screenshot and then you compare it pixel by pixel with your baseline imagine. And you store those baseline images with your source code in your repository.
Computers are really good at comparing images pixel by pixel, and theyâll tell you:
Oh, it used to be blueâŚ
The computer doesnât have a concept of blue, but it says:
It used to be this. Now itâs a different color. Hereâs where it changed
And then you can say:
Did they mean to change the CSS here? Why is it no longer blue and now red?
Visual testing to me is such an effective tool, paired with a full end-to-end test.,You can literally load the application, take a screenshot, and now you know it will never change accidentally. Do something where the application reacts, changes the layout, new DOM elements appear. Then you take another screenshot.
Boom. Now you tie it so close, any visual change (any CSS, SVG, anything) will change with pixels, and youâll know that you accidentally broke the styling. And youâll know that accidentally you made Craigslist look like RedditâŚ
And youâre like
No, no, no. Revert it, revert it, revert it!
So to me, the most useful pyramid right now is a pyramid thatâs wholly end-to-end, functional and visual tests. Thatâs it.
And then, I can track code coverage instrumenting my application code; the end-to-end tests that will go through the whole flow, like a real user, are so effective at code coverage.
People say
Well, hit 80% of your code (coverage).
And I usually say
Well, because end-to-end tests exercise the whole application, a single test can cover most of it, if it goes through the whole user story.
And then you look at the lines not covered, and you write end-to-end tests for those edge cases. And if you cannot reach both lines, because there could be edge cases that are unreachable through a well-designed interface, then you write end-to-end tests, API tests and component tests, and hit those lines, so you know that those components and unit tests of code work as well.
But it becomes a pyramid of end-to-end tests, little triangles for other types of tests, and to me it looks like a crab, because itâs a big kind of helmet shell, and little armored legs under it.
But itâs all about end-to-end tests, in my opinion.
Our conversation with Gleb doesnât stop there. Listen to the entire episode to hear the story of Cypress, how they found a business model around open source, how you can make your end-to-end tests less brittle, and much more. Play it from the start right here đ
Oh, and donât forget to subscribe to JS Party in your favorite podcast app so you donât miss future episodes and insights. âď¸
Discussion
Sign in or Join to comment or subscribe