Gleb Bahmutov, PhD joins the show for a fun conversation around end-to-end testing. We get the skinny on Cypress, find out how it’s structured as both an open source library and a SaaS business, tease apart the various types of tests you may (or may not) want to have, and share a lot of laughs along the way.
Gleb Bahmutov: No conflict, right? So 99% of the time I’m on a podcast - or even during Cypress workshops - I don’t try to sell you anything, really. I only talk about the open source tool. But then we needed something to actually earn money, so we tried robbing banks, forgery… It’s difficult to make a living, actually, especially in Atlanta and Boston. People are smart around these areas; I could never – anyway. But we decided “Okay, you run the tests, Cypress saves you a video of a test run, it takes screenshots on the failure, you have those test results… All of these test artifacts.” Well, what happens to them? If you are an individual developer or a small team, you can just store those test artifacts on your CI, on your Jenkins box, and if something fails, you can just look at those test artifacts, download the video, watch it on your machine, understand why the test failed, fix it, and so on. Absolutely fine.
But imagine you’re a business, and you’re running hundreds of tests. All the test artifact management, looking at videos, looking at screenshots, trying to understand which test is failing is difficult. So then we decided “Why don’t we build a business storing those test artifacts and allowing to easily view them, allowing you to look at the history, allowing you to see the history of a test? Which test is flaky, which test is slow?” and so on. All the things that don’t make much sense to small projects, but become hard problems in a large organization. And for large organizations - I mean, they can totally do it themselves. Some companies that use Cypress don’t pay us a dime, because they made their own dashboard, where they store results and look at them. But when you do that, you have to think “Money.” An hour of engineering time to just maintain the dashboard. If you spend one hour per week just maintaining or updating a dependency, or redeploying the dashboard, one hour a week four times a month - it probably is at least $150/$200. Just maintenance, not even writing it; just maintaining it.
[28:02] So what happens? Well, our cheapest price for storing your test artifacts and maintaining all that so you don’t have to do it is $99. If you record more, it’s $199, so $200, and then $300, and then there are custom plans for large enterprises.
It’s a no-brainer. If you’re a large organization and you want everyone to just see the test results whenever they want to, understand the test history, understand which tests are failing - no-brainer is to pay us. That’s how we make money, and that’s what allowed us to actually take those earnings and put them into working on open source full-time… Because what happened before was really interesting - we tried for a while to survive not by having a dashboard, but instead providing training and support, which is how many open source projects try to survive. And what happened there was really interesting - we were swamped. Every large project, we’d have so many issues, because we use it across every operating system, every CI provider, every hosting solution, every web framework, and library, you name it.
So obviously there were problems and we were trying to add features, but we could not. Half of the time we were supporting existing customers, which prevented us from having even time to actually solve a problem in the first place for everyone. So for us, moving away from private support towards a subscription model where we do test storage and dashboard for businesses allowed us to put money into solving everyone’s problems in the test runner itself. So I think it’s a really good balance, which we were lucky to discover.
Going back to Chris - the first page feature that actually allowed us to survive, that actually became why people wanted to pay money, was not even the test artifacts and the video; it was parallelization, running tests in parallel. So imagine you have many tests; if you run them one by one, you open the browser, you load things - it takes time. So once you have even a small set of end-to-end tests, imagine you’re waiting for half an hour for all of them to finish on a single continuous integration box.
So one night I was sitting around, playing with Mocha, because I love writing wrappers around Mocha - to randomize order of tests, the load balance, or repeat the same test and figure out flakes… So I was playing with Mocha and I was thinking “How cool would it be to have a little Mocha just sitting somewhere on the internet, waiting? Already running, just waiting for tests to run.” So you don’t have to spin the new thing, you’re just like “Hey, run this test.” You’re already there. “And then when you’re finished, come back; I’ll give you a second test to run.” Kind of like a little Mocha worker. And that was a success, this nighttime experiment.
Then we decided “Well, Cypress is nothing but, literally, a Mocha engine with controlling the browser.” So we’ve written this way of running Cypress with API, that knows all the tests; so every time Cypress finishes a spec, it goes back and says “Do you have another spec for me to run?” And the API can say “Yeah, here’s this next one. When you’re done, come back to me.” So each CI machine doesn’t just run all 30 spec files; instead, each CI machine goes to the API and says “I’m about to run 30 specs. Or should I do something else?” and now API says “wait ten machines already joining this group, for this particular test run; I really gave nine jobs, nine different spec files to nine machines. You are number ten. You run number ten spec.” And we all keep this history of test runs, so we can optimize the order, so it’s optimal… But it all became just like as an experiment around spinning and running tests as fast as possible. And that became a really sellable feature, because people pay for it. It makes sense for organizations with lots of spec files to run them faster, so that developers don’t wait for it.
[32:30] And again, test artifact storage and faster test runs, all for $99 a month? Sign me up. I wanna discuss it. [laughter] It makes such a good business sense… And I’ll tell you more. In previous organizations where I actually used Cypress, we could not use the dashboard, like external services. We were all about finance, so we kept everything in-house. We hosted our GitLab in GitLab CI, and everything. So we actually manage all this parallelization ourselves. Everytime you commit code, it would regenerate with CI files, so we’d spin our own boxes… We open sourced it as a tool; feel free to use it… But then I was like “Why? Why would I manage that? I can just concentrate on what I’m doing best, and not manage my CI server or my parallelization or my test artifacts.”
In conclusion, here’s my advice to anyone trying to make a business out of open source. As an individual contributor, you can probably support your project a little bit by providing private trainings, workshops, maybe support. And a good example of that is the maintainer of Redis. He just supports one company, gets paid a lot of money, and invests in open source work. But the thing is it only works for maybe one developer at a time, for smaller projects. As soon as you have a group of people, you need something else. Not just support, but additional services.
A good example would be Ghost, the blog editing. Open source project, but they host it for you, so you don’t have to do anything, and so you pay for that. So they provide extra service on top of it. If you can provide a service that makes sense for a company using your open source project, completely complementary; not light versus full version, but instead saying “If your company really loves this tool and uses it a lot, well, you will have this problem, and then we can provide a solution to that.” So all the individual users, smaller plan users can still do everything for free, and yet the companies who really are a big fan, they kind of self-select by being big fans, they can come back and say “Hey, I have these 100 things now. What can I do?” and you say “Well, we came up with a solution for this particular need. We know you’ll love us, because you have 1,000 tests… So for 1,000 tests, here’s a thing.”
So I think we are super-lucky… Because we were sitting around a table - it probably was a virtual table back in the day - and we were like “We know we have hit a home run with Cypress test runner, but we need to hit a second home run with the dashboard. What value can it provide to actually justify our existence, and paying for the whole team?” And that was really scary. We literally were like “Well, open source it. What can we do in the meanwhile?” It wasn’t like we knew what we will do with a dashboard. Parallelization came as an idea later. So it was really scary, but I’m glad we kind of got lucky.