This hit my radar a few times in the past year – this week, and then most recently a few months back when we had Richard Hipp on The Changelog (again) – but, I didn’t post it to the newsfeed.

Here’s what’s interesting about the SQLite test suite – it’s their secret sauce…the sustaining enablement of building a support business around SQLite. Here’s a direct quote from Richard Hipp in that episode.

Originally we thought we were gonna sell this and make money from it, and that’s how we were gonna support ongoing development. That didn’t really play out, nobody ever bought it. It does sort of become our business value, our intellectual property. I mean, you can take the SQLite code and fork it and start your own thing…but you don’t have the full test suite. You’ve got a lot of tests, but not all of them. So we’ve got a little bit of advantage over you there.

Click here to play that episode from this quote.

Jacob Kaplan-Moss, who has been writing a lot about good interview questions and how to hire well:

Work sample tests are an exercise, a simulation, a small slice of real day-to-day work that we ask candidates to perform. They’re practical, hands-on, and very close or even identical to actual tasks the person would perform if hired. They’re also small, constrained, and simplified enough to be fair to include in a job selection process.

To give you a more concrete idea of what I’m talking about, here are several examples of work sample tests I’ve used…

And just in case you think he’s prescribing whiteboarding…

However, work sample tests are also a minefield: the space is littered with silly practices like whiteboarding, FizzBuzz, Leetcode, and “reverse a linked list”-style bullshit. The point of this series is to separate these silly practices from the good ones and to give you a framework and several examples to use in your hiring rounds.

Property-based tests give us more confidence in our code. They’re great at catching edge-cases we may not have thought of otherwise. But this confidence comes at a cost. Property tests take more effort to write. They force you to think hard about what the code is doing, and what its expected behaviour should be. It’s hard work. And on top of that, running 100+ tests is always going to take longer than running 3–5 example-based tests. This cost is real, and it raises the question:

How do we keep ourselves from over-specifying or writing unnecessary tests?

Property-based testing helps us write better tests, with less code, and greater coverage. This leads to more confidence in our code, and fewer bugs in our applications. But, as always, there’s a price. Property tests take more effort to write, and they take longer to run. But they’re worth it.

Recently I discovered additional reasons to write tests for my code! Besides obvious and well-known ones described in books and many blog posts, I realized that the process of writing tests has also psychological effects for me. These effects are especially helpful in cases of hotfixes that need to be deployed as soon as possible.

This week Gerhard is joined by Justin Searls, Test Double co-founder and CTO. Also a 🐞 magnet. They talk about how to deal with the pressure of shipping faster, why you should optimize for smoothness not speed, and why focusing on consistency is key. Understanding the real why behind what you do is also important. There’s a lot more to it, as its a nuanced and complex discussion, and well worth your time.

Expect a decade of learnings compressed into one hour, as well as disagreements on some ops and infrastructure topics — all good fun. In the show notes, you will find Gerhard’s favorite conference talks Justin gave a few years back.

The top 1% of companies spend thousands of hours building their own A/B testing platforms in-house. The other 99% are left paying for expensive 3rd party SaaS tools or hacking together unmaintained open source libraries.

Growth Book gives you the flexibility and power of a fully-featured in-house A/B testing platform without needing to build it yourself.

Rainer Hahnekamp:

On 24th April 2021, Angular announced the deprecation of their E2E testing tool protractor. It was unclear if there will be a successor or if Angular delegates this to its users. In this article, I will provide a short overview over the differences between the various E2E frameworks, and argue why you should use Cypress.

I’ve heard nothing but high praise of Cypress and thoroughly enjoyed our conversation with Gleb Bahmutov when he joined us on JS Party.

By adding visual elements to your tests, you can gain more options to add meaningful ways in maintaining a high level of quality for your app. Colby Fayock explains how.

This article pairs nicely with what Angie Jones had to say about visual testing on Our recent JS Party episode.

Fuzzing is coming to the standard library. We speak to Katie Hockman and Jay Conrod who were part of the team responsible for designing and implementing it. We dig into the details, hear some best practices, where fuzzing can help your code, and learn more about how it works.

This week we chat with Angie Jones about all things testing. We’ll cover unit testing, visual testing, end-to-end testing, and more!

We discuss how Test Driven Development (TDD) can help you write better code, and build better software. Packed with tips and tricks, gotchas and best practices, the panel explore the subject and share their real-world experiences.

In this article, I’d like to explore a couple of implementations of sorting algorithms. However, I’d like to do so driven by unit tests. The examples are written in Go, but don’t worry if you never worked with Go before. The emphasis here is on the journey and the joy of building solutions guided by tests!

We first talked fuzzing with Katie Hockman back in August of 2020. Fast-forward 10 months and native fuzzing in Go is ready for beta testing! Here’s Katie explaining fuzzing, for the uninitiated:

Fuzzing is a type of automated testing which continuously manipulates inputs to a program to find issues such as panics or bugs. These semi-random data mutations can discover new code coverage that existing unit tests may miss, and uncover edge case bugs which would otherwise go unnoticed. Since fuzzing can reach these edge cases, fuzz testing is particularly valuable for finding security exploits and vulnerabilities.

It looks like the feature won’t be landing in Go 1.17, but they’re planning on it sometime after that. Either way, you can use fuzzing today on its development branch.

“The tests are timing out again!”, someone yells. “Alright I’ll bump them”, you instinctively respond. Then you pause and feel uneasy. Is there another way?

In this blog post, I share my growing disconnect with code-coverage and unit-testing. I then detail the method I’ve been using for the greater part of 7 years and how it still allows me to preach at length that being correct is the single most important thing for a developer.

Testing can be hard, how to test, where to test, what is a good test? All questions that can be deceptively difficult to answer. In this episode we talk about the trials and tribulations of testing and why it can be argued to be especially difficult in Go.

Here’s a pretty useful idea for library authors and their users: there are better ways to test your code!

I give three examples of how user projects can be self-tested without actually writing any real test cases by the end-user. One is hypothetical about django and two examples are real and working: featuring deal and dry-python/returns. A brief example with deal:

import deal

@deal.pre(lambda a, b: a >= 0 and b >= 0)
@deal.raises(ZeroDivisionError)  # this function can raise if `b=0`, it is ok
def div(a: int, b: int) -> float:
    if a > 50:  # Custom, in real life this would be a bug in our logic:
        raise Exception('Oh no! Bug happened!')
    return a / b

This bug can be automatically found by writing a single line of test code: test_div = deal.cases(div). As easy as it gets! From this article you will learn:

• How to use property-based testing on the next level
• How a simple decorator @deal.pre(lambda a, b: a >= 0 and b >= 0) can help you to generate hundreds of test cases with almost no effort
• What “Monad laws as values” is all about and how dry-python/returns helps its users to build their own monads

I really like this idea! And I would appreciate your feedback on it.

Benjamin Coe joins Amal and Divya to discuss his wide-ranging open source projects, test coverage with Istanbul, and the future of testing in JavaScript.

Alabe Duarte shares his personal exposure with TDD over the years. This includes:

1. why he believes TDD is important
2. the subjectiveness of “good design”
3. when doing TDD doesn’t help

Amal and Divya turn our spotlight inward and interview our very own Christopher “Boneskull” Hiller about maintaining Mocha.js. Mocha has been a mainstay in the JavaScript testing community for ten (!) years now! They discuss the secret to Mocha’s success, what it’s like to maintain it, and how to make maintainers (and users) happy!

Mat Ryer hosts a spectacular panel with expert debuggers Derek Parker, Grant Seltzer Richman, and Hana Kim from the Go Team. Let’s face it, even the best-intended code doesn’t always do what you want it to. What’s a Gopher to do? Listen to this, that’s what!

Devon C. Estes:

It’s fairly common for folks who haven’t used mutation testing before to not immediately see the value in the practice. Mutation testing is, after all, still a fairly niche and under-used tool in the average software development team’s toolbox. So today I’m going to show a few specific types of very common problems that mutation testing is great at finding for us, and that are hard or impossible to find with other methods

He goes on to detail the “multiple execution paths on a single line” problem, the “untested side effect” problem, and the “missing pin” problem.

Typically, people say that testing is like a pyramid. A wide base of unit tests and very few end-to-end tests. I believe we’ve come to a point where a crab strategy is a better approach.

Gleb Bahmutov, PhD joins the show for a fun conversation around end-to-end testing. We get the skinny on Cypress, find out how it’s structured as both an open source library and a SaaS business, tease apart the various types of tests you may (or may not) want to have, and share a lot of laughs along the way.

Justin Searls from Test Double joins the party to talk about patterns he’s identified that lead to failure, minimalism, and of course, testing!