Testing Icon

Testing

Testing is the practice of systematically checking if code functions as intended.
37 Stories
All Topics

Clojure github.com

A testing library that turns your README into executable tests

This is a very cool idea coming out of the Clojure community. I dig it because the examples in your README are guaranteed to never become stale as your project evolves.

It works by parsing your README and looking for executable code samples with expected outputs. For each one it finds, it generates a test ensuring that executing the code produces the output.

There are, as you might expect, caveats.

Bash github.com

Dead simple testing framework for bash with coverage reporting

critic.sh exposes high level functions for testing consistent with other frameworks and a set of built in assertions. One of my most important goals was to be able to pass in any shell expression to the _test and _assert methods, so that one is not limited to the built-ins.

The coverage reporting is currently rudimentary, but it does indicate which lines haven’t been covered. It works by running the tests with extended debugging, redirecting the trace output to a log file, and then parsing it to determine which functions/lines have been executed. It can definitely be improved!

See a demo of critic.sh in action on asciinema 📽️

Practical AI Practical AI #74

Testing ML systems

Production ML systems include more than just the model. In these complicated systems, how do you ensure quality over time, especially when you are constantly updating your infrastructure, data and models? Tania Allard joins us to discuss the ins and outs of testing ML systems. Among other things, she presents a simple formula that helps you score your progress towards a robust system and identify problem areas.

Kubernetes github.com

A chaos engineering platform for Kubernetes

Chaos Mesh is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments. At the current stage, it has the following components:

  • Chaos Operator: the core component for chaos orchestration. Fully open sourced.
  • Chaos Dashboard: a visualized panel that shows the impacts of chaos experiments on the online services of the system; under development; curently only supports chaos experiments on TiDB(https://github.com/pingcap/tidb).

For the uninitiated, chaos engineering is when you unleash havoc on your system to prove out its resiliency (or lack thereof).

A chaos engineering platform for Kubernetes

Kent Beck Increment

Testing the boundaries of collaboration

Kent Beck, for Increment:

It’s 2030. A programmer in Lagos extracts a helper method. Seconds later, the code of every developer working on the program around the world updates to reflect the change. Seconds later, each of the thousands of servers running the software updates. Seconds later, the device in my pocket in Berlin updates, along with hundreds of millions of other devices across the globe.

Perhaps the most absurd assumption in this story is that I’ll still have a pocket in 10 years.

Ned Batchelder nedbatchelder.com

Why your mock doesn’t work

Mocking is a powerful technique for isolating tests from undesired interactions among components. But often people find their mock isn’t taking effect, and it’s not clear why. Hopefully this explanation will clear things up.

Mocking isn’t always the best test isolation technique, but if/when you use it, you might as well use it correctly. Ned’s here to help you do just that.

Testing nngroup.com

Why you only need to test with 5 users

Some people think that usability is very costly and complex and that user tests should be reserved for the rare web design project with a huge budget and a lavish time schedule. Not true. Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.

This article is from the year 2000 (queue Conan O’Brien’s side kick), but it’s filled with timeless goodies. Its conclusions are a straight forward example of diminishing returns, but worth reading how they arrived at them from empirical evidence.

Go Time Go Time #91

if err != nil

Mat and Carmen along with guest panelists Dave Cheney, Peter Bourgon, and Marcel van Lohuizen discuss errors in Go, including the new try proposal. Many questions get answered…What do we think about how errors work in Go? How is it different from other languages/approaches? What do/don’t we like? What don’t we like? How do we handle errors these days? What’s going on with the try proposal?

Python hypothesis.works

Hypothesis seeks to automate your test process

This interesting testing tool was pointed out to me by Ned Batchelder when he was on The Changelog.

It combines human understanding of your problem domain with machine intelligence to improve the quality of your testing process while spending less time writing tests.

At its core, Hypothesis is a modern implementation of property based testing, which came out of the Haskell world 20 (!) years ago.

Hypothesis runs your tests against a much wider range of scenarios than a human tester could, finding edge cases in your code that you would otherwise have missed. It then turns them into simple and easy to understand failures that save you time and money compared to fixing them if they slipped through the cracks and a user had run into them instead.

Kevin Ball DEV.to

Let’s talk testing: 4 quick lessons on the philosophy of testing

Inspired by JSParty #70, 4 quick lessons on the philosophy of testing. The motivation?

Tools like Mocha, Jasmine and Jest have made writing tests far easier… But there’s still a gap. It’s extremely hard to find information on the philosophy of testing. What to test and why. How much is enough? What type of tests should I be writing, and when does it fit into my process?

Shlomo Kraus github.com

Mockshot – automatic mock generation from snapshot tests

We made a silly joke on Twitter yesterday (this is what Twitter is for, no?) about test doubles and that unfortunate moment when they inevitably surprise you.

This prompted Shlomo Kraus to reach out and tell us about Mockshot. In brief:

Imagine you could:

  1. Never manually write a mock again
  2. Have a guarantee that your mocks are always valid

Sounds nice! It works by using Jest’s snapshot tests output to generate mocks to be used in other tests.

This is purposeful coupling, which seems like it could backfire in the long-run. However, the team behind the library has been using it for over a year and are still singing its praises. For more on their experience creating and using it, read this.

 Itamar Turner-Trauring pythonspeed.com

10× faster database tests with Docker

Testing code that talks to the database can be slow. Fakes are fast but unrealistic. What to do? With a little help from Docker, you can write tests that run fast, use the real database, are easy to write and run.

I tried Itamar’s technique on changelog.com’s test suite and the 679 tests complete in ~17 seconds. The same tests run directly against Postgres complete in ~12 seconds.

A net loss for me, but that may have something to do with how Docker for Mac works? I’d love to hear other people’s experiences.

Tanya Janca medium.com

Security bugs are fundamentally different than quality bugs

Tanya Janca compares and contrasts quality bugs and security bugs, arguing that they’re quite different and should be treated differently. This logic resonates with me and she has a lot of insights to share along the way. I particularly enjoyed this bit:

You cannot have a high-quality product that is insecure; it is an oxymoron. If an application is fast, beautiful and does everything the client asked for, but someone breaks into the first day that it is released, I don’t think you will find anyone willing to call it a high-quality application.

A good read all the way through to the end. 👍

Nikita Sobolev DEV.to

Best engineering practices: how to fix a bug?

This is a great article that covers the 🐛 gamut:

  1. spotting bugs
  2. reporting bugs
  3. reproducing bugs
  4. fixing bugs

I love the “lifehack” snippets Nikita sprinkles in as well. Like this little gem right here:

Lifehack: sometimes you might want to submit a broken code to your branch so it will trigger a CI build. After the build, it will be saved in your project. And your colleagues will be able to link to this problem. Your next commit will have to solve the issue.

Typicode jsonplaceholder.typicode.com

A fake online REST API for testing and prototyping

JSONPlaceholder is a free online REST API that you can use whenever you need some fake data. It’s great for tutorials, testing new libraries, sharing code examples, …

It comes with a set of 6 common resources. You know, the usual suspects like /posts and /comments. Prefer to use your own data? The whole thing is powered by json-server, which will get you up and running in 30 seconds-ish.

0:00 / 0:00