Tooling Icon

Tooling

Tooling and apps used to create and deliver awesome software.
275 Stories
All Topics

Tooling doordash.engineering

Filibuster – automating resilience testing of microservice applications

Christopher Meiklejohn has been working for two years on developing an automated resilience testing tool called Filibuster and was able to test it out at DoorDash during his research internship:

Access to DoorDash’s real-world industrial microservice application was extremely valuable both to ensure that Filibuster’s design matches how microservice applications are developed today and to influence future features that I would not have identified in a lab setting.

The results were pretty good! Maybe we won’t need chaos monkeys as tools like Filibuster advance?

In short, Filibuster can identify many of the technical resilience issues that exist in applications earlier, at the time of development, without testing in production as required in more traditional chaos engineering techniques.

Cloud github.com

Store files as YouTube videos == infinite disk space

YouTubeDrive is a Wolfram Language (aka Mathematica) package that encodes/decodes arbitrary data to/from simple RGB videos which are automatically uploaded to/downloaded from YouTube. Since YouTube imposes no limits on the total number or length of videos users can upload, this provides an effectively infinite but extremely slow form of file storage.

Filed under: ways-no-youtube-engineer-ever-imagined-people-would-use-their-software

Rust github.com

fclones – an efficient duplicate file finder and remover

Duplicates taking up tons of space on your home NAS? fclones quickly identifies duplicates, even when there’s 10s of thousands that you’re scanning over the network:

fclones treats your data seriously. You can inspect and modify the list of duplicate files before removing them. There is also a –dry-run option that can tell you exactly what changes on the file system would be made.

Also check out the algorithm used to detect duplicates.

Go github.com

jq but using Go instead

The second jq alternative we’ve discovered this week! (first here)

jq is hard to use. There are alternatives like fq and zq, but they still make you learn a new programming language. Im tired of learning new programming languages.

gq is not optimized for speed, flexibility or beauty. gq is optimized for minimal learning/quick usage. gq understands that you don’t use it constantly, you use it once a month and then forget about it. So when you come back to it, gq will be easy to relearn. Just use the builtin library just like you would any other go project and you’re done. No unfamiliar syntax or operations, or surprising limits. Thats it.

I don’t know if Go is a great fit for this use-case, but if you already know it well… makes sense.

JSON brimdata.io

Introducing zq: an easier (and faster) alternative to jq

If you’ve found the (excellent) jq tool for working with JSON a bit unwieldy… check out zq and see if you like its API any better. I wouldn’t put too much weight on the faster aspect, though:

We will cover zq’s performance in a future article, but to cut to the chase here, zq is almost always at least a bit faster than jq when processing JSON inputs

Almost always at least a bit faster is not something you’re likely to notice in practice.

Python bloomberg.github.io

Memray is a memory profiler for Python

From the engineering team at Bloomberg:

It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.

It has a lot of nice outputs so you can grok what’s going on.

Memray is a memory profiler for Python

Productivity xit.jotaen.net

A plaintext file format for todos and check lists

For now, [x]it! is merely a file format specification.

You don’t have to use a specific tool for .xit files, the basic operations like creating items or checking them off can be done in every text editor. Tools can make the experience more convenient, though, and provide support for common use cases.

The cool thing is anybody/everybody can now develop integrations for their favorite tools.

Max Howell Medium

Something new is brewing

Max Howell, creator of Homebrew, has gone back to his notes on brew2 to apply web3 concepts to help “distribute value to open source.” He’s calling this new brew tea.

Tools like Homebrew lie beneath all development tools, assisting developers to actually get development done. We know the graph of all open source, which means we’re uniquely placed to innovate in interesting and exciting ways. This is exactly what tea will do. We’re taking our knowledge of how to make development more efficient and throwing innovations nobody has ever really considered before.

With plans to move the package registry on-chain, Max lays out the numerous benefits due to “inherent benefits of blockchain technology”:

  • Packages will be immutable (no more left-pad incidents)
  • Packages will always be available (we’ll use decentralized storage)
  • Releases will be signed by the maintainers themselves (rather than a middleman you are told you can trust)
  • Tools can be built to fundamentally verify the integrity of your app’s open source constitution
  • Token can flow through the graph

Max says “token flowing is where things get really interesting,” and goes on to say “with our system people who care about the health of the open source ecosystem buy some token and stake it.”

(Thanks to Omri Gabay for sharing this first in our community Slack)

Open Source vermaden.wordpress.com

ZFS compatibility

ZFS has become very portable in recent years of its development, supporting six (6) operating systems: FreeBSD, Illumos, Linux, MacOS, NetBSD, and Windows. But what if you wanted to create a ZPool compatible with all of them? Which options and ZFS features should you choose?

If you haven’t yet, check out The Changelog #475 where I talk with Matt Ahrens (co-founder of the ZFS project) about making the ZFS file system.

Security trickster.dev

Decrypting your own HTTPS traffic with Wireshark

Wireshark is a seriously cool piece of software for packet sniffing and analysis. Why might you want to use it on yourself?

This opens up possibilities to not only reverse engineer web app private APIs in a deeper way, but also to do the same kind of research against desktop apps for purposes such as data scraping, automation, vulnerability research and privacy analysis.

0:00 / 0:00