This week we’re talking with Nick Janetakis about modern unix tools, and the various commands, tooling, and ways we use the commmand line. Do you Bash or Zsh? Do you use cat or bat? What about man vs tldr? Today’s show is a deep dive into unix tools you know and love, or should know and maybe love.

Angle-grinder allows you to parse, aggregate, sum, average, min/max, percentile, and sort your data. You can see it, live-updating, in your terminal. Angle grinder is designed for when, for whatever reason, you don’t have your data in graphite/honeycomb/kibana/sumologic/splunk/etc. but still want to be able to do sophisticated analytics.

Angle grinder can process well above 1M rows per second (simple pipelines as high as 5M), so it’s usable for fairly meaty aggregation. The results will live update in your terminal as data is processed. Angle grinder is a bare bones functional programming language coupled with a pretty terminal UI.

I’m not gonna lie, they had me with the name on this one.

Difftastic is an experimental structured diff tool that compares files based on their syntax. It is very much unfinished. It works reasonably on very parenthesised data (lisps, JSON), it works sometimes on other languages with sufficient parentheses (Rust, JS), and falls back to a line-oriented diff otherwise.

A bad CLI can easily discourage users from interacting with it. Building successful CLIs requires attention to detail and empathy for the user in order to create a good user experience. It is very easy to get wrong.

In this guide I have compiled a list of best practices across areas of focus which aim to optimize for an ideal user experience when interacting with a CLI application.

You’ve probably seen many of these commands hit Changelog News over the years, but now you can see them all again in one hand-curated place. Who knows, maybe one or two will be new to you. I hadn’t heard of curlie previously, which looks like a nice merging of cur and httpie.

This isn’t merely a speech-to-text thing. It also provides intent recognition, which makes it great for doing voice commands. For example, when trained with this template, the following command:

$ voice2json transcribe-wav \
      < turn-on-the-light.wav | \
      voice2json recognize-intent | \
      jq .

Produces this JSON event:

{
    "text": "turn on the light",
    "intent": {
        "name": "LightState"
    },
    "slots": {
        "state": "on"
    }
}

And it can be retrained quickly enough to do it at runtime. Cool stuff!

rpg-cli is a bare-bones JRPG-inspired terminal game written in Rust. It can work as an alternative to cd where you randomly encounter enemies as you change directories.

You’ll want to practice a bit first, then once you get good at it go ahead and override the builtin cd by adding this function to your bash profile.

cd () {
   rpg-cli "$@"
   builtin cd "$(rpg-cli --pwd)"
}

Bash is great, but when it comes to writing scripts, people usually choose a more convenient programming language. JavaScript is a perfect choice, but standard Node.js library requires additional hassle before using. zx package provides useful wrappers around child_process, escapes arguments and gives sensible defaults.

I wouldn’t say JavaScript is a perfect choice for this kind of scripting, but it’s definitely a suitable one (especially if it’s the language you already know well). Here’s what scripting looks like with zx:

#!/usr/bin/env zx

await $`cat package.json | grep name`

let branch = await $`git branch --show-current`
await $`dep deploy --branch=${branch}`

await Promise.all([
  $`sleep 1; echo 1`,
  $`sleep 2; echo 2`,
  $`sleep 3; echo 3`,
])

let name = 'foo bar'
await $`mkdir /tmp/${name}`

Top-level await sure makes things nicer. (Deno supports this out of the box, btw.)

Thanks to Muhammad Raza for sharing some of his favorite command line one liners to make his workflow productive and more efficient.

As a regular Linux user, I have been using commandline extensively to perform daily tasks such as creating files, navigating through directories, moving files and editing files using vim.

You can think of Julia Evans’ new dnspeep tool as similar to tcpdump but specifically for watching your machine’s DNS queries.

One thing I like about this tool is that it gives me a sense for what programs on my computer are using the Internet! For example, I found out that something on my computer is making requests to ping.manjaro.org from time to time for some reason, probably to check I’m connected to the internet.

A friend of mine actually discovered using this tool that he had some corporate monitoring software installed on his computer from an old job that he’d forgotten to uninstall, so you might even find something you want to remove.

It also probably comes in handy when debugging those pesky “could it be DNS?” issues, but this might be a limitation on that front:

One thing this program doesn’t do is tell you which process made the DNS query, there’s a tool called dnssnoop I found that does that. It uses eBPF and it looks cool but I haven’t tried it.

This doesn’t aim to entirely replace find and ls, but if you already know SQL (like many of us do), why not be able to leverage that knowledge for your more advanced file-finding needs? Here’s a couple of examples so you get the idea:

Find temporary or config files (full path and size):

fselect size, path from /home/user where name = '*.cfg' or name = '*.tmp'

Use aggregate functions:

fselect "MIN(size), MAX(size), AVG(size), SUM(size), COUNT(*) from /home/user/Downloads"

Find by date and time intervals:

fselect path from /home/user where modified gte 2017-05-01

grex is a library as well as a command-line utility that is meant to simplify the often complicated and tedious task of creating regular expressions. It does so by automatically generating a single regular expression from user-provided test cases. The resulting expression is guaranteed to match the test cases which it was generated from.

Unfortunately, the tool’s authors still think you need learn how to write regexes, even when grex works flawlessly (and I tend to agree with them).

monsoon is a so-called command-line HTTP enumerator: A tool that iterates over a list of values, for example a word list or a range of integers, and sends one HTTP request per item towards a given server.

The team behind monsoon enumerated some common examples in their introductory blog post.

jq is a hugely useful tool for anyone dealing with JSON of varying shapes and sizes. If that’s you, but you haven’t given jq a serious try, this is a great little primer on its use and use cases.

Matthew Monahan:

dug is designed to help you check the global status of your DNS records. You can use the built in servers, update them from remote or local sources, specify servers, whatever. It also supports templated output in CSV or JSON for use in monitoring applications or just piping the results around.

Crush is an attempt to make a traditional command line shell that is also a modern programming language. It has the features one would expect from a modern programming language like a type system, closures and lexical scoping, but with a syntax geared toward both batch and interactive shell usage.

Check out the overview of features right here.

From the foreword:

Most people today don’t know what the command line is, much less why they would want to bother with it. As computing pioneer Alan Kay said in a 2017 interview, “Because people don’t understand what computing is about, they think they have it in the iPhone, and that illusion is as bad as the illusion that ‘Guitar Hero’ is the same as a real guitar.”

Off to a good start…

Inspired by traditional UNIX philosophy, driven by an interest in encouraging a more delightful and accessible CLI environment, and guided by our experiences as programmers, we decided it was time to revisit the best practices and design principles for building command-line programs.

Used to be written in Python, but is now 100% Rust.

It was designed to help remind *nix system administrators of options for commands that they use frequently, but not frequently enough to remember.

Let’s imagine a completely hypothetical world where it’s the umpteenth time you’ve used it, but you still can’t remember which flags to send to tar… so you run:

cheat tar

You’ll be greeted by:

# To extract an uncompressed archive:
tar -xvf '/path/to/foo.tar'

# To extract a .gz archive:
tar -xzvf '/path/to/foo.tgz'

# To create a .gz archive:
tar -czvf '/path/to/foo.tgz' '/path/to/foo/'

# To extract a .bz2 archive:
tar -xjvf '/path/to/foo.tgz'

# To create a .bz2 archive:
tar -cjvf '/path/to/foo.tgz' '/path/to/foo/'

The cheatsheets themselves are community-sourced.

dog is a command-line DNS client, like dig.
It has colourful output, understands normal command-line argument syntax, supports the DNS-over-TLS and DNS-over-HTTPS protocols, and can emit JSON.

The only downside is that (for now) you have to build it from source if you want to use it.

In this episode we discuss Mislav’s experience building not one, but two Github CLIs - hub and gh. We dive into questions like, “What lead to the decision to completely rewrite the CLI in Go?”, “How were you testing the CLI, especially during the transition?”, and “What Go libraries are you using to build your CLI?”

ffmpeg is an incredibly powerful tool, but its many flags and options make it not the easiest thing to wield (especially if you use it just infrequently enough to forget the magic syntax you ginned up last time).

vdx makes ffmpeg more approachable for many of the common video processing operations you may need on a regular basis. Examples!

$ vdx '*.mov' --crop=360,640    # Crop to width 360, height 640
$ vdx '*.mov' --format=gif      # Convert to GIF
$ vdx '*.mov' --fps=12          # Change the frame rate to 12
$ vdx '*.mov' --no-audio        # Strip audio
$ vdx '*.mov' --resize=360,-1   # Resize to width 360, maintaining aspect ratio
$ vdx '*.mov' --reverse         # Reverse
$ vdx '*.mov' --rotate=90       # Rotate 90 degrees clockwise
$ vdx '*.mov' --speed=2         # Double the speed
$ vdx '*.mov' --trim=0:05,0:10  # Trim from time 0:05 to 0:10
$ vdx '*.mov' --volume=0.5      # Halve the volume

Command line tool to extract the main content from a webpage, as done by the “Reader View” feature of most modern browsers. It’s intended to be used with terminal RSS readers, to make the articles more readable on web browsers such as lynx. The code is closely adapted from the Firefox version and the output is expected to be mostly equivalent.

I could see this fitting in nicely in a pipeline between curl and, well, lots of other commands.

Plaintext storage, git-backed, Markdown formatting, and all in a single script. 👌

Ruby is my favorite tool for slightly-longer-than-one-liners, but I don’t often reach for it directly from the command line. This little cookbook might change my mind on that:

A shell utility like bash provides built-in commands and scripting features to make it easier to solve and automate various tasks. External *nix commands like grep, sed, awk, sort, find, parallel etc can be combined to work with each other. Depending upon your familiarity with those tools, you can either use ruby as a single replacement or complement them for specific use cases.