Tooling Icon

Tooling

Tooling and apps used to create and deliver awesome software.
243 Stories
All Topics

Rust github.com

fclones – an efficient duplicate file finder and remover

Duplicates taking up tons of space on your home NAS? fclones quickly identifies duplicates, even when there’s 10s of thousands that you’re scanning over the network:

fclones treats your data seriously. You can inspect and modify the list of duplicate files before removing them. There is also a –dry-run option that can tell you exactly what changes on the file system would be made.

Also check out the algorithm used to detect duplicates.

Go github.com

jq but using Go instead

The second jq alternative we’ve discovered this week! (first here)

jq is hard to use. There are alternatives like fq and zq, but they still make you learn a new programming language. Im tired of learning new programming languages.

gq is not optimized for speed, flexibility or beauty. gq is optimized for minimal learning/quick usage. gq understands that you don’t use it constantly, you use it once a month and then forget about it. So when you come back to it, gq will be easy to relearn. Just use the builtin library just like you would any other go project and you’re done. No unfamiliar syntax or operations, or surprising limits. Thats it.

I don’t know if Go is a great fit for this use-case, but if you already know it well… makes sense.

JSON brimdata.io

Introducing zq: an easier (and faster) alternative to jq

If you’ve found the (excellent) jq tool for working with JSON a bit unwieldy… check out zq and see if you like its API any better. I wouldn’t put too much weight on the faster aspect, though:

We will cover zq’s performance in a future article, but to cut to the chase here, zq is almost always at least a bit faster than jq when processing JSON inputs

Almost always at least a bit faster is not something you’re likely to notice in practice.

Python bloomberg.github.io

Memray is a memory profiler for Python

From the engineering team at Bloomberg:

It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.

It has a lot of nice outputs so you can grok what’s going on.

Memray is a memory profiler for Python

Productivity xit.jotaen.net

A plaintext file format for todos and check lists

For now, [x]it! is merely a file format specification.

You don’t have to use a specific tool for .xit files, the basic operations like creating items or checking them off can be done in every text editor. Tools can make the experience more convenient, though, and provide support for common use cases.

The cool thing is anybody/everybody can now develop integrations for their favorite tools.

Max Howell Medium

Something new is brewing

Max Howell, creator of Homebrew, has gone back to his notes on brew2 to apply web3 concepts to help “distribute value to open source.” He’s calling this new brew tea.

Tools like Homebrew lie beneath all development tools, assisting developers to actually get development done. We know the graph of all open source, which means we’re uniquely placed to innovate in interesting and exciting ways. This is exactly what tea will do. We’re taking our knowledge of how to make development more efficient and throwing innovations nobody has ever really considered before.

With plans to move the package registry on-chain, Max lays out the numerous benefits due to “inherent benefits of blockchain technology”:

  • Packages will be immutable (no more left-pad incidents)
  • Packages will always be available (we’ll use decentralized storage)
  • Releases will be signed by the maintainers themselves (rather than a middleman you are told you can trust)
  • Tools can be built to fundamentally verify the integrity of your app’s open source constitution
  • Token can flow through the graph

Max says “token flowing is where things get really interesting,” and goes on to say “with our system people who care about the health of the open source ecosystem buy some token and stake it.”

(Thanks to Omri Gabay for sharing this first in our community Slack)

Tooling vermaden.wordpress.com

ZFS compatibility

ZFS has become very portable in recent years of its development, supporting six (6) operating systems: FreeBSD, Illumos, Linux, MacOS, NetBSD, and Windows. But what if you wanted to create a ZPool compatible with all of them? Which options and ZFS features should you choose?

If you haven’t yet, check out The Changelog #475 where I talk with Matt Ahrens (co-founder of the ZFS project) about making the ZFS file system.

Security trickster.dev

Decrypting your own HTTPS traffic with Wireshark

Wireshark is a seriously cool piece of software for packet sniffing and analysis. Why might you want to use it on yourself?

This opens up possibilities to not only reverse engineer web app private APIs in a deeper way, but also to do the same kind of research against desktop apps for purposes such as data scraping, automation, vulnerability research and privacy analysis.

Samuel Berthe github.com

A Lodash-style Go library using generics

Samuel Berthe:

This project have started as an experiment to discover generics implementation. It may look like Lodash in some aspects. I used to code with the awesome go-funk package, but it uses reflection and therefore is not typesafe.

As expected, benchmarks demonstrate that generics will be much faster than implementations based on reflect stdlib package. Benchmarks also shows similar performances to pure for loops.

Awesome Lists github.com

An awesome list of ngrok alternatives & other tunneling solutions

ngrok rocks, but it’s 2nd version is famously closed source*, so many open source alternatives have been developed over the years. So many, in fact, that they now have their own awesome list.

The purpose of this list is to track and compare tunneling solutions. This is primarily targeted toward self-hosters and developers who want to do things like exposing a local webserver via a public domain name, with automatic HTTPS, even if behind a NAT or other restricted network.

*We spoke with Alan Shreve about this decision back when he made it, if you’re curious about his thinking.

Cloud garagehq.deuxfleurs.fr

Garage - a self-hosted distributed object storage solution

Garage is a distributed storage solution, that automatically replicates your data on several servers. Garage takes into account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems.

It has an S3-compatible API and can be used as a storage backend for things like NextCloud, Matrix, and Mastodon. It’s being built by a non-profit in France that is “working to promote self-hosting and small-scale hosting.” Why do they do this?

self-hosting means running our own hardware at home, and providing 24/7 Internet services from there. We have many reasons for doing this. One is because this is the only way we can truly control who has access to our data. Another one is that it helps us be aware of the physical substrate of which the Internet is made: making the Internet run has an environmental cost which we want to evaluate and keep under control. The physical hardware also gives us a sense of community, calling to mind all of the people that could currently be connected and making use of our services, and reminding us of the purpose for which we are doing this.

Tooling freebsdfoundation.org

RAID-Z expansion feature for ZFS goes live

This major undertaking was discussed by Matt Ahrens on our recent ZFS episode. How it works:

The feature reflows existing data, essentially rewriting it onto a new arrangement of disks – meaning the original group plus a newly added disk. In so doing, a new adjacent chunk of free space is created at the end of the logical RAID-Z group and thus at the end of each physical disk.

With a disclaimer:

While all capabilities of this feature have been implemented and all tests so far have been passed, there are still a few loose ends to tie up. Specifically, there is some code cleanup to do, some verbose logging to remove, some code documentation to write, and similar relatively minor tasks. We aim for this to be integrated by Q3.

0:00 / 0:00