Tooling Icon

Tooling

Tooling and apps used to create and deliver awesome software.
275 Stories
All Topics

Natasha Lekh crawlee.dev

Crawlee is a web scraping & browser automation library for Node.js

Here’s Natasha Lekh from Apify describing the project:

This project really is a culmination of 4 years of work trying to make the best library for web scraping in production. Web scraping is a very dynamic environment and what works today might not work tomorrow, so we at Apify had to go through a lot of trial and error to figure out the most reliable and convenient ways of crawling the web and scraping data. We hope that we finally cracked it and that now many developers will enjoy working with our new library and it will make their scrapers more reliable and time to production faster.

I like how it starts with simple HTTP-based scraping, but can switch to browser-based automation when a site has JavaScript rendering. I don’t love the built-in proxy rotation features. Not because they’re bad, per se, but because they make spammers lives easier…

Tooling about.maps.earth

Open source maps for everyone 🗺

maps.earth is a planet-scale installation of Headway, but you can easily set up your own server on a smaller scale for your own personal use by running just a few commands.

This is the first I’ve heard of Headway, but I love their mission:

The Headway Project was born out of a frustration with the need to send current and future location data to a corporation in order to figure out how we’re getting from A to B. Offline-only maps apps can be frustrating to use, and sometimes suffer from performance problems, poor data coverage, or other technical limitations.

Headway aims to remedy these issues by bundling industry-standard software into a web app that’s easy to set up for yourself or your friends. No need to send your location data to anyone you don’t trust, not even maps.earth.

Cloud resoto.com

Resoto is a meta layer on top of your cloud infra

As best I can tell, this provides simplified search across your infra, generates reports so you can easily audit resource usage, and lets you create/trigger jobs such as cleaning up unused resources and enforcing tag structures.

The search looks pretty powerful and you can pipe search results directly to jobs for quick processing:

search is(resource) and tags.owner==null | tag update owner "John Doe"

Medium Icon Medium

Fzf: a tool that will transform your CLI life

Suraj Pillai, singing fzf’s praises:

I’m a CLI junkie, addicted to Vim motions, and never miss an opportunity to bring those two in to any part of my workflow. Naturally, I love to geek out about command line utilities and am always on the lookout for the next tool that can improve my productivity or just make CLI more fun to use. I can confidently say that Fzf is one of the handful of tools I’ve discovered over the years that has done both and has,thus, significantly improved the quality of my command line life.

Go github.com

Comcast simulates trash network connections so you can build better systems

Testing distributed systems under hard failures like network partitions and instance termination is critical, but it’s also important we test them under less catastrophic conditions because this is what they most often experience. Comcast is a tool designed to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets.

It works by wrapping up some system tools in a portable(ish) way. On BSD-derived systems such as OSX, we use tools like ipfw and pfctl to inject failure. On Linux, we use iptables and tc. Comcast is merely a thin wrapper around these controls.

TFW you come up with the perfect name for your open source project ✨

Awesome Lists github.com

A powerful open source toolkit for hackers & security automation

Scanners Box also known as scanbox, is a powerful hacker toolkit, which has collected more than 10 categories of open source scanners from Github, including subdomain, database, middleware and other modular design scanner etc. But for other Well-known scanning tools, such as nmap, w3af, brakeman, arachni, nikto, metasploit, aircrack-ng will not be included in the scope of collection.

Toolkit might be a bit misleading. I was imagning some kind of Docker container or Linux distro with all the tools baked in. This is more of a collection of tools (which is why we applied the Awesome topic to it) that you can pick and choose from. Nice collection, though!

JavaScript bun.sh

Bun is a fast all-in-one JavaScript runtime

Bundle, transpile, install and run JavaScript & TypeScript projects — all in Bun. Bun is a new JavaScript runtime with a native bundler, transpiler, task runner and npm client built-in.

Bun is competing with Node and Deno, with the following goals:

  1. Start fast (it has the edge in mind).
  2. New levels of performance (extending JavaScriptCore, the engine).
  3. Being a great and complete tool (bundler, transpiler, package manager).

I like those goals, and it’s designed to be a drop-in replacement for your current runtime.

Tooling prql-lang.org

PRQL is a modern language for transforming data

The P in PRQL (pronounced “Prequel”) stands for Pipelined, which I’m convinced is a great way of writing and reasoning about queries:

A PRQL query is a linear pipeline of transformations

Each line of the query is a transformation of the previous line’s result. This makes it easy to read, and simple to write.

It compiles to SQL, which means it’s compatible with most databases already and there are currently bindings for Python, JS & Rust, which is the compiler itself.

Try it out in their web-based playground. (Thanks, Wasm!)

Brave github.com

Brave Search Goggles

Goggles enable anyone, be it individuals or a community, to alter the ranking of Brave search by using a set of instructions (rules and filters). Anyone can create, apply, or extend a Goggle. Essentially Goggles act as a custom re-ranking on top of Brave’s search index.

This could be really cool! A few examples use cases:

  • No Pinterest - Rerank results to remove pages / threads hosted on Pinterest.
  • Rust programming - Rerank results to boost content related to the Rust programming language.
  • Hacker News / 1k short – Prioritizes domains popular with the Hacker News community, minus those that would rank among the top 1000 most-viewed websites.

Martin Heinz martinheinz.dev

Data and system visualization tools to boost your productivity

As files, datasets and configurations grow, it gets increasingly difficult to navigate them. There are however many tools out there, that can help you to be more productive when dealing with large JSON and YAML files, complicated regular expressions, confusing SQL database relationships, complex development environments and many others.

Security github.com

Chain-bench – a tool for auditing your software supply chain

Chain-bench is an open source tool for auditing your software supply chain stack for security compliance based on a new CIS Software Supply Chain benchmark.

You can run the tool from a CLI, assuming your code is hosted on GitHub (more SCM hosts coming soon):

chain-bench scan --repository-url <REPOSITORY_URL> --access-token <TOKEN> -o <OUTPUT_PATH>

I couldn’t find a comprehensive list of what checks are in the benchmark, but it appears they are referring to this guide. You can see what an example run’s results like like in the README.

Tooling doordash.engineering

Filibuster – automating resilience testing of microservice applications

Christopher Meiklejohn has been working for two years on developing an automated resilience testing tool called Filibuster and was able to test it out at DoorDash during his research internship:

Access to DoorDash’s real-world industrial microservice application was extremely valuable both to ensure that Filibuster’s design matches how microservice applications are developed today and to influence future features that I would not have identified in a lab setting.

The results were pretty good! Maybe we won’t need chaos monkeys as tools like Filibuster advance?

In short, Filibuster can identify many of the technical resilience issues that exist in applications earlier, at the time of development, without testing in production as required in more traditional chaos engineering techniques.

0:00 / 0:00