SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable. A new SQL engine - DuckDB - makes SQL competitive with other high performance dataframe libraries, making SQL a good candidate for data of all sizes.
You can make a similar argument for SQL that Gary Bernhardt made for Vim. Here’s Gary on Vim, run your own
s/Vim/SQL/g filter on this as you read it:
…just for me, 15 years; at the beginning of that time, TextMate was just becoming popular. Then it was Sublime Text was cool. Then Atom was cool. Then VS Code was cool. A lot of people switched between two of those, three of those, maybe all four of those, and that whole time I was just getting better and better and better at Vim… And you multiply that out by the length of a career, you use Vim for 40 years - you’re gonna be so good at it by the end, and it’s still gonna be totally relevant, I think.