Vladimir Dementyev shares a deep dive into some necessary profiling he did on one of the Go services and how he finally figured out what was going wrong:

Here’s a question: how can we see what’s happening inside an arbitrary Go process? Or, more precisely, how can we see what all the goroutines are doing at any given moment? If we could crack that, it could help us figure out why they’re not processing our requests.