Community

Short-form technical ramblings. Formerly VP of Architecture at Community.com, I'm Member of Technical staff at Mozi and co-author of "Docker: Up and Running" from O'Reilly Media.

Activity

I've been writing a new Event archiving service for the new event-based environment at my new gig. This is a second chance to iterate on what we built at Community that I talked about in my blog post on the subject. This time, I'm writing in #Golang because I'm on my own on this stuff at the moment. I've taken a different approach with it by archiving events to a local copy of #DuckDB as they come off the wire. Then I use DuckDB's native Parquet and S3 support to write the events batches out to S3, where they can then be queried with Athena.

This approach seems to be good so far. I will learn more when I get it into production. I feel another blog post coming later this year...

Working on CI builds is always so painful. I prefer getting them running under Compose locally and making the remote build just do the same thing. This is better. But the turnaround times are still so long.

Leslie Lamport, of LaTeX fame, is a very accomplished mathematician and computer scientist with a Turing award for his work on “fundamental contributions to the theory and
practice of distributed and concurrent systems”. He just published a draft of his new book:

"A science of concurrent programs"

https://lamport.azurewebsites.net/tla/science.pdf

True to his pedagogic approach to everything he does, "The book assumes only that you know the math one learns before entering a university." Even the appendices are fantastic. Can only wish I'll remain this lucid at his 82 years old.

#multithreading #maths #LesleyLamport

Finally, a sensible approach to multiple #AWS accounts in #Terraform with the same configs, without wrappers or external tools...

This is nice: https://cloudcity.io/blog/2023/01/12/Handling-Multiple-Separate-Accounts-With-Terraform/

Whatever anyone says about impact on performance, running Go pprof in your production app is a huge source of wins. I have over time made many corrections to CPU and memory performance because of real data from profiling in production. These have hugely outweighed the perf impact of ongoing profiling.

one thing ktistec related that i haven't had the time for is working on build and deployment tools. there are a bunch of outstanding requests—and a few PRs—for docker builds, packaged deployments for various hosting environments, etc.

if you're interested in contributing, let me know. you only have to agree to maintain them—i won't be able to.

#ktistec

We're finally moving off of AmazonMQ with its high cost and terrible performance, onto self-hosted #RabbitMQ on Kubernetes.

Wins:
1. MUCH faster, support for quorum queues
2. Less than 1/5 the cost
3. Can run latest RabbitMQ
4. Better I/O, network
5. Can tune it however we need to for our use case

Goodbye expensive, slow, terrible AmazonMQ! Major props to team members Dan Pilch and João Britto for making it happen!

I'm making note of this "Container permission denied: How to diagnose this error" article, as I'm sure I'll run into this more, and I'm really wishing that Linux reliably reported somewhere when EPERM happens (see the author's linked FriendlyEPERM feature proposal from a decade ago): https://www.redhat.com/sysadmin/container-permission-denied-errors

And if you wonder why I care about this, see: https://blog.gregor.com/designing-for-failure-88be805de1ac

There is obviously some kind of security contest going on right now to open PRs to fix security issues, because I'm getting very unhelpful, clearly automated, PRs opened on some of my projects.

Well, JFrog apparently shut down their free tier. It was pretty crummy, treated by them as a demo and not a real free tier. But I had some images hosted there and have now moved them. The lack of notice and the way they responded to my earlier feedback about the service means that I will not be using them for anything any time soon.