Over the past 48 hours I have been glued to my screen trying to figure out how to make Beehaw more robust during this Reddit exodus.
My eyes are burning but I am thankful for so much financial support as well as the work of two sysadmins that have given us all a breath of fresh air.
One of the sysadmins was up until 2:30 am helping us out as a volunteer. I am so very grateful for persons such as this.
Thank you all for your continued support and patience.
Yeah, that’s what really scares me about database programming. I can have something work perfectly on my dev machine, but I’ll never find out how well it works under a real-world workload, and my employer really doesn’t like it when stuff blows up in a customer-visible way.
I decided to write a stress-test tool for my project that generates a bunch of test data and then hits the server with far more concurrent requests than I expect to see in production any time soon. Sure enough, the first time I ran it, my application crashed and burned just like Beehaw did. Biggest problem: I was using serializable transactions everywhere, and with lots of concurrent requests, they’d keep failing and retrying over and over, never making progress.
That’s a lesson I’m glad I didn’t learn in production…but it makes me wonder what lessons I will learn in production.
This is why I love canary and mirror releases when feasible. Hard to do with some projects though