Unfortunately we had an issue updating to the latest version of Lemmy. This caused database corruption and we had to roll back to a backup that was unfortunately from 2 days ago!
An abundance of caution in the wrong error resulted in me wanting to push this change out to help with the exploit relating to custom emoji. As other instances were attacked, I didn’t want reddthat to fall to the same issue.
I am so sorry !
I have since updated our deployment scripts to automatically take a back up before all updates, as well as investigating why our cron backups didn’t work as we expected them too.
We also lost all new accounts that were created over the past 2 days, all new posts, and all subscriptions. I’m very sorry about that and hope that you will reregister with those account details again, repost the memes and pictures that make our server great.
The good news is that we have updated to the latest version of Lemmy and everything is back to “normal”.
Fixing Deployments
I’m going to use some of our money for a dedicated development server as our current deployment testing is clearly lacking.
Unfortunately that will have to wait until the weekend when I have some extra time to put into developing these new processes.
Tiff
When you have now… 500! active users and over 2.8k user accounts I feel somewhat responsible for people having a good experience. When have you used a service which lost data twice in a week! I’m annoyed in that these things are easily avoidable but they are causing issues. Moving forward I’ll be putting in better more checks around deployments to make sure these cannot happen again.
I keep putting too much trust into the upgrade process when new versions come out expecting it to work without issues.
I really appreciate the apology, and every web admin and sys admin has either made the same mistake, or watched someone else make it. I could absolutely see my younger self creating an instance of Lemmy and doing the exact same.
The fact that you acknowledged the mistake, learned from it, and communicated the improvements are all great signs that we’re in the right place!
Thanks for all you’re doing!