Last night, Lemmygrad went down due to a hardware glitch on our VPS provider. Unfortunately, they were unable to do anything to help me fix the issue, and asked that I reinstall from scratch 🙁 . They did inform me that the hardware issue at fault is fixed at least.
I do daily backups, so we lost a day of posts.
Pictures are in the process of being restored now, but it should take a day or so to restore them all.
Let me know if there are any other ongoing issues so I can fix them. Again, sorry everyone.
Whenever Lemmygrad is down I just sigh and say ‘the bourgeoisie at it again 🙄’
Being involved in devops, I can relate to how much this sucks and I appreciate you taking the time to fix it. o7
Thanks comrade, ya these can be nightmare scenarios. I at least have several backup locations, so at most we lose a day’s worth of data.
I’m not involved in this type of work but I would just pull the plug, I can’t understand the amount of patience you need for this shit
Everything will be going fine for weeks, and then: oopsie poopsie everything’s gone!
Anecdotally, my instances have poofed into thin air even on AWS, so this seems not too rare. You use a managed database or something you’ve setup yourself?
Set up ourselves. Its postgres running in a docker container, with daily backups.
we lost a day of posts
Spongebob_suffering.jpeg
Pain
Have this consolation prize:
more pain
Honestly how dare you 😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾🤖😾😾😾😾😾😾😾😾😾😾
Our VPS on what went wrong:
My cat
nice to get an update, i was confused yesterday. every time i consistently can’t connect to lemmygrad i assume the worst hahaha
No worries, these things happen. Look on the bright side, thankfully it was just one day loss and not more.
I mean, you can’t stop a kitty cat from having fun, don’t be a monster
Can relate.
Is it feasible to restore the posts and/or comments that were federated to lemmy.ml? I can see my stuff from yesterday there
(oh and also thank you for dealing with this so quickly)
Is there even 20 people on lemmy?
Edit: nevermind, that’s not the reason, but yeah, definitely something weird happening over there, some comments are instantly getting a bunch of votes
Hrm… I’ll take a look.
I don’t think they can get sync’d correctly back unfortunately, because the original source is gone.
That’s unfortunate
might have to manually repost 👀
I’m glad it’s back, thank you for your work!
thanks for your hardwork. I was concerned there was some type of ddos or something
On the topic of connection issues. I haven’t been able to connect to the front page when I’m logged in to my account. But i can still visit the communities page or go to specific communities. If i log out the front page works.
Not sure about that one, you might have to open up an issue on the lemmy github.
Will do, thanks.