Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn’t too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we’re very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

  • blueoak@lemmy.world
    91·
    1 year ago

    I’m not sure if this has been said but, when I open lemmy on browser, my account would sometimes be someone elses. I don’t know if it’s a bug and I saw it happen three times to me so far, and it even happened again a few minutes ago. It’s like I i logged into someone elses account, I saw three other usernames so far. A few minutes ago it said my account was Professor -?-?-?- with that account’s profile picture shown too. It only does that for half a second before it returns back to my account.

    I’m just making sure this is said because I don’t want to one day accidentally log into someone else’s account by accident.

    • blueoak@lemmy.world
      5·
      1 year ago

      I tried to find out what is causing this and I think it happens when you change your theme and save, then close the page, and in a new tab open lemmy again. I think it makes accounts that were just created appear too, I searched up the persons username and it said their account was made almost 20 minutes ago. edit: https://imgur.com/a/I4rO1pV here is the bug that I was talking about.

      • ItsDedo@lemmy.world
        4·
        1 year ago

        Definitely a caching issue. @Ruud does the nginx instance also do caching? Maybe you can restrict it to non-logged-in requests

      • ItsDedo@lemmy.world
        3·
        1 year ago

        Definitely a caching issue. @Ruud does the nginx instance also do caching? Maybe you can restrict it to non-logged-in requests