We’ll give the upgrade new try tomorrow. I’ve had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.

Also there are newer RC versions with fixed issues.

Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.

We see a huge rise in new user signups (duh… it’s July 1st) which also stresses the server. Let’s hope the improvements in 0.18.1 will also help with that.

  • 00Lemming@lemmy.worldEnglish
    841·
    2 years ago

    PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.

    While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.

    @ruud@lemmy.world will provide a debrief once we have completed testing.

    • TurnItOff_OnAgain@lemmy.worldEnglish
      19·
      2 years ago

      We are happy to have all of you! Do what is needed to make this place the best. Even reddit sucked in the early days.

      502 it went through, 504 try once more

      • 00Lemming@lemmy.worldEnglish
        5·
        2 years ago

        Thanks for the kind words! Yeah, there are definite growing pains, and likely will be for some time (just do to the codebase we are working with, understandably). We have a really solid group though heading up lemmy.world though so we will be just fine ;)

        • illi@lemmy.worldEnglish
          5·
          2 years ago

          Not sure if this is a good place to ask but it was on my mind these days, with the big user boom and probably most of it being directed at .world - should some of us consider moving to other instance to make room? Would it make sense? Would it help? Or are you ok having so many users under the .world umbrella (possibly causing even greater flood with people seeing it is the “main” (with biig quotes there) instance?

          • 00Lemming@lemmy.worldEnglish
            7·
            2 years ago

            Technically speaking, yes, a portion of our issues are due to the highest user base of an Lemmy instance. So in theory, if half of our users dispersed to other instances, we would likely see some performance improvement here. However, lemmy.world is intended to be an accessible instance for the general population. The server itself that is running lemmy.world is beyond spec’d to handle much more than this user load. We are running up against code-level issues that we may or may not be able to get around with our internal configurations. This is just part of developing software in an environment were you go from a few thousand users total to hundreds of thousands in the space of a few weeks. There is no directive to have users create accounts on new instances, though if you are looking for an immediate performance improvement, that may be your best option currently. That is up to you to decide :)

            • illi@lemmy.worldEnglish
              3·
              2 years ago

              Gotcha. So pretty much if I want to personally avoid the growing pains (or possibly ease them for others), I should consider it, but doesn’t particularly help you directly. Thanks for clarifying and thanks for all the work you all put into it!

              • 00Lemming@lemmy.worldEnglish
                3·
                2 years ago

                Yup you nailed it. For additional context, Ruud is running an almost identical server for his Mastadon.world server which has 160k users. Relatively speaking, these are large, performant, and expensive servers. They can absolutely handle the current user influx we are getting from the Reddit exodus. Are hands are tied by software limitations unfortunately. I can confidently tell you were are constantly in communication about ways we can amplify user experience with the tools that we do have access to. For instance, this status page was recently spun up which you can access anytime you think there might be server issues to help confirm that what you are seeing is recognized at a server level. Things like that.

                All that being said, for users who are looking for a smoother experience right now, I can recommend lemm.ee as a solid home as well. Their admin Sunaurus has been very active and helpful throughout this process and handles his instance very professionally. He is essentially another Ruud (though Ruud is the best! ;)). Just something to keep in mind going forward as I can’t make any promises about the time frames for these issues being resolved. Hopefully once we get contact back from the Lemmy devs we can start expediting a resolution. They have a lot on their plates right now though, haha, so we will see. Cheers!

      • 00Lemming@lemmy.worldEnglish
        41·
        2 years ago

        The login form should be related to the overall spiking we were seeing, though I can’t say we had conclusive evidence of that. I have been able to get it to load properly with a few refreshes. Maybe try clearing your cache? I had to do that several times during testing.

      • Stanford@discuss.as200950.comEnglish
        3·
        2 years ago

        Please try to clear the browser change
        The new version is doing some of the API calls differently, which may cause issues with the old/cached version in your browser

  • simple@lemmy.world
    55·
    2 years ago

    Server feels a lot better now than it did an hour ago. Comments are going through quickly and upvotes are working for me.

    • ekZepp@lemmy.world
      25·
      2 years ago

      Upvote are still slow. I’m testing the comment right now.

      Ok, commented are fine ( edit too) 👍

  • MrCenny@lemmy.world
    30·
    2 years ago

    Heyo! Small update from someone who is watching the upgrade live; it’s ongoing still. Seems like they are still facing some performance issues. So grab your popcorn and wait! 🍿 (Btw, i am not a sysadmin, just a moderator)

    • The_Worst@lemmy.worldEnglish
      3·
      2 years ago

      Takes a while, but it is possible to upvote. Reloading the pages makes the upvotes appear.

  • rtxn@lemmy.world
    22·
    2 years ago

    Performance is extremely uneven. Sometimes loads instantly, sometimes I get a timeout. Upvotes don’t show up until reload. Still a lot better than the last attempt.

    The server hamsters seem to be on fire, but I won’t hold it against the service/community/individuals (except spez; fuck you spez). Lemmy is in the unfortunate position of being forced into a development model called “fuck it, we’ll do it live!”

  • WantsToPetYourKitty@lemmy.world
    14·
    2 years ago

    Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?

  • MrNemobody@lemmy.world
    13·
    2 years ago

    I can’t login on my lemmy.world account, neither on Jerboa nor in the browser, I’m getting “Incorrect login” :(

    (I’m posting from Liftoff because I was already logged in)

    • morgan423@lemmy.world
      3·
      2 years ago

      I get the feeling that it’s traffic. This instance has grown much larger, stupidly fast, over the last two weeks due to Reddexit (myself included, having arrived less than two weeks ago). I’m sure the admins are working to find ways to handle it.

      I’m just going to exercise patience. Good things are worth waiting for!

      • Coelacanth@lemmy.world
        1·
        2 years ago

        I hope for the sake of the exodus that the servers can cope with the pressure. Lemmy.world is shaping up to be somewhat of a primary instance and having it stable would be crucial for any sort of migration to stick.

  • AlmightySnoo 🐢🇮🇱🇺🇦@lemmy.worldEnglish
    11·
    2 years ago

    Cleared my cache and cookies as I thought it would be necessary like last time, but now I can’t log in. The login button just turns into the animated wheel and then nothing happens. I’m typing this on a mobile where I still have an active session.

  • riodoro1@lemmy.world
    11·
    2 years ago

    TFW you run a small server with an open source social platform for shits any giggles and one month it explodes a thousandfold because of one greedy pig boy.

    • IndiBrony@lemmy.world
      21·
      2 years ago

      I’ve spent almost an hour trying to just to sign up. Poor servers. This is my first comment here and I sincerely hope this takes off in the best way.

      • Guy_Fieris_Hair@lemmy.world
        2·
        2 years ago

        Been here for a month~. Other than a few growing pains it’s been pretty smooth. An initial learning curve on the whole instance/lemmy/fediverse thing and learning to sort by all/hot or active instead of just “local” to get ALL if the lemmy-verse instead of just your own instance. Once you learn how to use it it’s a much better experience the reddit. Right now it’s getting the reddit hug of death. But it will survive and flourish.

      • Ulu-Mulu-no-die@lemmy.world
        2·
        2 years ago

        Servers are being hammered hard right now lol, but it should calm down soon, you’ll see it’s pretty stable outside of reddit hugs of death :D

        • douglasg14b@lemmy.world
          1·
          2 years ago

          It’ll calm down as in more resources are provisioned or redditors looking to ditch reddit give up and go back to reddit because of stability issues, thus lowering use count closer to normal?

          Because the later isn’t ideal 😬

          I have business fiber and my own compute, can nodes be ran distributed? I’d be happy to spin up VMs to run a node service. Or does this rely on a single central host?

          • Admin@lemmy.magnor.ovh
            0·
            2 years ago

            You can host your own instance, which will be federated with this one by default. Every little bit helps.

      • Xilly@lemmy.world
        1·
        2 years ago

        I’ve been on here almost a month and there are usually very few issues and the admin team are quick to troubleshoot when an issue does arise. It should calm down soon!