Just watching the proverbial bodies stack up

  • thisbenzingring@lemmy.sdf.orgEnglish
    64·
    5 months ago

    I hate you. Today I discovered a server that appeared to be in working order was actually just hanging on by a thread. The crash when the crowdstrike driver caused the blue screen of death must have corrupted some of the network configuration in the registry. When I couldnt make the failover cluster live migrate to or from that server I ejected it from the cluster and rebooted it, only it never came back up 😥

    Now all my servers are suspect until I can verify that they are healthy.

    • RegalPotoo@lemmy.worldEnglish
      30·
      5 months ago

      Ah yes, the realisation that everything dependable and solid in the world is just a monument to hubris, built on foundations of sand

      • Laser@feddit.org
        6·
        5 months ago

        In general, I don’t think the concept of a configuration database is bad - at least not inherently moreso than every application putting its own configuration file(s) in whichever format it wants somewhere on the filesystem.

        Whether the window registry is a good implementation of this concept is surely debatable.

  • mesamune@lemmy.worldEnglish
    421·
    5 months ago

    This latest debacle is making my department move from windows to Linux. We were already planning it very slowly but then everything crashed at the same time…and all our other services worked except the ones on windows boxes. We can’t afford downtime so it was decided.

    It’s not a good reason but I’ll take it haha.

    • kautau@lemmy.world
      241·
      5 months ago

      I don’t know how CrowdStrike works on Linux, but it’s worth remembering that if it’s a kernel level driver like it is on windows, and they release a driver that crashes the Linux kernel, there’s a chance for the same thing to happen.

        • ramble81@lemm.ee
          271·
          5 months ago

          Thank you for mentioning that. I really hate how people on here think Linux is some panacea that will magically solve everything. It too it just another tool that depends on how it’s used. CrowdStrike exists for Linux, and it was crashing systems a few months ago.

          The bigger issue is most people who use Linux know what they’re doing. There are a lot of competent Windows Administrators too who didn’t have issues or were able to recover them in a timely manner. What happens though is you have a very large set of people who just need a computer, need is secured and don’t know how to administer or manage it. Doesn’t matter if they’re running Linux, or Windows, they’ll always have the greatest problems. They just happen to use Windows because it offers better Enterprise support options and usability. One day, Linux may be that, but I guarantee it won’t fix all of those pebkac issues.

          • frog_brawler@lemmy.world
            17·
            5 months ago

            I mostly agree. As someone that’s worked with both Windows and Linux for over 15 years, I think we need to ask the question of “why do we see so many incompetent admins?”

            If you aren’t paying people enough to give a shit about what they are doing, they won’t.

      • mesamune@lemmy.worldEnglish
        6·
        5 months ago

        I personally get nervous when any software wants to mess with drivers unless it’s graphic drivers.

        For work we don’t plan on using cloud strike. We needed to get everything up asap and the os allowed us to do so quickly. Seemingly unrelated systems and Azure was all down for quite some time.

  • Bakkoda@sh.itjust.worksEnglish
    39·
    5 months ago

    I was on vacation. They laid off my boss 2 weeks ago. They laid off my bosses boss 3 weeks ago. Guess who no one could contact?

    I found out about all of this 2 days after the fact. Best vacation ever.

    • wheeldawg@sh.itjust.worksEnglish
      1·
      5 months ago

      Wtf did they honestly expect from you in that situation anyway? Do they really think you would come back from vacation to deal with that? From what I can tell, fixing it is barely beyond a T1 hekpdesk amount of knowledge, just an annoying process. Unless there’s more to it, I haven’t looked heard at the details but that’s the gist I’m hearing.

      But unless they’re going to triple my vacation length (in terms of paid days off in other words) and maxing out all my upgrade options for the next one I’ma have to say they can get bent.

      Basically they gotta pay for the best version possible of that trip (assuming there was a trip) and give me enough time to take 2 more.

      I ain’t taking a work call on vacation.

      • Bakkoda@sh.itjust.worksEnglish
        2·
        5 months ago

        Funny story. My boss vetoed work phones as a cost savings and at first there was a squawking fest but honestly he handled all communication to us and protected us from so much bullshit. Now he’s gone and no one could figure out how to get in contact with us aside from email/teams.

  • radicalautonomy@lemmy.world
    13·
    5 months ago

    The Feels Good guy has a stick figure hand sticking up out of his arm. I cannot unsee it, and I suspect now neither can you.