Please just provide a short, medium or long story

  • linearchaos@lemmy.world
    link
    fedilink
    English
    arrow-up
    80
    ·
    4 months ago

    Used to work for a gaming company that had a really big Facebook presence.

    When most companies launched a new game they would just launch the game connect to Facebook and call it a day.

    But when we launched a new game we would turn on cross-promotion and send millions of users to the new game day one.

    The problem is that Facebooks load balancers don’t scale quickly. And if you start sending millions of people to a game that just errors, it’s pretty bad for marketing.

    So we had a script that you would run for an hour before you went to launch a game that would start making hundreds then thousands than tens of thousands of connections to Facebook giving their load balancers time to scale.

    Well, I didn’t write the tool I was just given it and the documentation on it was kind of sparse at the time. I started with the value of 30,000. Weeeellll, turns out a sane value would have been 3-5 as it multiplied whatever input you gave it by 10K.

    My cluster made sweet love with their cluster, and Facebook went down for about 3 hours.

    Within a week they had some pretty decent changes to their new API limit subsystem.

  • Shadow@lemmy.ca
    link
    fedilink
    arrow-up
    68
    ·
    edit-2
    4 months ago

    Two big ones in my younger days:

    Alt tabbed one too many times, clicked drop database, clicked ok, realized I’d just deleted the live user database for America’s Army. Thankfully it was the east coast site and west coast was the primary, and it was only one way replication. We shut down east coast auth and rebuilt the secondary.

    Someone distracted me while typing in a vlan command on a switch, I hit enter without double checking, took out our fiber between two datacenters in the middle of a move. Took me 15 minutes to run to the DC, plug in a console cable and fix it. Took all of our customers out.

    • Chozo@fedia.io
      link
      fedilink
      arrow-up
      10
      ·
      4 months ago

      America’s Army is such a bizarre game, at a conceptual level. Do you have any other fun stories from that? I have to imagine a lot of weird things went down behind the scenes.

      • Hadriscus@lemm.ee
        link
        fedilink
        arrow-up
        5
        ·
        4 months ago

        oh shit so that’s a game ! I thought this guy had taken down the entire US army in one click.

  • ickplant@lemmy.world
    link
    fedilink
    arrow-up
    49
    ·
    4 months ago

    I’m going to tell a story on behalf of my husband.

    He was 13 and in Boy Scouts. Their troop was told that some older scouts went missing and the troop had to look for them. They formed patrols and were searching for over 3 hours when the leaders said that the older scouts were located; one of them was disemboweled and needed a medical helicopter to come from Denver. That was the “oh, crap!” moment…

    Turns out the whole thing was staged. No one ever went missing. They just wanted the troop to learn how to do search and rescue. There were younger scouts there who were crying and terrified, definitely scarred by the experience.

    And that’s how things were done back in the 80s.

    • Raiderkev@lemmy.world
      link
      fedilink
      arrow-up
      15
      ·
      4 months ago

      I think this is a common thing with boy scouts. We happened upon an accident in between stations at camp. A kid ran up and tells us his dad crashed his car and he needed help. There was a man who was laying in front of the truck on the ground. His arm was bleeding profusely. We needed to administer first aid. The guy had a bunch of blood all over his arm and he was acting all incoherent. We decided to Jerry rig a tourniquet to stop the bleeding and send someone up the road to find a phone (pre cell phone days) The whole experience was super traumatic. All staged. Fuckers. They even had a pump shooting out fake blood from the guys arm.

  • Jo Miran
    link
    fedilink
    arrow-up
    45
    ·
    edit-2
    4 months ago

    September 12, 2001. I accidentally shut down an entire production plant. Management didn’t even get mad. They closed the plant for the day, I kicked off the boot cycle (takes hours for the system to be ready for production again) and everyone went home to be with their families. Nobody’s head was on right that day anyway.

    EDIT: A few years later I was testing some BigIP configs on a tertiary unit when suddenly the entire e-commerce site went down. Apparently this unit used to be a primary before being demoted and someone (not me) forgot to disable replication, so when I wiped all the rules from my “test unit” I inadvertently wiped all the rules to the production units. Technically it wasn’t my fault but it was still an “oh fuck” moment.

  • BlueSquid0741@lemmy.sdf.org
    link
    fedilink
    arrow-up
    37
    ·
    4 months ago

    Shit job from 10 years ago. Getting too drunk at the office Christmas party and talking so much shit about my horrible fucking boss to a couple of influential senior managers who are close to the CFO. Freaked out all weekend about it, severe anxiety attacks. That mixed with the hangover I was just vomiting all weekend.

    And then: Run into one of those senior managers early in the office on Monday morning, apologised for being inappropriate. Get a response “oh, we all know how bad she is, can’t believe you haven’t quit yet”.

    • HelixDab2@lemm.ee
      link
      fedilink
      arrow-up
      13
      ·
      4 months ago

      Senior managers were also awful then; part of their job is making sure that the lower level managers don’t suck, and they weren’t.

      • BlueSquid0741@lemmy.sdf.org
        link
        fedilink
        arrow-up
        13
        ·
        4 months ago

        Absolutely. A few months after this I went directly to the person above my boss and explained the situation- mental health, physical health, turned into dependence on weekly psychologist sessions.

        Basically told me to get over it.

        This was a non profit organisation dedicated to helping homeless, refugees, victims of domestic violence, and elderly people who are unable to afford aged care. The people on the ground doing the work were amazing people. But the people in charge were all cosplaying as big business shitheads.

        • HelixDab2@lemm.ee
          link
          fedilink
          arrow-up
          6
          ·
          4 months ago

          I’ve heard that kind of thing about a number of non-profits. Makes me wonder how they manage to attract so many awful people.

          Or maybe people in general are just awful.

  • mesamune@lemmy.world
    link
    fedilink
    English
    arrow-up
    25
    ·
    4 months ago

    My old company stopped contributing to our 401k even when they were taking the $$ out. The CEO Irma gave us her personal message that they were going to take care of it and that we were doing just fine.

    Then we got a surprise video meeting over memorial day. The oh shit moment was everyone was told via the call that we were indefinitely furloughed. I didn’t even know that was thing. Unfortunately when you get furloughed like we did, you can’t get unemployment since your not terminated, you can’t get insurance, and paychecks started to bounce from over a month ago. It was a bad situation. So yeah that was my oh shit situation.

    • linearchaos@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      ·
      4 months ago

      k even when they were taking the $$ out. Th

      Happened at a previous company for me too. They were paying people with their own 401k’s for quite some time before they went under. Some of my coworkers were putting in max contributions.

      No one ever saw any of that money back. The management just rolled into a new venture like nothing happened. There were tons of lawsuits but they had all their money hidden away.

    • ultranaut@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      4 months ago

      Did you ever get the money they owed you? I had a much less exciting version of that where a job I had was taking money out of paychecks for insurance that was never actually provided. Many years later the courts sorted things out and a few thousand showed up in the mail.

  • MostRandomGuy
    link
    fedilink
    English
    arrow-up
    24
    ·
    4 months ago

    Yesterday I was waiting for the Tram.

    As I stood there, I turned my head to the right and witnessed how a pigeon was hit by a car.

    Kinda traumatizing, especially when the cars that followed ran over and over the carcass.

    • Che Banana@beehaw.org
      link
      fedilink
      arrow-up
      5
      ·
      4 months ago

      A few years ago while commuting to work a squirrel fell out of a tree, bouncing off my hood and under the driver side tire.

      All I could think of was that little guy must’ve had some fucked up karma…

  • crimsoncobalt@lemmy.world
    link
    fedilink
    arrow-up
    23
    arrow-down
    1
    ·
    4 months ago

    I was once driving on some back roads I was unfamiliar with. I turned a corner and didn’t realize there was a stop sign until too late and went right through the intersection. As I went through it, I turned my head to the left and noticed that a car was heading right for me. It missed me by inches.

    • Today@lemmy.world
      link
      fedilink
      arrow-up
      12
      ·
      4 months ago

      I had two similar ones-

      1. driving across a bridge and just in front of me, at the end of the bridge, a volswagon eos flew threw the air over the street and landed on the grass on the other side.

      2. Driving on a street that’s parallel to railroad tracks and then jogs to cross them. As i crossed the tracks i glanced right and saw the front of the train. I had been driving beside and just on front of it and didn’t realize it.

      3. One extra - i was going down a hill in a really dark neighborhood. It was so dark in front of me that i stopped. i was sitting on a boat ramp into the gulf of Mexico.

      • KittenBiscuits@lemm.ee
        link
        fedilink
        English
        arrow-up
        5
        ·
        4 months ago

        I have had the train thing happen to me. There were no warning lights where the road crossed over. Fortunately it was a coal train going very slowly. Scared the living daylights out of me. It’s been 30 years since that happened and I’m still neurotic about railroad crossings with poor visibility.

        • Today@lemmy.world
          link
          fedilink
          arrow-up
          3
          ·
          4 months ago

          Yeah. On, i think - I’ve wondered about that over the years. I was 17 and it was an old 1978 Plymouth Fury. Scared the shit out of me! Still does.

  • TheChargedCreeper864
    link
    fedilink
    arrow-up
    20
    ·
    4 months ago

    My grandpa had developed the habit of falling out of his bed. The first time I was afraid that he was gonna die on the spot as I’d heard it, but it eventually became such a “regular” occurrence that I didn’t think of immediate death anymore. This particular day, he’d fallen twice. They brought him to a nearby hospital to get a check-up. I was worried sick that this time something was actually wrong, or that he might’ve broken a bone or something. Turns out he was fine! No broken bones or anything. Just one teeny tiny minor issue…

    When he was brought to the hospital, he was accidentally placed in the area with people who were brought there with covid. I hadn’t been able to see him in months because of the restrictions, and even when I did go the months prior it was always with far distance, masks and in short bursts. I did everything I had been told to do to “keep him safe”, “ease up the workload in the hospitals” and all those government campaigns and all that, only for him to die because of this (seeming) serious neglect from medical professionals.

  • nomad@infosec.pub
    link
    fedilink
    arrow-up
    17
    ·
    4 months ago

    I was testing some code late at night in the test system. Rolled out the changes, log on to the admin interface and write a short news article about how one of the more hated profs at the university had died suddenly and unexpectedly.

    Result looks good, roll out changes to prod, about to call it quits for the night. Think to myself: common reason people get fired, maybe delete the story from test system. Check test system, no story there… Uhoh.

    Story has been live for about three hours. Hope no spiders have caught it yet, hurry to delete it and learn how to purge all evidence from database.

    Turns out the shithead admin had copy and pasted the server config for the test system from live and forgotten to change the admin rewrite rules to test system. Phew…

  • GuyFi@lemmy.sdf.org
    link
    fedilink
    arrow-up
    21
    arrow-down
    4
    ·
    edit-2
    4 months ago

    I predict this will be quite tame compared to the rest of this thread but here you go anyway:

    Our scene opens on a person with a bass guitar, practicing some sick slap bass perched upon an IKEA office chair. The player is lost in the zone, all mental facilities put to work making every strike with the thumb is accurate and powerful, and every pop of the fingers sends the strings into the fretboard with a gorgeous thwack! The sound is heavenly, a deep, rich, cutting tone- full of appropriate levels of CLANK and SNAP. The short riff being performed to a concert of no-one reaches its magnificent height, a slammingly heavy riff sounding like a funky machine gun, the bassists hands become a blur, the strings vibrate with precision and power, a glorious cacophony of ma-

    kkrrmp.

    Oh crap, I broke a string

  • DudeDudenson@lemmings.world
    link
    fedilink
    arrow-up
    16
    ·
    4 months ago

    It was my sister’s birthday and I was over her place helping set everything up for the night party and then she said “dad’s not picking up the phone”

    I had the gut feeling then and there something had happened and indeed he had had a hearth attack that morning

    • texasspacejoey@lemmy.ca
      link
      fedilink
      English
      arrow-up
      9
      ·
      4 months ago

      I went to check on my brother in law while his parents where camping. Found him dead in their bed. It was the night before my birthday. Devastated…

  • dan@upvote.au
    link
    fedilink
    arrow-up
    12
    ·
    edit-2
    4 months ago

    I took down the home page of one of the top 5 websites for around 5 minutes.

    There were two existing functions that were written by a different team: An encode method that took a name of something (only used internally, never shown to the user) and returned a numeric identifier for it, and a decode method that did the opposite.

    Some existing code already used encode, but I had to use decode in my new code. Added the code, rolled it out to 80% of employees, and it seemed to work fine. Next day, I rolled it out to 5% public and it still seemed okay.

    Once I rolled it out to everyone, it all broke.

    Turns out that while the encode function used a static map built at build-time (and was thus just an O(1) lookup at runtime), decode connected to a database that was only ever designed for internal use. The DB only had ten replicas, which was nowhere near enough to handle hundreds of thousands of concurrent users.

    Luckily, it’s commonplace to use feature flags changes, which is how I could roll it out just to employees initially. The devops team were able to find stack traces of the error from the prod logs, find my code, find the commit that added it, find the name of the killswitch, and disable my code, before I even noticed that there was a problem. No code rollback needed.

    That was probably 7 years ago now. Thankfully I haven’t made any mistakes as large as that one again!

    Always use feature flags for major changes, especially if they’re risky!

  • Copythis@lemmy.world
    link
    fedilink
    arrow-up
    12
    ·
    4 months ago

    I heard that “who let the dogs out” by the Baha Men is about ugly girls coming to the club.

    I was explaining this to a coworker, and one of my female coworkers were around. After I said it, I looked at her and said “oh my gosh I’m so sorry” because I thought it was inappropriate to say at work.

    She took it as I was calling her ugly! (she was though)

      • ettyblatant@lemmy.world
        link
        fedilink
        arrow-up
        8
        ·
        4 months ago

        The lyrics support this.

        Well, the party was nice, the party was pumpin’ Heya, yippie yi yo And everybody havin’ a ball Huh, huh, yippie yi yo I tell the fellas start the name callin’ Yippie yi yo And the girls respond to the call I heard a woman shout out Who let the dogs out? Who, who, who, who, who?