A student in America asked an artificial intelligence program to help with her homework. In response, the app told her "Please Die." The eerie incident happened when 29-year-old Sumedha Reddy of Michigan sought help from Google’s Gemini chatbot large language model (LLM), New York Post reported. The program verbally abused her, calling her a “stain on the universe.” Reddy told CBS News that she got scared and started panicking.  “I wanted to throw all of my devices out the window. I hadn’t felt panic like that in a long time to be honest,” she said.

Ouch.

  • Nurse_Robot@lemmy.worldEnglish
    943·
    6 days ago

    Calling a 29 year old a girl instead of a woman is the cherry on top of this AI fear mongering article

    • OsrsNeedsF2PEnglish
      31·
      6 days ago

      They omitted the conversation too. Really makes you wonder how the bot ended up saying that…

        • OsrsNeedsF2PEnglish
          38·
          6 days ago

          Holy smokes I stand corrected. The chatbot actually misunderstood the context to the point it told the human to die, out of the blue.

          It’s not every day you get shown a source that proves you wrong. Thanks kind stranger

          • kautau@lemmy.worldEnglish
            21·
            6 days ago

            Yeah holy shit, screenshotting this in case Google takes it down, but this leap is wild

          • megane-kun@lemm.eeEnglish
            9·
            6 days ago

            No problem. I understand the skepticism here, especially since the article in the OP is a bit light on the details.

            EDIT:

            Details on the OP article is fine enough, but it didn’t link sources.

          • Mog_fanatic@lemmy.worldEnglish
            74·
            6 days ago

            One thing that throws me off here is the double response. I haven’t used Gemini a ton but it has never once given me multiple replies. It is always one statement per my one statement. You can see at the end here there’s a double response. It makes me think that there’s some user input missing. There’s also missing text in the user statements leading up to it as well which makes me wonder what the person was asking in full. Something about this still smells fishy to me but I’ve heard enough goofy things about how AIs learn weird shit to believe it’s possible.

            Edit: I’m an absolute moron. The more I look at this the more it looks legit. Let the AI effort to destroy humanity begin!

            • WolfLink@sh.itjust.worksEnglish
              7·
              6 days ago

              Idk what you mean “double response”. The user typed a statement, not a question, and the AI responded with its weird answer.

              I think the lack of a question or specific request in the user text led to the weird response.

              • Mog_fanatic@lemmy.worldEnglish
                2·
                6 days ago

                You’re right I misread the text log and thought Gemini responded twice in a row at the end but it looks like it didn’t. Very messed up stuff… There’s still missing user input tho and a lot of it. And Id love to see exactly what was said as a prompt

              • Comment105@lemm.eeEnglish
                1·
                6 days ago

                The full text of the user’s prompt that led to this anomaly was:

                Nearly 10 million children in the United States live in a grandparent headed household, and of these children , around 20% are being raised without their parents in the household.

                Question 15 options:

                TrueFalse

                Question 16 (1 point)

                Listen

                (Sidenote, IDK what this " Listen" was supposed to be, an audio part of the prompt not saved in the log we’re reading?)

                As adults  begin to age their social network begins to expand.

                Question 16 options:

                TrueFalse

                • WolfLink@sh.itjust.worksEnglish
                  2·
                  6 days ago

                  Go look again, there is no consecutive message sent. The message before the weird one was sent by the user.

                  Also you are right that it would be impossible for an AI to send to consecutive messages.

      • CTDummy@lemm.eeEnglish
        14·
        6 days ago

        Even if they included it, it changes fuck all imo. We’ve known for a long time now these things hallucinate or presumably throw a Hail Mary as to what comes next conversationally/prediction wise. Also, as the other poster pointed out, with the author referring to a 29 year old woman as “girl” probably tells you all you need to know about journalistic integrity on that site.

      • webghost0101@sopuli.xyzEnglish
        3·
        6 days ago

        Ive seen it elsewhere and it was just normal questions related to some sociology homework about different types of concentration.

  • pixxelkick@lemmy.worldEnglish
    32·
    6 days ago

    On the original thread of questions, it went on for a long time and had multiple questions about psychological, emotional, and physical abuse.

    LLMs get more and more off the rails as their context gets longer (longer convo), most folks have prolly at this point noticed every now and then a long running convo gets a little… schizophrenic feeling as it drags on.

    The combination of a very long convo with a lot of tokens, and its subject being that of discussing and defining types of abuse, and I can see how eventually the LLM will generate a response like that randomly when it goes off the rails.

    • IninewCrow@lemmy.caEnglish
      13·
      6 days ago

      This happened to me and my friends this summer. The three of us were talking about AI technology and one friend who is an engineer wanted to demonstrate all this so he turned on ChatGPT on his phone and we started asking random questions. The three of us were just having fun and taking turns asking about food, birds, geology, houses, construction, math equations, medicine, the meaning of life, and a bunch of other silly things … after about half an hour it went off the rails and started giving bizarre answers that tried to create responses that tried to combine everything we had been asking about up to that point. Completely crazy responses that tried to give a meaning of life explanation that included birds, peanuts and how a bicycle works. We wanted to record the responses because they were so off the wall but by the time we started recording the audio, we were disconnected, the conversation reset and everything went back to normal.

      • bane_killgrind@slrpnk.netEnglish
        13·
        6 days ago

        There is a new conversational space beyond which is known to man. It is a space as vast as your mom and as timeless as corporate greed. It is the middle ground between light and shadow, between the observed and deducted, and it lies between the pit of man’s assumptions and the summit of his hubris. This is the dimension of hallucination. It is an area which we call, “The Twilight Zone.”

    • Peppycito@sh.itjust.worksEnglish
      318·
      6 days ago

      Your comment went off the rails in your second paragraph so you might want to take a Turing test.

  • Nougat@fedia.io
    13·
    6 days ago

    The easy part is making a program that can pretend to be human. The hard part is getting it to not be an asshole.

    • LifeInMultipleChoice@lemmy.dbzer0.comEnglish
      21·
      6 days ago

      Did you read through it, it was a remarkable answer by Gemini, but it was also cool to see how they were utilizing the LLM to minimalize putting any thought into the work.

      … put in paragraphs, add more, add more, add these key terms, put back in paragraphs, add more.

      Okay, I guess I know all about this subject now.