Today, a prominent child safety organization, Thorn, in partnership with a leading cloud-based AI solutions provider, Hive, announced the release of an AI model designed to flag unknown CSAM at upload. It’s the earliest AI technology striving to expose unreported CSAM at scale.

  • floofloof@lemmy.ca
    link
    fedilink
    English
    arrow-up
    19
    arrow-down
    3
    ·
    1 month ago

    This seems like a potential actual good use of AI. Can’t have been much fun to train it though.

    And is there any risk of people turning these kinds of models around and using them to generate images?

    • Jimbabwe@lemmy.world
      link
      fedilink
      English
      arrow-up
      32
      arrow-down
      10
      ·
      1 month ago

      If AI was reliable, maybe. MAYBE. But guess what? It turns out that “advanced autocomplete” does a shitty job of most things, and I bet false positives will be numerous.

      • pearsaltchocolatebar@discuss.online
        link
        fedilink
        English
        arrow-up
        6
        arrow-down
        1
        ·
        1 month ago

        It’s possible to have a good AI system, but it takes millions of dollars and several thousand manhours to do, and most companies won’t put in the effort.

        But, there should always be a human in the loop.

      • AwesomeLowlander@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        4
        arrow-down
        3
        ·
        1 month ago

        “detect new or previously unreported CSAM and child sexual exploitation behavior (CSE), generating a risk score to make human decisions easier and faster.”

        False positives don’t matter if they stick to the stated intended purpose of making it easier to detect CSAM manually.

        • Voroxpete@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          11
          ·
          edit-2
          1 month ago

          The problem is that they won’t.

          Yes, AI tools, in the hands of skilled people, can be very helpful.

          But “AI” in capitalism doesn’t mean “more effective workers”, it means “fewer workers.” The issue isn’t technological so much as cultural. You fundamentally cannot convince an MBA not to try to automate away jobs.

          (It’s not even a money thing; it’s about getting rid of all those pesky “workers rights” that workers like to bring with us)

          • AwesomeLowlander@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 month ago

            Here’s the thing. This technology is unequivocally one of the things AI would be very useful for. It can potentially do a lot of good. Yes, MBAs could screw it up like they screw anything else up in society. That doesn’t mean we shouldn’t be happy that we’ve created this new tech.

    • FaceDeer@fedia.io
      link
      fedilink
      arrow-up
      14
      ·
      1 month ago

      And is there any risk of people turning these kinds of models around and using them to generate images?

      There isn’t really much fundamental difference between an image detector and an image generator. The way image generators like stable diffusion work is essentially by generating a starting image that’s nothing but random static and telling the generator “find the cat that’s hidden in this noise.”

      It’ll probably take a bit of work to rig this child porn detector up to generate images, but I could definitely imagine it happening. It’s going to make an already complicated philosophical debate even more complicated.

    • mspencer712@programming.dev
      link
      fedilink
      English
      arrow-up
      9
      arrow-down
      1
      ·
      1 month ago

      I think image generators in general work by iteratively changing random noise and checking it with a classifier, until the resulting image has a stronger and stronger finding of “cat” or “best quality” or “realistic”.

      If this classifier provides fine grained descriptive attributes, that’s a nightmare. If it just detects yes or no, that’s probably fine.