Okay, but this can’t happen for real. If you’re rational you should also be aware that things that appear in fiction also and in majority DO NOT happen irl. The possibility of something isn’t the likelihood of something. It’s possible I could die of an aneurysm in the next 15 minutes, not probable though, and I’m not gonna live in fear of it cause it could potentially happen without weighing that potential. Rockin Basslicks is low as hell on the probability scale and an absolute fucking zero on the I’d care if it was real scale. Same with any simulation idea or hell, even the idea of God. I don’t care if I’m in one or if there is one, there is work to be done making this simulation or created universe better for people in it right here and right now and navel gazing about speculations way beyond what we can understand is a selfish and useless waste of time. Fucking feed the poor.
LLMs in their current form are dangerous to society, but not existential threats. To see the danger of human extinction, multiply some probabilities together:
At some point, AIs will likely become super-intelligent, meaning they are smarter than the entire human race put together. Let’s say there’s a 50% chance of this happening in the next 30 years. Something this smart would indeed be capable of doing something very clever and killing all humans (if it wanted to); there are various theories about how it might go about doing that, and if this is the part that sounds outlandish to you then I can elaborate. Needless to say, if something is extraordinarily smarter than you, it can figure out a way to kill you that you didn’t think of, even if they’re sandboxed in a high-security prison. (Mind you, it will probably be exposed like ChatGPT to the general public, not very likely to be sandboxed if you ask me.)
Okay but surely nobody would build an AI that would want to kill us all right? This is the “alignment problem” – we currently don’t know how to make sure an AI has the same “goals” that its creators want it to have. There’s that meme – an AI tasked with optimizing a silverware production process might end up turning the whole universe and everyone in it into spoons. Because almost nobody is actually taking this problem seriously, I think the first superintelligent AI has an 80% chance to be unaligned.
Would an unaligned AI want to kill us? It might be unaligned but conveniently still value human life. Let’s say be generous and say it’s a 50% chance that the unaligned AI works out in our favour. (It’s probably less likely.)
So that’s a 20% chance in the next 30 years that someone will create an AI which is clever enough to kill us all, and (by unhappy accident) wants to kill us all. You can put your own numbers on all of these things and get your own probability. If you put very low numbers (like 0.1%) on any of these steps, you should ask yourself if in 2010 you thought it was likely that AI would be where it is today.
Edit: yes I know it sounds absurd and like fantasy, but a lot of real things sound absurd at first. One of the most pervasive arguments against global warming is that it sounds absurd. So if you’re going to disagree with this, please at least have a better reason than “it sounds absurd.”
Percentages and probability don’t work like that. You can’t just make up percentages for no reason to confidently proclaim a probability of an outcome in the future.
AI killing us all discourse is silly and is promoted by the people who want more funding to align AI with their goals to create the modern version of the steam engine to automate away white collar work. Even if it was a likely event, it’s an appeal to an unknowable, far away event, which distracts from the very real impacts it already is having today.
One of the most pervasive arguments against global warming is that it sounds absurd
I don’t know? Seemed pretty straight forward to me when I was as young as eight.
Probabilities do work like that, if you believe in Bayesian probability anyway. The more we learn about the situation more accurate a probability we can get. The Drake equation works exactly in this way.
It may be true that the discourse is promoted by people who want more funding for AI, but that does not invalidate the point. I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI. And there is overlap between commonplace concerns about AI and existential ones – for instance, a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
Bayesian reasoning is only as reliable as the prior probabilities that go into the algorithm. If you can’t justify your priors, it’s no better than saying “this just feels likely to me” but with a window dressing of mathematics. It’s a great algorithm for updating concrete, known probabilities in the face of new concrete evidence, but that is not at all what’s going on with the vast majority of what the Rationalists do.
Even if you want to use it for estimating the probability of very uncertain events, the uncertainty compounds at each step. Once you get more than a step or two down that path of “let’s say the probability of x is p” without empirical justification, you should have no confidence at all that the number you’re getting bears any relationship to “true” probabilities. Again, it’s just a fancy way of saying “this feels true to me.”
Yes that’s right (that it’s only reliable as the prior probabilities that go into it).
Look at this another way, using the perspective you just shared: before applying bayesian reasoning, one might think that AI as an X-risk sounds super fantastical, and assign it ultra-low probability. But when you break it into constituent components like I did, it starts to sound much more plausible. We’re replacing how one feels intuitively about a certain (improbable-seeming) event with how one feels intuitively about other (more plausible) events. That isn’t a fallacy, that’s actually good off-the-cuff reasoning. Now we can look at whichever of those sounds the most implausible and break it down further.
My goal here isn’t to actually find the exact probability of an AI apocalypse, it’s to raise a warning flag that says “hey, this is more plausible than you might initially think!”
Another user already touched the Bayesian point, so I’m not going to follow that rabbit.
I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI.
Ok? AI will become Skynet is such a popular idea that has permeated society since before I was even born, and I’m guessing before you were, or at least was something you were exposed to in your early years. It’s frankly not an original thought you came up with in 2016, but rather something you and everyone else has inherited from popular media. Saying we need to slow research on AI to align it with “human values” still allows for this idea that we can control AI to not kill us. Moreover, it allows for the idea that only large companies can align the AI to human values, and the “human values” they are currently aligning it with have nothing to do with saving humanity. Instead, the human values are to reinforce dominant classes in society, accelerate climate change through forcing scale as the only path forward (at least until deepseek dropped), and spark mass layoffs as white collar work is automated away.
We’re not going to create a paper clip machine that kills us all because it wants to simply make paper clips. We’re going to make a sophisticated bullshit generator whose primary role is to replace labor. Hopefully, I don’t need to spell out what this means in a capitalist society which is currently free falling into fascism. We’re reaching a point where LLMs have slightly preferable error rates at scale than human workers, and that’s the real danger here.
a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
I’m all for a Butlerian jihad, mount up. I’m not going to join you for a Yudkowskian Jihad, though.
In my view, the danger remains that if the only concern being talked about is AI will kill us all in some fantastical war or apocalyptic scenario, it creates a “hero” (i.e. Sam Altman or some other ghoul) who alone can fix it. The apocalypse argument is not currently pushing anyone towards any moratorium on AI development, but rather just creating a subfield of “alignment” which is more concerned with making sure LLMs don’t say mean things, follow the narrative, and don’t suggest people use irons to smooth out the wrinkles in their balls.
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
This part is tangential, but it actually helps as an allegory to this issue. Exxon new in the late 70s the effects their production would have, that climate change was due to our use of fossil fuels. Rather than act accordingly and pivot away, they protected their profits and muddied the waters by bringing these talking points to media and conservative outlets. Conservatives didn’t organically think this is ridiculous, they were told it was absurd by media empires, and they ate it up and spread it.
I get the feeling you are here in good faith, so if you want to read more about the very real, current, actually happening dangers of AI, I would point you to Atlas of AI, Resisting AI, and the work of Bender and Gebru.
Those are good points, I’ll take a look at the resources you suggested. I think my counter-argument to you right now can basiclaly be summed up as: I do agree that the danger of AI you are talking about is serious and is the more current and pressing concern, but that doesn’t really invalidate the X-risk factor of AI. I am not saying that X-risk is the only risk, and your point warning about a “hero” (which I agree with!) also doesn’t invalidate the concern. I mean, if it turns out that only a heroic space agency can save us from that asteroid, does that mean the threat from the asteroid isn’t real?
Following the asteroid analogy, I view it as this: If there’s a 20% chance that an asteroid could hit us in 2050, does that supplant the threat of climate change today?
I’m not trying to say that AI systems won’t kill us all, just that they are using to directly harm entire populations right now and the appeal to a future danger is being used to minimize that discussion.
Another thing to consider: If an AI system does kill us all, it will still be a human or organization that gave it the ability to do so, whether that be through training practices, or plugging it in to weapons systems. Placing the blame on the AI itself absolves any person or organization of the responsibility, which is in line with how AI is used today (i.e. the promise of algorithmic ‘neutrality’). Put another way, do the bombs kill us all in a nuclear armageddon or do the people who pressed the button? Does the gun kill me, or does the person pulling the trigger?
Your second paragraph first sentence is a nonsense assumption I’m not reading the rest. Your entire premise is fantastical and simply has no basis in how computers work. They can’t be smarter than a person because they don’t have any intelligence to begin with. People can use them to trick rubes like you into believing they do or even potentially can, but they can’t. They don’t output what we don’t input, simple as. You are a simpleton
LLMs in their current form are dangerous to society, but not existential threats
I’m assuming you think LLMs are not dangerous to society? But they’re already putting people out of work. And they can cause havoc on social media. And they just piss me off.
I think we have a semantics disagreement over what “intelligence” means so let’s just put that word away. Machines can already solve problems that are out of human reach (this has been known for a very long time – take the 4-colour theorem for instance, which I think is a pretty clear example of a machine outputting something that wasn’t input). It doesn’t really matter if an AI is “intelligent” if it is capable of writing code to advance a goal that it was given or analyzing a situation and producing a good recommendation about how to proceed in order to advance a goal that was given. LLMs are already capable of this, and if we imagine in 10 years AIs might be 100x better at it, that is already too scary for me.
By the way, I don’t really know what I did to deserve that hostility.
Look, I know you’re arguing in good faith here, but…quite a few of us are very familiar with Big Yud’s arguments, and heck, I’m even partially sympathetic to a couple. There is some motivated suspicion here since rationalists and their singularitarian forebears have a habit of going reactionary (Anissimov, Bostrom, SSC, countless others) or having close ties to reactionaries (Thiel, most notably). I see no reason to believe you are like them but do realise a lot of transhumanist arguments are weaponised as right wing stalking horses.
But as someone who has been following and occasionally interacting with him since our Halcyon Youth on the Extropian mailing list, I’ve learned that
a) Yud can be spectacularly convincing and also spectacularly wrong by his own admission (see early work like “Staring into the Singularity” for this.) so you should cognitively guard against his rhetorical tools.
b) He has a habit of eliding and abstracting concepts so that they appear as a single consistent thing when they are actually several things, while appearing to do the exact opposite. This is most apparent at subjects he is not an expert in, quantum mechanics most notably (I think many worlds is correct, I think his argument for many worlds is bad. I am semi competent at the math of quantum mechanics)
I have specific, nuanced critiques of his main (not entirely invalid) argument on AI doom, but this is generally hard to get across because most of his own supporters don’t fully understand it.
Okay, but this can’t happen for real. If you’re rational you should also be aware that things that appear in fiction also and in majority DO NOT happen irl. The possibility of something isn’t the likelihood of something. It’s possible I could die of an aneurysm in the next 15 minutes, not probable though, and I’m not gonna live in fear of it cause it could potentially happen without weighing that potential. Rockin Basslicks is low as hell on the probability scale and an absolute fucking zero on the I’d care if it was real scale. Same with any simulation idea or hell, even the idea of God. I don’t care if I’m in one or if there is one, there is work to be done making this simulation or created universe better for people in it right here and right now and navel gazing about speculations way beyond what we can understand is a selfish and useless waste of time. Fucking feed the poor.
Yeah Roko’s Basilisk is fiction, it’s not worth worrying about. The danger of AI killing us all is real though.
Explain how
LLMs in their current form are dangerous to society, but not existential threats. To see the danger of human extinction, multiply some probabilities together:
At some point, AIs will likely become super-intelligent, meaning they are smarter than the entire human race put together. Let’s say there’s a 50% chance of this happening in the next 30 years. Something this smart would indeed be capable of doing something very clever and killing all humans (if it wanted to); there are various theories about how it might go about doing that, and if this is the part that sounds outlandish to you then I can elaborate. Needless to say, if something is extraordinarily smarter than you, it can figure out a way to kill you that you didn’t think of, even if they’re sandboxed in a high-security prison. (Mind you, it will probably be exposed like ChatGPT to the general public, not very likely to be sandboxed if you ask me.)
Okay but surely nobody would build an AI that would want to kill us all right? This is the “alignment problem” – we currently don’t know how to make sure an AI has the same “goals” that its creators want it to have. There’s that meme – an AI tasked with optimizing a silverware production process might end up turning the whole universe and everyone in it into spoons. Because almost nobody is actually taking this problem seriously, I think the first superintelligent AI has an 80% chance to be unaligned.
Would an unaligned AI want to kill us? It might be unaligned but conveniently still value human life. Let’s say be generous and say it’s a 50% chance that the unaligned AI works out in our favour. (It’s probably less likely.)
So that’s a 20% chance in the next 30 years that someone will create an AI which is clever enough to kill us all, and (by unhappy accident) wants to kill us all. You can put your own numbers on all of these things and get your own probability. If you put very low numbers (like 0.1%) on any of these steps, you should ask yourself if in 2010 you thought it was likely that AI would be where it is today.
Edit: yes I know it sounds absurd and like fantasy, but a lot of real things sound absurd at first. One of the most pervasive arguments against global warming is that it sounds absurd. So if you’re going to disagree with this, please at least have a better reason than “it sounds absurd.”
Percentages and probability don’t work like that. You can’t just make up percentages for no reason to confidently proclaim a probability of an outcome in the future.
AI killing us all discourse is silly and is promoted by the people who want more funding to align AI with their goals to create the modern version of the steam engine to automate away white collar work. Even if it was a likely event, it’s an appeal to an unknowable, far away event, which distracts from the very real impacts it already is having today.
I don’t know? Seemed pretty straight forward to me when I was as young as eight.
Probabilities do work like that, if you believe in Bayesian probability anyway. The more we learn about the situation more accurate a probability we can get. The Drake equation works exactly in this way.
It may be true that the discourse is promoted by people who want more funding for AI, but that does not invalidate the point. I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI. And there is overlap between commonplace concerns about AI and existential ones – for instance, a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
Bayesian reasoning is only as reliable as the prior probabilities that go into the algorithm. If you can’t justify your priors, it’s no better than saying “this just feels likely to me” but with a window dressing of mathematics. It’s a great algorithm for updating concrete, known probabilities in the face of new concrete evidence, but that is not at all what’s going on with the vast majority of what the Rationalists do.
Even if you want to use it for estimating the probability of very uncertain events, the uncertainty compounds at each step. Once you get more than a step or two down that path of “let’s say the probability of x is p” without empirical justification, you should have no confidence at all that the number you’re getting bears any relationship to “true” probabilities. Again, it’s just a fancy way of saying “this feels true to me.”
Yes that’s right (that it’s only reliable as the prior probabilities that go into it).
Look at this another way, using the perspective you just shared: before applying bayesian reasoning, one might think that AI as an X-risk sounds super fantastical, and assign it ultra-low probability. But when you break it into constituent components like I did, it starts to sound much more plausible. We’re replacing how one feels intuitively about a certain (improbable-seeming) event with how one feels intuitively about other (more plausible) events. That isn’t a fallacy, that’s actually good off-the-cuff reasoning. Now we can look at whichever of those sounds the most implausible and break it down further.
My goal here isn’t to actually find the exact probability of an AI apocalypse, it’s to raise a warning flag that says “hey, this is more plausible than you might initially think!”
Another user already touched the Bayesian point, so I’m not going to follow that rabbit.
Ok? AI will become Skynet is such a popular idea that has permeated society since before I was even born, and I’m guessing before you were, or at least was something you were exposed to in your early years. It’s frankly not an original thought you came up with in 2016, but rather something you and everyone else has inherited from popular media. Saying we need to slow research on AI to align it with “human values” still allows for this idea that we can control AI to not kill us. Moreover, it allows for the idea that only large companies can align the AI to human values, and the “human values” they are currently aligning it with have nothing to do with saving humanity. Instead, the human values are to reinforce dominant classes in society, accelerate climate change through forcing scale as the only path forward (at least until deepseek dropped), and spark mass layoffs as white collar work is automated away.
We’re not going to create a paper clip machine that kills us all because it wants to simply make paper clips. We’re going to make a sophisticated bullshit generator whose primary role is to replace labor. Hopefully, I don’t need to spell out what this means in a capitalist society which is currently free falling into fascism. We’re reaching a point where LLMs have slightly preferable error rates at scale than human workers, and that’s the real danger here.
I’m all for a Butlerian jihad, mount up. I’m not going to join you for a Yudkowskian Jihad, though.
In my view, the danger remains that if the only concern being talked about is AI will kill us all in some fantastical war or apocalyptic scenario, it creates a “hero” (i.e. Sam Altman or some other ghoul) who alone can fix it. The apocalypse argument is not currently pushing anyone towards any moratorium on AI development, but rather just creating a subfield of “alignment” which is more concerned with making sure LLMs don’t say mean things, follow the narrative, and don’t suggest people use irons to smooth out the wrinkles in their balls.
This part is tangential, but it actually helps as an allegory to this issue. Exxon new in the late 70s the effects their production would have, that climate change was due to our use of fossil fuels. Rather than act accordingly and pivot away, they protected their profits and muddied the waters by bringing these talking points to media and conservative outlets. Conservatives didn’t organically think this is ridiculous, they were told it was absurd by media empires, and they ate it up and spread it.
I get the feeling you are here in good faith, so if you want to read more about the very real, current, actually happening dangers of AI, I would point you to Atlas of AI, Resisting AI, and the work of Bender and Gebru.
Those are good points, I’ll take a look at the resources you suggested. I think my counter-argument to you right now can basiclaly be summed up as: I do agree that the danger of AI you are talking about is serious and is the more current and pressing concern, but that doesn’t really invalidate the X-risk factor of AI. I am not saying that X-risk is the only risk, and your point warning about a “hero” (which I agree with!) also doesn’t invalidate the concern. I mean, if it turns out that only a heroic space agency can save us from that asteroid, does that mean the threat from the asteroid isn’t real?
Following the asteroid analogy, I view it as this: If there’s a 20% chance that an asteroid could hit us in 2050, does that supplant the threat of climate change today?
I’m not trying to say that AI systems won’t kill us all, just that they are using to directly harm entire populations right now and the appeal to a future danger is being used to minimize that discussion.
Another thing to consider: If an AI system does kill us all, it will still be a human or organization that gave it the ability to do so, whether that be through training practices, or plugging it in to weapons systems. Placing the blame on the AI itself absolves any person or organization of the responsibility, which is in line with how AI is used today (i.e. the promise of algorithmic ‘neutrality’). Put another way, do the bombs kill us all in a nuclear armageddon or do the people who pressed the button? Does the gun kill me, or does the person pulling the trigger?
Your second paragraph first sentence is a nonsense assumption I’m not reading the rest. Your entire premise is fantastical and simply has no basis in how computers work. They can’t be smarter than a person because they don’t have any intelligence to begin with. People can use them to trick rubes like you into believing they do or even potentially can, but they can’t. They don’t output what we don’t input, simple as. You are a simpleton
I’m assuming you think LLMs are not dangerous to society? But they’re already putting people out of work. And they can cause havoc on social media. And they just piss me off.
Second paragraph first sentence.
Oops my bad.
I think we have a semantics disagreement over what “intelligence” means so let’s just put that word away. Machines can already solve problems that are out of human reach (this has been known for a very long time – take the 4-colour theorem for instance, which I think is a pretty clear example of a machine outputting something that wasn’t input). It doesn’t really matter if an AI is “intelligent” if it is capable of writing code to advance a goal that it was given or analyzing a situation and producing a good recommendation about how to proceed in order to advance a goal that was given. LLMs are already capable of this, and if we imagine in 10 years AIs might be 100x better at it, that is already too scary for me.
By the way, I don’t really know what I did to deserve that hostility.
Look, I know you’re arguing in good faith here, but…quite a few of us are very familiar with Big Yud’s arguments, and heck, I’m even partially sympathetic to a couple. There is some motivated suspicion here since rationalists and their singularitarian forebears have a habit of going reactionary (Anissimov, Bostrom, SSC, countless others) or having close ties to reactionaries (Thiel, most notably). I see no reason to believe you are like them but do realise a lot of transhumanist arguments are weaponised as right wing stalking horses.
But as someone who has been following and occasionally interacting with him since our Halcyon Youth on the Extropian mailing list, I’ve learned that
a) Yud can be spectacularly convincing and also spectacularly wrong by his own admission (see early work like “Staring into the Singularity” for this.) so you should cognitively guard against his rhetorical tools.
b) He has a habit of eliding and abstracting concepts so that they appear as a single consistent thing when they are actually several things, while appearing to do the exact opposite. This is most apparent at subjects he is not an expert in, quantum mechanics most notably (I think many worlds is correct, I think his argument for many worlds is bad. I am semi competent at the math of quantum mechanics)
I have specific, nuanced critiques of his main (not entirely invalid) argument on AI doom, but this is generally hard to get across because most of his own supporters don’t fully understand it.
Because you’re a silly person fearing silly things