Probabilities do work like that, if you believe in Bayesian probability anyway. The more we learn about the situation more accurate a probability we can get. The Drake equation works exactly in this way.
It may be true that the discourse is promoted by people who want more funding for AI, but that does not invalidate the point. I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI. And there is overlap between commonplace concerns about AI and existential ones – for instance, a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
Bayesian reasoning is only as reliable as the prior probabilities that go into the algorithm. If you can’t justify your priors, it’s no better than saying “this just feels likely to me” but with a window dressing of mathematics. It’s a great algorithm for updating concrete, known probabilities in the face of new concrete evidence, but that is not at all what’s going on with the vast majority of what the Rationalists do.
Even if you want to use it for estimating the probability of very uncertain events, the uncertainty compounds at each step. Once you get more than a step or two down that path of “let’s say the probability of x is p” without empirical justification, you should have no confidence at all that the number you’re getting bears any relationship to “true” probabilities. Again, it’s just a fancy way of saying “this feels true to me.”
Yes that’s right (that it’s only reliable as the prior probabilities that go into it).
Look at this another way, using the perspective you just shared: before applying bayesian reasoning, one might think that AI as an X-risk sounds super fantastical, and assign it ultra-low probability. But when you break it into constituent components like I did, it starts to sound much more plausible. We’re replacing how one feels intuitively about a certain (improbable-seeming) event with how one feels intuitively about other (more plausible) events. That isn’t a fallacy, that’s actually good off-the-cuff reasoning. Now we can look at whichever of those sounds the most implausible and break it down further.
My goal here isn’t to actually find the exact probability of an AI apocalypse, it’s to raise a warning flag that says “hey, this is more plausible than you might initially think!”
Another user already touched the Bayesian point, so I’m not going to follow that rabbit.
I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI.
Ok? AI will become Skynet is such a popular idea that has permeated society since before I was even born, and I’m guessing before you were, or at least was something you were exposed to in your early years. It’s frankly not an original thought you came up with in 2016, but rather something you and everyone else has inherited from popular media. Saying we need to slow research on AI to align it with “human values” still allows for this idea that we can control AI to not kill us. Moreover, it allows for the idea that only large companies can align the AI to human values, and the “human values” they are currently aligning it with have nothing to do with saving humanity. Instead, the human values are to reinforce dominant classes in society, accelerate climate change through forcing scale as the only path forward (at least until deepseek dropped), and spark mass layoffs as white collar work is automated away.
We’re not going to create a paper clip machine that kills us all because it wants to simply make paper clips. We’re going to make a sophisticated bullshit generator whose primary role is to replace labor. Hopefully, I don’t need to spell out what this means in a capitalist society which is currently free falling into fascism. We’re reaching a point where LLMs have slightly preferable error rates at scale than human workers, and that’s the real danger here.
a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
I’m all for a Butlerian jihad, mount up. I’m not going to join you for a Yudkowskian Jihad, though.
In my view, the danger remains that if the only concern being talked about is AI will kill us all in some fantastical war or apocalyptic scenario, it creates a “hero” (i.e. Sam Altman or some other ghoul) who alone can fix it. The apocalypse argument is not currently pushing anyone towards any moratorium on AI development, but rather just creating a subfield of “alignment” which is more concerned with making sure LLMs don’t say mean things, follow the narrative, and don’t suggest people use irons to smooth out the wrinkles in their balls.
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
This part is tangential, but it actually helps as an allegory to this issue. Exxon new in the late 70s the effects their production would have, that climate change was due to our use of fossil fuels. Rather than act accordingly and pivot away, they protected their profits and muddied the waters by bringing these talking points to media and conservative outlets. Conservatives didn’t organically think this is ridiculous, they were told it was absurd by media empires, and they ate it up and spread it.
I get the feeling you are here in good faith, so if you want to read more about the very real, current, actually happening dangers of AI, I would point you to Atlas of AI, Resisting AI, and the work of Bender and Gebru.
Those are good points, I’ll take a look at the resources you suggested. I think my counter-argument to you right now can basiclaly be summed up as: I do agree that the danger of AI you are talking about is serious and is the more current and pressing concern, but that doesn’t really invalidate the X-risk factor of AI. I am not saying that X-risk is the only risk, and your point warning about a “hero” (which I agree with!) also doesn’t invalidate the concern. I mean, if it turns out that only a heroic space agency can save us from that asteroid, does that mean the threat from the asteroid isn’t real?
Following the asteroid analogy, I view it as this: If there’s a 20% chance that an asteroid could hit us in 2050, does that supplant the threat of climate change today?
I’m not trying to say that AI systems won’t kill us all, just that they are using to directly harm entire populations right now and the appeal to a future danger is being used to minimize that discussion.
Another thing to consider: If an AI system does kill us all, it will still be a human or organization that gave it the ability to do so, whether that be through training practices, or plugging it in to weapons systems. Placing the blame on the AI itself absolves any person or organization of the responsibility, which is in line with how AI is used today (i.e. the promise of algorithmic ‘neutrality’). Put another way, do the bombs kill us all in a nuclear armageddon or do the people who pressed the button? Does the gun kill me, or does the person pulling the trigger?
I think we completely agree – there can be both a 20% threat of extinction and also the threat of climate change
no, I don’t agree with this; that’s like saying the threat of the asteroid is used to supplant the threat of climate change. The X-risk threat of AI does not invalidate the other threats of AI, and I disagree with anyone who thinks it does. I have not seen anyone use the X-risk threat of AI to invalidate the other threats of AI, and I implore you to not let such an argument sway you for or against either of those threats, which are both real (!).
I do not blame the gun, I blame the manufacturer. I am calling for more oversight over AI companies and for people who research AI to take this threat more seriously. If an AI apocalypse happens, it will of course be the fault of the idiotic AI development companies who did not take this threat seriously because they were blinded by profits. What did I say that made you think I was blaming the AI itself?
Probabilities do work like that, if you believe in Bayesian probability anyway. The more we learn about the situation more accurate a probability we can get. The Drake equation works exactly in this way.
It may be true that the discourse is promoted by people who want more funding for AI, but that does not invalidate the point. I was on board with this concern since around 2016 or so, long before LLMs, and I don’t have a vested interest in AI. And there is overlap between commonplace concerns about AI and existential ones – for instance, a moratorium would advance both goals. Frankly, if people see AI as an existential threat, that should be a great boon for other anti-AI parties, no?
Global warming made sense to me when I was 8 too, but it’s a common talking point among conservatives that it’s ludicrous to suggest that humans could have an impact on something as large as the planet as a whole.
Bayesian reasoning is only as reliable as the prior probabilities that go into the algorithm. If you can’t justify your priors, it’s no better than saying “this just feels likely to me” but with a window dressing of mathematics. It’s a great algorithm for updating concrete, known probabilities in the face of new concrete evidence, but that is not at all what’s going on with the vast majority of what the Rationalists do.
Even if you want to use it for estimating the probability of very uncertain events, the uncertainty compounds at each step. Once you get more than a step or two down that path of “let’s say the probability of x is p” without empirical justification, you should have no confidence at all that the number you’re getting bears any relationship to “true” probabilities. Again, it’s just a fancy way of saying “this feels true to me.”
Yes that’s right (that it’s only reliable as the prior probabilities that go into it).
Look at this another way, using the perspective you just shared: before applying bayesian reasoning, one might think that AI as an X-risk sounds super fantastical, and assign it ultra-low probability. But when you break it into constituent components like I did, it starts to sound much more plausible. We’re replacing how one feels intuitively about a certain (improbable-seeming) event with how one feels intuitively about other (more plausible) events. That isn’t a fallacy, that’s actually good off-the-cuff reasoning. Now we can look at whichever of those sounds the most implausible and break it down further.
My goal here isn’t to actually find the exact probability of an AI apocalypse, it’s to raise a warning flag that says “hey, this is more plausible than you might initially think!”
Another user already touched the Bayesian point, so I’m not going to follow that rabbit.
Ok? AI will become Skynet is such a popular idea that has permeated society since before I was even born, and I’m guessing before you were, or at least was something you were exposed to in your early years. It’s frankly not an original thought you came up with in 2016, but rather something you and everyone else has inherited from popular media. Saying we need to slow research on AI to align it with “human values” still allows for this idea that we can control AI to not kill us. Moreover, it allows for the idea that only large companies can align the AI to human values, and the “human values” they are currently aligning it with have nothing to do with saving humanity. Instead, the human values are to reinforce dominant classes in society, accelerate climate change through forcing scale as the only path forward (at least until deepseek dropped), and spark mass layoffs as white collar work is automated away.
We’re not going to create a paper clip machine that kills us all because it wants to simply make paper clips. We’re going to make a sophisticated bullshit generator whose primary role is to replace labor. Hopefully, I don’t need to spell out what this means in a capitalist society which is currently free falling into fascism. We’re reaching a point where LLMs have slightly preferable error rates at scale than human workers, and that’s the real danger here.
I’m all for a Butlerian jihad, mount up. I’m not going to join you for a Yudkowskian Jihad, though.
In my view, the danger remains that if the only concern being talked about is AI will kill us all in some fantastical war or apocalyptic scenario, it creates a “hero” (i.e. Sam Altman or some other ghoul) who alone can fix it. The apocalypse argument is not currently pushing anyone towards any moratorium on AI development, but rather just creating a subfield of “alignment” which is more concerned with making sure LLMs don’t say mean things, follow the narrative, and don’t suggest people use irons to smooth out the wrinkles in their balls.
This part is tangential, but it actually helps as an allegory to this issue. Exxon new in the late 70s the effects their production would have, that climate change was due to our use of fossil fuels. Rather than act accordingly and pivot away, they protected their profits and muddied the waters by bringing these talking points to media and conservative outlets. Conservatives didn’t organically think this is ridiculous, they were told it was absurd by media empires, and they ate it up and spread it.
I get the feeling you are here in good faith, so if you want to read more about the very real, current, actually happening dangers of AI, I would point you to Atlas of AI, Resisting AI, and the work of Bender and Gebru.
Those are good points, I’ll take a look at the resources you suggested. I think my counter-argument to you right now can basiclaly be summed up as: I do agree that the danger of AI you are talking about is serious and is the more current and pressing concern, but that doesn’t really invalidate the X-risk factor of AI. I am not saying that X-risk is the only risk, and your point warning about a “hero” (which I agree with!) also doesn’t invalidate the concern. I mean, if it turns out that only a heroic space agency can save us from that asteroid, does that mean the threat from the asteroid isn’t real?
Following the asteroid analogy, I view it as this: If there’s a 20% chance that an asteroid could hit us in 2050, does that supplant the threat of climate change today?
I’m not trying to say that AI systems won’t kill us all, just that they are using to directly harm entire populations right now and the appeal to a future danger is being used to minimize that discussion.
Another thing to consider: If an AI system does kill us all, it will still be a human or organization that gave it the ability to do so, whether that be through training practices, or plugging it in to weapons systems. Placing the blame on the AI itself absolves any person or organization of the responsibility, which is in line with how AI is used today (i.e. the promise of algorithmic ‘neutrality’). Put another way, do the bombs kill us all in a nuclear armageddon or do the people who pressed the button? Does the gun kill me, or does the person pulling the trigger?
On each of your paragraphs:
I think we completely agree – there can be both a 20% threat of extinction and also the threat of climate change
no, I don’t agree with this; that’s like saying the threat of the asteroid is used to supplant the threat of climate change. The X-risk threat of AI does not invalidate the other threats of AI, and I disagree with anyone who thinks it does. I have not seen anyone use the X-risk threat of AI to invalidate the other threats of AI, and I implore you to not let such an argument sway you for or against either of those threats, which are both real (!).
I do not blame the gun, I blame the manufacturer. I am calling for more oversight over AI companies and for people who research AI to take this threat more seriously. If an AI apocalypse happens, it will of course be the fault of the idiotic AI development companies who did not take this threat seriously because they were blinded by profits. What did I say that made you think I was blaming the AI itself?