I’m guessing you’re not a programmer yourself? Because it’s really really not that east to /just/ detect in the server side, hacks can be super sofisticsted these days and there are often many client side exploits that you simply cannot detect serverside.
Using rootkit anti-cheat is a shortcut that reduces cost for both dev time and hosting time at the expense of your customers’ security and CPU. You also have to lay your cards on the table for those who are attacking you. It is not the right solution for this problem.
Authoritative servers.
Never trust the client, especially with information the player shouldn’t have right now.
Look at behaviors and group players based on if you think they cheat or not - let the cheaters play together, no need to spoil their fun and let them realize you know they cheat.
People do some or all of this on the server now, but root kitting all machines to try to solve this problem to play video games is one of the dumbest approaches ever and we will realize it one day when a state level actor pops their zero day against a big install base.
This. Having worked on some in-house anti-cheat solutions myself, it absolutely is just offsetting the processing and security cost to the players. The attack vector of having such a rootkit running on so many devices is just not even close to be worth the trade off of catching marginally (if really measurably at all?) more cheaters.
But… have you considered having control of 0-ring software that runs on hundreds of millions of computers, that can perform targetted updates to change behaviour on just a select few computers, even interact with the network adapters unbeknownst to the OS.
I’m not talking about zero days popping up for this. But rather, this being part of the design?
A less nefarious application: The root kit anti cheats already continuously monitor processes. Say it finds a crypto mining one. It can request the instructions needed to search for a wallet and snatch that off.
A more nefarious one: RK is known to be in the device owned by the kid of a military contractor. Etc.
Trusting the client is a fools errand. So we are in complete agreement. I never understood why the effort isn’t placed on server side. People are very good at knowing when others have cheated. They know this from information that exists on the server side, so with the correct classifier, the server should also be able to know this.
Never trust the client, especially with information the player shouldn’t have right now.
This is a big part of the problem, but it’s not the only problem. If you do all of that stuff right, you can’t build a responsive first person shooter. There’s some level of trust you need to put in the client.
Disclaimer: This is based on my experience playing shooters and as a programmer. I have not worked on anticheat systems hands on.
We see less and less of the “god mode” hacks where players can send the packet for a carpet bomb and the server just blindly trusts it. Or the ludicrous spinbots that spin at an extreme speed and headshot anyone that comes into line of sight.
What we’re seeing is increasingly sophisticated cheats that provide “buffs” to a player’s ability. An AI enhanced aimbot that when you click gently nudges your hand to “auto correct” the shot and then clicks is borderline impossible to detect server side. It looks just like a player moved the mouse and fired.
The “best” method to prevent these folks from cheating seems to be to detect the system or the game has been tampered with.
Maybe the way to deal with that is to just let it happen and deal with smurfs down ranking… So these “soft” cheaters just exist in the “pro tier” where the pros can possibly stand a chance.
One strategy I have seen that I wish more developers would do is sending “honeypot” information to the game client (like a player on the other side of the wall that isn’t really there but an aimbot or a wall hack might incorrectly expose).
Maybe the increasing presence of hardware cheats will result in new strategies that make these things unnecessary. I keep wondering if a TPM could be used to solve this problem someday… But I’m not sure exactly how/we may need faster TPMs.
I think by the end of your message you were starting to arc around a little bit to the right way you need to think about clients: as outside your security envelope. (TPM is a joke in my mind, just like client side anti-cheat.)
There are many ways to try to identify and stop cheating on the server side that have not been explored because executives have directed use of off-the-shelf anti-cheat because they do not understand why it is snake oil.
I thought this at first as well, but they have an interesting property.
They have a manufacturer signed private key. If you get the public key from the manufacturer of the TPM, you can actually verify that the TPM as it was designed by the manufacturer performed the work.
That’s a really interesting property because for the first time there’s a way to verify what hardware is doing over the network via cryptography.
Hmmm… I was going to say no because it’s asymmetric crypto, but you’re right if you are somehow able to extract the signed private key, you can still lie… Good point
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
Even on consoles after a while it becomes obvious that you cannot control the hardware, let alone the software on the client side. Those are the very best argument for this kind of approach and they get cracked eventually.
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
That’s true, if the player suddenly has higher kill rates. However, that doesn’t work if they’ve been using the cheat from the start on that account. A sufficiently advanced AI powered aim bot would also be nearly indistinguishable from a professional player. Kind of similar to how Google created the CAPTCHA that uses mouse movement … but had to go back to (at least in some cases) the additional old school captcha.
I agree with your points, but I do not think there’s any chance for that.
Riot is a Chinese company for many years now. It’s not hard to imagine which state actor will abuse the kernel access. No way they will do anything about that unless they suddenly lose a significant fraction of players.
I’m a programmer, yes it is. It’s not easy in the sense of easy to implement, it’s easy in the sense that everything else is impossible. Client-side anti-cheat is impossible, and by that I don’t mean hard, I mean perpetual-motion level of impossibility. If someone tells you they implemented a foolproof client-side anti-cheat you should be just as skeptical as if someone tells you they created a perpetual motion. It’s impossible, never going to happen, want an example? Robot using a camera to watch the screen and directly moving the mouse and keyboard, completely undetectable from the client side.
From the server perspective the person is cheating or is behaving like a human. If they’re behaving like a human their behavior is completely indistinguishable from a human, so who cares if they’re cheating?, whatever they’re doing has them still at human level so if the game has skill based matchmaking (which most of these games do) he’ll rise up until his cheating puts him in the same level of more skilled humans and everyone has fun. If he keeps rising forever he’s not on a human level, therefore a cheater. More importantly this also penalizes people who buy bot leveled accounts, because their matches will be all against people they can’t hope to win and the game will not be fun.
Server side can also trick clients into giving up that they’re cheating, e.g. sending ghosts behind walls to check for wall hacks or other similar things to gauge player responses.
But what do I know? I’m just a senior programmer who’s been working on servers for some years. l never worked on the client side anti-cheat though, also never tried to build a perpetual motion machine.
It’s not easy, but it’s really not worth the massive gaping security vulnerability you are giving your users. One disgruntled employee giving out the keys to the castle or one programmer plugging in an infected USB, and every user now has a persistent malicious rootkit. The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
This can’t be right.
Don’t throw your hard drive in the trash. Quarantine the infected computer, and then wipe that hoe and slap your choice of OS back on it and scan/monitor to see if any issues arise.
Edit: since folks may or may not read though the rest of the conversation: I am wrong, throw that SSD/HDD in the garbage like barbarian said.
I’m sorry to disappoint, but with rootkits, that is very real. With that level of permissions, it can rewrite HDD/SSD drivers to install malware on boot.
There’s even malware that can rewrite BIOS/UEFI, in which case the whole motherboard has to go in the bin. That’s much less likely due to the complexity though, but it does exist.
Outside of monitoring individual packets outside of your computer (as in, man in the middle yourself with a spare computer and hoping the malware phones home right when you’re looking) there’s no way of knowing.
Once ring 0 is compromised, nothing your computer says can be trusted. A compromised OS can lie to anti-malware scanners, hide things from the installed software list and process manager, and just generally not show you what it doesnt want to show you. “Just remediate” does not work with rootkits.
Dude… That’s fucked. They should really go a little more in depth on rootkits in the CompTIA A+ study material. I mean, I get that it’s supposed to be a foundational over view of most IT concepts, but it would have helped me not look dumb.
Please don’t walk away from this feeling dumb. Most IT professionals aren’t aware of the scale of the issue outside of sysadmin and cybersecurity. I’ve met programmers who shrug at the most egregious vulnerabilities, and vendors who want us to put dangerous stuff on our servers. Security just isn’t taken as seriously as it should be.
Unrelated, but I wish you the best of luck with your studies!
Good morning! If anything this was a great example of not being able to know everything when it comes to IT and especially cybersecurity. Thank you for your well wishes! I earned my A+ last month and I’m currently working on a Google cybersec certificate, since it’ll give me 30% off on the sec+ exam price. I really appreciate your insight on rootkits and it’s definitely going in my notes!
Could they harden their clients somehow or maybe randomize memory locations for things? Seems like their should be a better solution than installing malware to prevent cheating.
You’re asking good questions but factor this in: a development team at a game company will only want to spend as little time as possible on this process: it doesn’t make them more money - it costs it. Conversely a hacker / cheater is being paid (or gaining) directly from breaking this code. Which is more motivated? Now remember that the protection has to be in place first. Who has the advantage? Client side code will always be breakable. A rootkit doesn’t change the game - it just adds a new vector to attack for other hackers to exploit.
I’m guessing you’re not a programmer yourself? Because it’s really really not that east to /just/ detect in the server side, hacks can be super sofisticsted these days and there are often many client side exploits that you simply cannot detect serverside.
Actually, I am.
Using rootkit anti-cheat is a shortcut that reduces cost for both dev time and hosting time at the expense of your customers’ security and CPU. You also have to lay your cards on the table for those who are attacking you. It is not the right solution for this problem.
Authoritative servers. Never trust the client, especially with information the player shouldn’t have right now. Look at behaviors and group players based on if you think they cheat or not - let the cheaters play together, no need to spoil their fun and let them realize you know they cheat.
People do some or all of this on the server now, but root kitting all machines to try to solve this problem to play video games is one of the dumbest approaches ever and we will realize it one day when a state level actor pops their zero day against a big install base.
This. Having worked on some in-house anti-cheat solutions myself, it absolutely is just offsetting the processing and security cost to the players. The attack vector of having such a rootkit running on so many devices is just not even close to be worth the trade off of catching marginally (if really measurably at all?) more cheaters.
But… have you considered having control of 0-ring software that runs on hundreds of millions of computers, that can perform targetted updates to change behaviour on just a select few computers, even interact with the network adapters unbeknownst to the OS.
I’m not talking about zero days popping up for this. But rather, this being part of the design?
A less nefarious application: The root kit anti cheats already continuously monitor processes. Say it finds a crypto mining one. It can request the instructions needed to search for a wallet and snatch that off.
A more nefarious one: RK is known to be in the device owned by the kid of a military contractor. Etc.
Trusting the client is a fools errand. So we are in complete agreement. I never understood why the effort isn’t placed on server side. People are very good at knowing when others have cheated. They know this from information that exists on the server side, so with the correct classifier, the server should also be able to know this.
This is a big part of the problem, but it’s not the only problem. If you do all of that stuff right, you can’t build a responsive first person shooter. There’s some level of trust you need to put in the client.
Disclaimer: This is based on my experience playing shooters and as a programmer. I have not worked on anticheat systems hands on.
We see less and less of the “god mode” hacks where players can send the packet for a carpet bomb and the server just blindly trusts it. Or the ludicrous spinbots that spin at an extreme speed and headshot anyone that comes into line of sight.
What we’re seeing is increasingly sophisticated cheats that provide “buffs” to a player’s ability. An AI enhanced aimbot that when you click gently nudges your hand to “auto correct” the shot and then clicks is borderline impossible to detect server side. It looks just like a player moved the mouse and fired.
The “best” method to prevent these folks from cheating seems to be to detect the system or the game has been tampered with.
Maybe the way to deal with that is to just let it happen and deal with smurfs down ranking… So these “soft” cheaters just exist in the “pro tier” where the pros can possibly stand a chance.
One strategy I have seen that I wish more developers would do is sending “honeypot” information to the game client (like a player on the other side of the wall that isn’t really there but an aimbot or a wall hack might incorrectly expose).
Maybe the increasing presence of hardware cheats will result in new strategies that make these things unnecessary. I keep wondering if a TPM could be used to solve this problem someday… But I’m not sure exactly how/we may need faster TPMs.
I think by the end of your message you were starting to arc around a little bit to the right way you need to think about clients: as outside your security envelope. (TPM is a joke in my mind, just like client side anti-cheat.)
There are many ways to try to identify and stop cheating on the server side that have not been explored because executives have directed use of off-the-shelf anti-cheat because they do not understand why it is snake oil.
I thought this at first as well, but they have an interesting property.
They have a manufacturer signed private key. If you get the public key from the manufacturer of the TPM, you can actually verify that the TPM as it was designed by the manufacturer performed the work.
That’s a really interesting property because for the first time there’s a way to verify what hardware is doing over the network via cryptography.
Or, if I can extract that key from the hardware, I can pretend to be that hardware whenever I want, right?
Hmmm… I was going to say no because it’s asymmetric crypto, but you’re right if you are somehow able to extract the signed private key, you can still lie… Good point
Got some bad news. They already can do that. It’s a very low effort attack too. Current TPM spits its key out in clear text. Funny right?
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
Even on consoles after a while it becomes obvious that you cannot control the hardware, let alone the software on the client side. Those are the very best argument for this kind of approach and they get cracked eventually.
That’s true, if the player suddenly has higher kill rates. However, that doesn’t work if they’ve been using the cheat from the start on that account. A sufficiently advanced AI powered aim bot would also be nearly indistinguishable from a professional player. Kind of similar to how Google created the CAPTCHA that uses mouse movement … but had to go back to (at least in some cases) the additional old school captcha.
I agree with your points, but I do not think there’s any chance for that.
Riot is a Chinese company for many years now. It’s not hard to imagine which state actor will abuse the kernel access. No way they will do anything about that unless they suddenly lose a significant fraction of players.
I’m a programmer, yes it is. It’s not easy in the sense of easy to implement, it’s easy in the sense that everything else is impossible. Client-side anti-cheat is impossible, and by that I don’t mean hard, I mean perpetual-motion level of impossibility. If someone tells you they implemented a foolproof client-side anti-cheat you should be just as skeptical as if someone tells you they created a perpetual motion. It’s impossible, never going to happen, want an example? Robot using a camera to watch the screen and directly moving the mouse and keyboard, completely undetectable from the client side.
From the server perspective the person is cheating or is behaving like a human. If they’re behaving like a human their behavior is completely indistinguishable from a human, so who cares if they’re cheating?, whatever they’re doing has them still at human level so if the game has skill based matchmaking (which most of these games do) he’ll rise up until his cheating puts him in the same level of more skilled humans and everyone has fun. If he keeps rising forever he’s not on a human level, therefore a cheater. More importantly this also penalizes people who buy bot leveled accounts, because their matches will be all against people they can’t hope to win and the game will not be fun.
Server side can also trick clients into giving up that they’re cheating, e.g. sending ghosts behind walls to check for wall hacks or other similar things to gauge player responses.
But what do I know? I’m just a senior programmer who’s been working on servers for some years. l never worked on the client side anti-cheat though, also never tried to build a perpetual motion machine.
It’s not easy, but it’s really not worth the massive gaping security vulnerability you are giving your users. One disgruntled employee giving out the keys to the castle or one programmer plugging in an infected USB, and every user now has a persistent malicious rootkit. The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
This can’t be right.
Don’t throw your hard drive in the trash. Quarantine the infected computer, and then wipe that hoe and slap your choice of OS back on it and scan/monitor to see if any issues arise.
Edit: since folks may or may not read though the rest of the conversation: I am wrong, throw that SSD/HDD in the garbage like barbarian said.
I’m sorry to disappoint, but with rootkits, that is very real. With that level of permissions, it can rewrite HDD/SSD drivers to install malware on boot.
There’s even malware that can rewrite BIOS/UEFI, in which case the whole motherboard has to go in the bin. That’s much less likely due to the complexity though, but it does exist.
not all rootkits are made to do that. So yes in some cases, throw it in the trash. In others, remediate your machine and move on.
Outside of monitoring individual packets outside of your computer (as in, man in the middle yourself with a spare computer and hoping the malware phones home right when you’re looking) there’s no way of knowing.
Once ring 0 is compromised, nothing your computer says can be trusted. A compromised OS can lie to anti-malware scanners, hide things from the installed software list and process manager, and just generally not show you what it doesnt want to show you. “Just remediate” does not work with rootkits.
Dude… That’s fucked. They should really go a little more in depth on rootkits in the CompTIA A+ study material. I mean, I get that it’s supposed to be a foundational over view of most IT concepts, but it would have helped me not look dumb.
Please don’t walk away from this feeling dumb. Most IT professionals aren’t aware of the scale of the issue outside of sysadmin and cybersecurity. I’ve met programmers who shrug at the most egregious vulnerabilities, and vendors who want us to put dangerous stuff on our servers. Security just isn’t taken as seriously as it should be.
Unrelated, but I wish you the best of luck with your studies!
Good morning! If anything this was a great example of not being able to know everything when it comes to IT and especially cybersecurity. Thank you for your well wishes! I earned my A+ last month and I’m currently working on a Google cybersec certificate, since it’ll give me 30% off on the sec+ exam price. I really appreciate your insight on rootkits and it’s definitely going in my notes!
Could they harden their clients somehow or maybe randomize memory locations for things? Seems like their should be a better solution than installing malware to prevent cheating.
You’re asking good questions but factor this in: a development team at a game company will only want to spend as little time as possible on this process: it doesn’t make them more money - it costs it. Conversely a hacker / cheater is being paid (or gaining) directly from breaking this code. Which is more motivated? Now remember that the protection has to be in place first. Who has the advantage? Client side code will always be breakable. A rootkit doesn’t change the game - it just adds a new vector to attack for other hackers to exploit.