Always call out Cloudflare for their bullshit. For those working for companies in devops, share this with your teams…
They were in violation of the TOS for abusing CF’s IPs with site rotation to circumvent IP bans to their online casino. They need an enterprise plan to BYOIP with their level of traffic. They were given 48 hours notice of site deletion but were given almost 2 weeks before doing so. Read the comment at the bottom of the substack post for further detail.
If that’s the case, and from what I read it could be, then I still blame cloudflare for 2 big things. First communication, because they clearly were confused about what was happening and felt like they didn’t have anyone technical explain it to them and it felt like a sales pitch. Second is still communcation, but an offramp plan. You have 1 week to come into compliance, and we can tell you exactly what is not in compliance, and then your services will be terminated. They gave them a very, very short timeline, did not tell them exactly what was out of compliance, and then just turned it off.
As someone who has accidentally been on the wrong side of TOS before, it’s a nightmare. These large corporations don’t tell you what you’re doing wrong, or where the issue even is, they just say “You’re suspended, gtfo”. That has happened to me for personal accounts, I can’t imagine what it’s like when your business depends on it.
First communication, because they clearly were confused about what was happening and felt like they didn’t have anyone technical explain it to them and it felt like a sales pitch.
I don’t think that was the case.
The substack post is a one-sided and very partial account, and one that doesn’t pass the smell test. They use an awful lot of weasel worlds and leave about whole accounts on what has been discussed with cloud flare in meetings summoned with a matter of urgency.
Occam’s razor suggests they were intentionally involved in multiple layers of abuse, were told to stop it, ignored all warnings, and once the consequences hit they decided to launch a public attack on their hosting providers.
From what I’ve been reading on reddit, Lemmy and hackernews, this public attack is only bringing to light what a scummy company they are, not CF, so it has failed tremendously. It doesn’t help that no one wants to defend an online casino in the first place.
so it has failed tremendously. […] no one wants to defend an online casino in the first place.
dunno about that with OP defending them
deleted by creator
Yeah, that’s alarming to me as someone who architects solutions for companies myself. Under no circumstance (unless idk the client in question is somehow in legal trouble like serving csam or something horrible, and that’s clearly not the case here since they’re going for more money) should an account be taken offline that quickly. They knew this would cause downtime and they did it anyway
Most companies would shut your shit down for TOS violations and then maybe allow you to reach their legal/finance to figure out how to offboard your ass. Two weeks was more than fair.
deleted by creator
Especially a TOS violation that isn’t a violation in the more expensive package. It’s clearly a tactic to kick out customers who cost them too much.
it’s because the cheaper tiers share IPs with literally every other CF customer. Of course it’s more expensive to not do that, it has nothing to do with trying to make a sale.
This is how I view it. Even if they were doing it maliciously, it is still a company and cloud flare should make the assumption they didn’t know they were doing anything wrong. Tell them it’s actually not allowed, that you just caught it, and give a clear (reasonable) timeline on getting to compliance.
From the post: I’m a SysOps engineer at a fairly large online casino. We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the “Business” plan which has some neat features and costs $250/month for “unlimited” traffic.
This seems a bit like abuse of the business plan not cloudflare bs. They are using the cdn for 4m users for $250 a month.
Maybe you’re right that ultimately they were not on the correct plan and Cloudflare was right to make them move. I don’t know enough about Cloudflare’s different plans to say. But what I do know is that:
When we told them we were also in talks with Fastly, they suddenly “purged” all our domains
Is utterly indefensible anticompetitive behaviour. So is a 24 hour ultimatum for a 40x increase in cost. I don’t care if they were on the free plan and should have been on enterprise. If the ToS violation isn’t actively causing harm to the public, any adjustment should be done with sufficient notice that arrangements can actually be made. 30 days seem s reasonable.
As they stated in the article, they were fully open to them calling out anything that was against the ToS, but CF never explained to them what was wrong, or how they could rectify it. They attempted multiple meetings with them to try to figure out what was the culprit, but cloudflare hit them with a 120k/month bill insisting it was necessary and never telling them why.
Clouflare fucked up in multiple ways:
- It should have never happened in the first place. (If they should have been on a different plan than this is a billing system bug, they should have just fixed their bug with billing, or it should have been locked behind a paywall.)
- They had multiple opportunities to tell them what was wrong, and how they could rectify it.
- Absolutely no service provider should ever cancel an account with such short notice. Full stop. Unless there is a legitimate legal reason not to, which at this point we have not seen anything (and I mean feds could be involved legal), it is extremely unprofessional to do that, and I would not trust them with my business. There is zero reason they couldn’t have given then at least a month’s grace period.
That and also, what company of that scale can you just go to finance and be like “Hey Cloudflare just jacked up our rate from $250/mo to $10000/mo and they want the whole 120k for the whole year right now and we need it done within 24h or they’ll cut us off”. Even for companies spending a million a month on AWS costs that’s 12% of the budget.
And also asking it all upfront, like, what? What happened to monthly billing? What company has the money to pay infrastructure bills yearly like that, especially on such notice?
Large companies have big cash flows, they don’t have 120k just laying around, it’s tied in some assets somewhere especially with the inflation, having large amounts of plain cash is bad finances. They probably need to take out a loan or sell some stocks or whatever. You can’t do that in 24h.
I have no doubt the author is omitting important details in the story, they may have been getting warnings for a while at this point and they just ignored them because “we’re happy with our business plan”. But the whole upfront part, then terminating the account as soon as they expressed looking at competitors pricing which is absolutely normal to want to do when your bill goes up 40x, if not required by company policy. Shady as fuck from Cloudflare.
just want to add in theory there are legitimate reasons to cancel other than legal, e.g. customers system is compromised and must be taken off to stop the attacker.
Fair, but then communication is key. They should have been extremely clear “This is what’s happening, you have X time to rectify it or we will need to take drastic steps”
Legal is the only one I can think of that would mean radio silence from CF, for example if they were hosting illegal content and the feds were building a case. Seeing how it was all coming from sales and they were pushing for a massive chunk of money all of a sudden I doubt that was what was happening.
Nothing to do with usage imo. They were in very obvious TOS violation that was affecting IPs belonging to CF and therefore affecting all CF clients. After a 48 hour warning they were still given two weeks to switch to enterprise plan and bring their own IPs. Instead they fucked around.
Play stupid games, win stupid prizes. I hope the CTO got fired for this.
And the article stated that they were probably abusing it and were ok negotiating a new contract. Did you read past the first few sentences?
Yes I read the whole thing. This is large company A being mad at company B for cutting off their way below market rate service and company B being a dick about the situation. I did some more digging and 4m monthly users seem to be around 1/3 of Fanduel (Flutter entertainment). This guy is probably working for a company with over $1b revenue per year. Any company that relies on their website for all of their business should have had contracts in place with CF to ensure they were fully within the ToS or contingencies in place to pivot off of CF should CF decide you aren’t in compliance.
CF said their account was flagged for domain rotation activities which is against the ToS. “This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.”. They had 2 weeks to stop doing that or upgrade to the enterprise account. Instead they didn’t do that and as soon as they said they said they were looking at alternatives, CF stopped giving them grace on the ToS violation in the most malicious compliance way possible.
Yeah. And I also take these posts by outraged people with a grain of salt. Sure large companies do shady shit, but we’re getting just one side of the argument and it’s from an angry person. Also, in this case, an online casino. Not exactly a source I trust 100%.
for what traffic again?
That it may well be, but it definitely falls on Cloudflare that they were able to take advantage of this for so long, and that the “unlimited traffic” was displayed as one of the perks in the Business plan (although I haven’t seen any evidence that that was listed). The decision to charge $10k a month would seem fairer if they weren’t insanely aggressive, and claimed there were violations of ToS where there don’t seem to be any.
It baffles me that you can advertise something as “unlimited” and then impose arbitrary limits after the fact. AWS and Google advertise their CDN rates with tiers for certain bandwidth limits. It seems like CF is advertising as “unlimited” and then once you’re fully invested, they pull the bait and switch and say you’re over the limit for that tier. Based on those HackerNews links, it seems systemic and something the FTC should fine them, like they did with AT&T over the same thing.
We’ve been using CF for a long time as enterprise and non enterprise customers and while their support went to absolute shit compared to what it was, I agree with the first comment on that article.
Casino was in violation of TOS and the only solution was BYOIP with enterprise plan. They were given 48 hours to correct, but tried to weasel their way out of it for two weeks when CF finally shut down their account.
I’m 100% on the side of CF in this instance. This also explains the sales calls. There was no tech issue to resolve.
It baffles me that you can advertise something as “unlimited” and then impose arbitrary limits after the fact.
I didn’t saw anything on the post that suggests that was the case. They start with a reference to a urgent call for a meeting from cloud flare to discuss specifics on how they were using the hosting provider’s service, which sounds a lot like they were caught hiding behind the host doing abusive things,and afterwards they were explicitly pointed out for doing abusing stuff that violated terms of service and jeopardized the hosting service’s reputation as a good actor.
That’s how I read this too, they were upset that this company was using so much and “only” on their lower tier. But, if they didn’t want that then that means either their billing/account code has bugs in it because it didn’t lock them out and force an upgrade… or it was mislabeled (intentionally or not). On all accounts, CF’s problem, not this company’s
This reads like the kind of stuff big corporations pull with end users. Where they would get sued to oblivion and back from an affected company. I mean, surely, there’s gotta be some service guarantee in the contract that prevents them from ToSing you like that.
I get randomly blocked on 2 websites since some days when using firefox on linux. Does not happen with chromium. One outright tells me its because of cloudflaire, but is sispect cloudflair behind the other one two.
Cloudflare fix your firefox on linux support! There are dozens of us, DOZENS!
Curious that I’m also using Firefox but in Windows and I get “rate limited” on the 2nd link I open on this specific site.
paypal login don’t work for mee since quite some time in firefox for for some time now too, i suspect similar
I’m on CFs side. These guys should’ve known better and 120k from a casino is a drop in a very large bucket.
Should’ve known better than to what? Use Cloudflare?
To use a service tier that they’ve clearly exceeded for quite some time and then act surprised when they’re asked to upgrade. Then to make matters worse they try to play hard ball by mentioning that they’re already talking to other providers. I’m sure the casino this is all for is way more lenient and let’s it’s customers play games or make bets they haven’t paid for.
try to play hard ball by mentioning that they’re already talking to other providers
It’s not “playing hard ball” to mention that you are look at other options. Jesus. As the author said, of course they would be. Wouldn’t you if you were facing a 40x price increase?
To use a service tier that they’ve clearly exceeded for quite some time and then act surprised when they’re asked to upgrade.
We must have read totally different posts.
Now admittedly, $250 is probably fairly low for the amount of traffic we were pushing through Cloudflare. We mainly use CF for the CDN (caching all our static content) and DDOS protection, for which it works pretty well. It’s easy to use and you don’t usually have to think about it much.
Nope. Same article. You’re crazy if you think you can average 7mil visitors a month and stick to the $250/mo plan. It blows my mind that a casino of all things is mad about that and we’ve got people like you so desperate to shit on cloudflare that you can’t see how ridiculous that is.
Wasn’t the “Business” plan advertised as “unlimited”? Also the problem was a totally different one. They were pressured into upgrading to the “Enterprise” plan without being given any reason. I would be on CF’s side if they just said: “With your current plan you are violating TOS. Here’s what you did wrong: Lists ToS violations The features you need in order to fix this are only included in the Enterprise plan, so please upgrade or we have to terminate your services, we give you 1 week”
I misunderstood before and I agree.
You’ve misunderstood the issues completely. They had no issue with paying more. The issue is with the extortion of demanding they pay for 12 months up front, the 40x increase (7 million visits a month is 3 visits per second on average; not huge - definitely doesn’t cost $10k/month), the terrible communication, and the retaliatory shutdown of their account. If you can’t see the issue with any of those I’m not sure what’s wrong with you.
Consider how Cloudflare should have handled this:
- Instead of connecting them to sales repeatedly, connect them to a technical team who could explain the actual issue.
- Let them pay monthly (honestly based on the emails from them it sounds like there actually is a monthly option; they just were trying to strongarm them into paying for a year).
- Have less insane pricing.
- Give them proper warning before shutting their site down. Not “your site may be closed”, but “we will close your site on XXX”.
Cloudflare are the clear villains here, assuming none of this is an outright fabrication, which seems unlikely based on the similar experiences of other people. You desperately grasping at the fact that its a casino to swing the moral balance is hilarious. Why do you want to defend Cloudflare so much? Do you have shares in them or something? (Pretty sure I do.)
This should have known better than to checks notes believe what they were told
