@k_o_t
admin
link
411Y

:)

@resynth1943
link
3
edit-2
8M

deleted by creator

@koavf
link
281Y

Did you ever determine if it was a denial of service attack or just an influx from Hacker News (hi!)

Dessalines
mod
admin
creator
link
141Y

Seemed to be a combination, we don’t have any major monitoring set up, but looking at our nginx logs, it was 55k requests in a short period of time. We posted to reddit a few weeks ago and hit the front page of some linux / OSS related subs, but the server held up fine.

Luckily adding an nginx rate limiter fixed most of the issues, as the problem wasn’t with rust, but with some clients w/o caching (IE bots of some kind), requesting all the files constantly.

@developred
link
2
edit-2
8M

deleted by creator

Dessalines
mod
admin
creator
link
31Y

Limiter? I think we have 10 posts per hour, 2 communities / sign-ups per hour.

@Volodymyr
link
101Y

That’s why I came here anyway. Or maybe DDOS was also attracted by the sudden exposure on HN?

Hal
link
81Y

Ahh… the Hacker News hug…

@stax
link
171Y

Just gotta say that I really appreciate the work you are doing

Dessalines
mod
admin
creator
link
81Y

Thanks, we’re trying :)

@resynth1943
link
2
edit-2
8M

deleted by creator

@nutomic
admin
link
15
edit-2
1Y

All good now, the site might just load slower than usual now (we need to rework the database).

@Qaz
link
81Y

Yeah, I have been noticing long loading times.

@creativeBoarClimate
link
11Y

Is this still ongoing? I’ve been noticing longer load times

@nutomic
admin
link
21Y

Yes its gonna take some time.

@bluefish009
link
51Y

is that you reddit?

Dessalines
mod
admin
creator
link
91Y

We have > 2200 connections to the server right now, its a DDOS. Rust seems to be handling it fine, but the nginx is having issues.

@stanleypar11
banned
link
61Y

2200 does not seem like a DDoS in my opinion, I would expect hundreds of thousands in that case

@eneville
link
41Y

I’m reading it as the 2200 connections are between nginx and the rust component, not 2200 to the edge nginx.

@zodvik
link
2
edit-2
1Y

I agree to this as well.

I’m surprised nginx is having issues. I’ve personally handled nginx instances at more than an order of magnitude of connections.

There should be good low-hanging fruits in configuration to exploit to help handle more load.

Dessalines
mod
admin
creator
link
61Y

I’m guessing we got x-posted to the donald or something.

@schrutefarms
link
161Y

I just made an account and came from HN. I hope it is just a good ole regular hug of death!

@SirLotsaLocks
link
91Y

Welcome, I like this because it feels like HN but it has other communities as well if you get what I mean. Hope you enjoy it here :)

@TheAnonymouseJoker
link
21Y

Yikes.

@Molly666
link
11Y

Hmm, this is very strange. I never thought that this type of attack is possible in a decentralized p2p application. By the very least, I did not find a single mention of this in all the sources that I found https://utopia.fans/networks/ddos-attack-what-it-is-and-how-to-trace-a-ddos-attack/ https://lemmy.ml/post/35712 .

無門關
link
21Y

FLAT THE CURVE! FLAT THE CURVE!!!

@ajz
mod
link
2
edit-2
1Y

While we’re at the topic of DDOS, fyi KeyCDN has DDOS protection, and they provide free CDN accounts for open source projects : https://www.keycdn.com/open-source-cdn Perhaps not needed for Lemmy after moving to OVH hosting, but maybe useful to know for others as Cloudflare alternative.

Site breaking so much it took ten minutes to post this (if it goes through this time).

Dessalines
mod
admin
creator
link
71Y

Apologies, my main priority rn is DB rework.

@resynth1943
link
2
edit-2
8M

deleted by creator

@resynth1943
link
2
edit-2
8M

deleted by creator

@lem135
link
01Y

hmmm…isnt p2p kinda prone to those attacks 🤔

@lem135
link
11Y

I understood lemmy is decentralized p2p network how is this possible then? ELI5?

@nxlemmy
link
31Y

this OP post is a month old. It wasn’t being DDOSd it was trending on HackerNews. Also lemmy isn’t p2p its federated. This means it still works like a traditional server but can connect to other servers. In other words it’s Decentralized but not Distributed.

@stanleypar12
banned
link
-41Y

Why did my other account get banned? Was it because I suggested Cloudflare?

Dessalines
mod
admin
creator
link
161Y

Post dog dicks on your own instance lol. And also I’m not linking anyone your posts where you did that.

@Atemu
link
-51Y

Just because of that?
I’m not into dog dicks either but as long as the community is properly marked as NSFW and stays within its bounds, I don’t see why it should be removed or have its creator banned.

@nutomic
admin
link
91Y

Porn is against the rules, simple as that.

Stanley is probably just trolling with this dog dick nonsense fwiw.

Can the TOS have a strict definition of what encompasses that? Would cut down on smart-assed posters arguing about what constitutes porn in the future.

Pornography is a nebulous term and can be interpreted in many ways. Just look at the neverending porn Vs erotic art debate for example.

@nutomic
admin
link
4
edit-2
1Y

If we make it more specific, theres a risk that people post questionable things, arguing that they are not technically against the rules. So I’m not sure thats a good idea. In the end its always a subjective decision of admins/mods which content gets removed or not.

I hope the subjectiveness allows for the time honoured tradition of Chapo users posting pigpoopballs.jpg?

@nutomic
admin
link
31Y

I’m not a fan of that, but it certainly cant be considered as porn.

@stanleypar13
banned
link
-91Y

I did not realize it was against the rules (yes I read the rules)

@ReformIsFailure
link
8
edit-2
7M

deleted by creator

@blaha
link
11Y

I just wanna know if they were posted in c/dogdicks or somewhere irrelevant as an obvious troll

Dessalines
mod
admin
creator
link
51Y

This site has an open modlog, you can view it there.

@developred
link
11
edit-2
8M

deleted by creator

@stanleypar11
banned
link
-31Y

Wouldn’t it be a good idea to put the whole site behind Cloudflare?

@nutomic
admin
link
141Y

Even ignoring the privacy issues, Lemmy is really lightweight and can deal with the load just fine. We just had to make some adjustments in the nginx config.

@DFatDuck
link
31Y

What issues are there with Cloudflare?

@nutomic
admin
link
91Y

It gets all the data that is sent between the user and the server in plaintext. So they can easily track everything and even sell the data (in theory). Not to mention that they block Tor users.

@DFatDuck
link
51Y

oh, damn. i’ll find ways to avoid Cloudflare services then.

@resynth1943
link
1
edit-2
8M

deleted by creator

@resynth1943
link
1
edit-2
8M

deleted by creator

@ajz
mod
link
121Y

Cloudflare is an unethical choice imo. There are more options to deal with DDOS. Apart from DDOS protection, a nice CDN provider is BunnyCDN. https://hub.fosstodon.org/moving-away-from-cloudflare - https://bunnycdn.com/

@resynth1943
link
1
edit-2
8M

deleted by creator

@k_o_t
admin
link
71Y

that would be a horrible idea (i’m hoping it’s just a weirdly framed joke…)

@stanleypar11
banned
link
11Y

Why is it a horrible idea?

@onlooker
link
191Y

Try browsing the web using Tor or a VPN. You will spend the day doing Google recaptchas for like 90% of the websites you visit, I guarantee it. All thanks to Cloudflare. Recaptchas are also an issue in itself, because what solving recaptchas does is it teaches Google’s AI to better perform at image recognition without the user’s consent, but this is never pointed out to the user. Users are basically doing Google’s work for them without even knowing it.

Then there was the whole Cloudbleed fiasco. Private messages, credit card info, passwords, etc. All kinds of stuff was leaked online, which was not a good look.

Then there’s their uptime. It’s not great. How many times have you seen this image? Your PC’s fine, the website you’re trying to reach is fine, but Cloudflare is down. Thus, no website for you.

I’m sure there’s more, this is just off the top of my head. If Cloudflare is the answer, I’d rather have the problem, honestly.

@Golden
link
2
edit-2
1Y

Hum I did not know that the captchas were because of Cloudflare!

@stanleypar11
banned
link
-71Y
  1. The captcha thing is configurable, it can disabled entirely
  2. Cloudbleed was fixed ages ago
  3. You’ve just cherry-picked an image showing a Cloudflare error, I guarantee you Cloudflare has much higher uptime than the actual host has
@ajz
mod
link
41Y

Re: 1. Is that correct for non paying CF customers ? As Tor or VPN user CF is a big annoyance. If it is correct what you are saying, who is going to have all those users change the settings ?

@stanleypar11
banned
link
21Y

Yes, in the Cloudflare dashboard you can either make a Firewall setting to “Allow” access to 0.0.0.0/0 or you can set your security options to “Essentially Off”… or both

I’ve just tried it against my site and haven’t seen a captcha prompt

I believe no one ever bothers to configure these which is why Tor users run into so much trouble with captchas on CF sites

@ajz
mod
link
3
edit-2
1Y

Thanks for the follow up. Is it documented somewhere ? That (“Allow access to 0.0.0./0”) sounds a bit like turning off DDOS protection or allowing visitors to reveal your real ip address. I’ve read in the past the non paying CF customers can not turn it off, but maybe that information is outdated or was not correct. I’ve only had to use CF dashboard for a customer some time ago. I remember that in the DNS settings one could mark a DNS entry for DDOS protection or not.

@stanleypar11
banned
link
21Y

I’m sure theres a difference between the security setting “Essentially Off” and “Off” (which is reserved for enterprise customers only). It could possibly mean that there are still some ip ranges or ASNs which are still forced to have a captcha displayed to them with the “Essentially Off” setting, but I haven’t encountered any.

Also I may have been mistaken about the Firewall 0.0.0.0/0 setting, it seems that Cloudflare’s “security level” is its own version of Firewall rules that apply even before the customer’s rules. So, for example if I set my site’s security setting to “I’m under attack” then I will be displayed with a javascript challenge even if I have the Allow all from 0.0.0.0/0 firewall rule enabled

@ajz
mod
link
11Y

Cheers.

@k_o_t
admin
link
5
edit-2
1Y

yeah, as u/onlooker pointed out, it’s very hostile against Tor users, though it became slightly less problematic after CF switched from recaptcha. There’s also a ton of ethical and security problems. I was trying to find a github gist that summarized and elaborated on it nicely, but it appears i’ve lost it.

If anyone can find it, please link to it.

not the one that i originally meant, but here’s another one, suggested by u/ajz

@flux
link
21Y

it’s very hostile against Tor users, though it became slightly less problematic after CF switched from recaptcha

I guess the pragmatic option is to provide a tor-hosted service for them. I imagine it is also protected against DoS-attacks, or if not, then it only impacts tor users.

@stanleypar11
banned
link
-21Y

That “codeberg” repository reads similar to a 5G conspiracy theorist blog

@k_o_t
admin
link
31Y

yeah, it’s written in a somewhat weird style, but most of the stuff mentioned there you can literally verify yourself…

@aeroplain
link
13
edit-2
1Y

deleted by creator

@Panzerfaust
banned
link
91Y

I have seen that. One of the users said that Lemmy “… must be getting the HN hug of death …”.

Dessalines
mod
admin
creator
link
51Y

ahhh ok.

@resynth1943
link
7
edit-2
8M

deleted by creator

Dessalines
mod
admin
creator
link
61Y

We definitely will never use Cloudflare, they’re an absolutely awful company that’s managed to weasel their way as a snoop in the middle to nearly half of all US internet communication.

@nutomic
admin
link
41Y

Definitely not.

@resynth1943
link
6
edit-2
8M

deleted by creator

@resynth1943
link
1
edit-2
8M

deleted by creator

Dessalines
mod
admin
creator
link
41Y

It is, luckily I have the DB fixes finished, and should be deployed to here within a day or so.

Lemmy Announcements


Feel free to announce new communities here.

Other than that, this is reserved for admin use only.

  • 0 users online
  • 5 users / day
  • 16 users / week
  • 71 users / month
  • 293 users / 6 months
  • 11602 subscribers
  • 568 Posts
  • 2908 Comments
  • Modlog