hey folks, here’s another meta-post. this one isn’t specifically in response to the massive surge of users, but the surge is fortuitously timed because i’ve been intending to give a good idea of what our financial stability is like. as a reminder, we’re 100%-user funded. everything you donate to us specifically goes to the website, or any outside labor we pay to do something for us.
thanks to your generous support, we’re pretty confident we have passed our current break-even point for this month, at which we wouldn’t eventually need to pay out of our own pockets to keep the site running. that point in our estimation is about $26 a month or $312 a year. (please ignore OC’s estimated yearly budget–we don’t determine it lol)
our expenses are currently:
- $18/mo toward our host, Digital Ocean. (yesterday we upgraded from DO’s $12 tier to its $18 tier to mitigate traffic issues and lag, and it’s really worked out!)
- $2/mo for weekly backups
- $4/mo for daily snapshots of the website, which would allow us to restore the website in between the weekly backups if need be.
for a total of at least $26/mo in expenses. this may vary from month to month though, so we’re baking in a bit of uncertainty with our estimation.
we currently have, for the month of June:
- $70/mo in recurring donations (at least for June)
- $200 this month in one-time donations
for a total of $270 this month. our total balance now stands at $331.31.
that balance means we now have about a year months of reserves currently, if we received no other donations and have no unexpected expenses.[1] the recurring donations put us well into the green at this point.
this is good! everything past our break-even point each month is, to be clear, money we can save and put toward scaling up our infrastructure. there is no downside to donating after we’ve already met our “goal” of basic financial stability. doing so will have pretty straightforward practical implications for you: fewer 500s, 503s, better image support (this takes a lot of space!), and the website generally being run on more than potato hardware.[2] if you’d like to do so in light of this information, our OpenCollective page is this post’s link. thanks folks!
minor PSA: we splurged a bit thanks to your donations and got slightly better hardware. you’ve probably noticed a more fluid website that’s erroring way less accordingly. enjoy! i’ll update this post’s numbers tomorrow to give you a reflection of the new break-even. thanks for using the site folks
It will be interesting to see how many users stick around. It’s busy right now but in a month you may be able to backtrack to the previous hardware.
well, in a month we might be dealing with a second wave of users (and i think we won’t be short on people who will stick around), but yeah point taken lol. it does help we’re also not using a complex set-up, so “scaling up” isn’t a big commitment right now
I think it’s normal after a big surge to see participation drop off. If you can hit the critical mass to keep content flowing, that would be awesome.
And at least in the short term, it’s appreciated that the site is now running much smoother! That’s also important for retention of users :)
oh yeah no we’ll definitely see some (probably a lot of) dropoff, but if even 20% of this wave sticks around and starts contributing at all that will add several times over what our previous userbase was. we were pretty close to a point as a community where there’d always be activity of some kind, and this event has pushed us well over that threshold
Haha yes, it’s like a big foot reaching out to give reddit a kick in the pants: https://the-federation.info/node/details/25274
I would like to know to what extent you are able to scale up the hardware. Is there a theoretical limit in the number of users that can handled?
There is room to scale the VPS immensely. However, we made a small bump to the next tier yesterday and things are working very well now.
Is there a theoretical limit in the number of users that can handled?
Maybe…I don’t know.
Glad to see the donations came in handy!
At the risk of sounding like Roy from Ted Lasso… This is FUCKING REFRESHING. This is how it should be! Thank you so much for the transparency, and for all the work you’re doing to implement this instance. I’ll definitely be sticking around, and hopefully contributing whatever I can. This is amazing. Please keep it up!
I discovered this whole thing just a few days ago (of course from that reddit post) and I love how this has such strong early-internet vibes! I mean where anymore do you ever read the owner of a site casually go “Yeah, since we are a couple hundred dollars in surplus, I was able to splurge on new hardware so that you’ll see loading errors a little less often because the server is no longer powered just by my child being strapped into an oversized hamster wheel”?
Do you think you could link me to that Reddit post? I’m out of the loop
Thx
Oh sorry, it seems I didn’t get a notification about you replying to my comment. But yeah, that is the post I was talking about.
I’m new to BeeHaw but its really encouraging to see such transparency and, frankly, competent admin communication. Thanks for sharing this!
Massively appreciate this level of transparency on the instance’s finances. To me, that’s a promising sign
Scaling up can become quickly cost-prohibitive with large-scale servers. I’ve noted that the most affordable option with Digital Ocean, at $12/month, offers only a basic droplet with 1 vCPU, 2GB of RAM, and 50GB of SSD storage. When you consider a higher-end configuration with 16GB of RAM, 8 vCPUs, and 320GB of SSD storage at $96/month, it may not seem economical, especially as storage and backup needs increase with server scaling.
As an alternative approach to minimize costs while scaling, consider purchasing used servers from platforms like eBay and setting up a small-scale hosting operation in your garage. While this route does introduce overheads like business internet services and electricity costs, along with regular maintenance such as HDD replacements, it could be more cost-effective in the long run.
For instance, you could acquire a server on eBay for about $300, offering 20 CPU cores, 64GB of RAM, and 8TB of SAS HDD storage. Comparatively, a similar setup on Digital Ocean would cost around $544/month or $6528/year, making the used server a strong competitor against cloud services.
Just some food for thought if you’re contemplating scaling in the future.
Digitalocean is pretty expensive. For example on hetzner.com you can get 8 vCPU for 15 euros per month. OVH should be similar. So thats really affordable if you have hundreds or thousands of users and a couple of donors.
While I’d be more than happy to purchase and use actual server space for doing this kind of thing, I’m not nearly tech saavy enough to actually run that. I would also worry about bandwidth considerations and other issues. Perhaps there are people willing to contribute to the cause to find affordable ways to run a website like this, but being accessible by multiple people to do things like power cycle the server, not being concerned about our personal IPs being attacked, and having access to support I think is worth the extra cost.
There is a point at which something akin to this needs to be done for financial reasons, but I think we’re pretty far from that being a reality and while it relies on slightly more donations from our lovely userbase, even at $96/mo that’s not that much money to collect when we have the thousands of users that would require that kind of hardware.
I appreciate your thoughtful response to my comment. There are indeed several strategies that can be employed to decrease bandwidth and storage costs. Leveraging a Content Delivery Network, such as the free service provided by Cloudflare, can help mitigate these costs by caching your webpages and images. As for the cost of internet service, it greatly varies based on your location. If you’re located closer to the internet backbone, the likelihood of finding a more reasonably priced business internet plan increases.
While it may seem premature at this stage, I firmly believe in the success of this website, even in the face of numerous failures in this space.
- Edited to add -
You’ve correctly highlighted the potential threat of attacks on your server. Cloudflare is known for its prowess in mitigating substantial distributed denial-of-service (DDoS) attacks and could be an excellent security asset in this context.
Regarding the value of professional support, I acknowledge that the cost can often be justified. My suggestions are merely alternatives, providing you with additional options should you require them.
Using cloudflare as proxy would essentially mean letting them MITM all the traffic though right? All things considered they’re one of the trustworthy companies but is there some other alternative that you can basically self host?
Cloudflare can’t be accurately labeled as a Man-in-the-Middle (MitM), given its integral role in the service stack. The same logic would falsely accuse platforms like Linode, AWS, and Azure of the same. Moreover, self-hosting is entirely feasible. The main challenge arises from Internet Service Providers, which often restrict upload speeds unjustifiably. I highlight this to explain why it typically becomes more economical to locate closer to the internet backbone, where the cost and the plan tend to be more reasonable.
Hey person :) I didn’t accuse anyone of anything. Just pointing out that if you use clouddlare as a proxy specifically, they are technically decrypting your traffic? AWS/Azure/Linode are primarily hosts for webapps and VPSs not proxy providers as far as I’m aware.
It’s more apt to say that when you observe what it offers and what it needs, you’re basically coming to a decision that you could choose to forgo some of the security by giving them the TLS cert in exchange for CDN to alleviate server load as well as preventing DDOS attacks. And Linode does allows proxy to be run on their infrastructure when I last contacted them, so they are somewhat a proxy provider although not directly.
And I didn’t mean to say you’re accusing those providers, but only pointing out that when you voluntarily give the providers your configuration/certificate, there isn’t any malice in this case for it to be attributed to Man in the Middle attack, and there were consent involved.
I agree that you could host your own proxy on any provider if you wanted to which is nice :)
My problem with Cloudflare is that they aren’t that transparent about what they’re doing.
What I’ve usually seen is this: people switch to cloudflare DNS because frankly it’s one of the best services available. They see the little cloud next to their A records which says it uses proxy to make your websites load faster and think this is great. At no point there is a warning saying that by clicking this you’re essentially letting us manage TLS on your website.
I do use cloudflare proxy because it is pretty neat but definitely not on all content I use.
I also have to say my concern is not that cloudflare is going to read my passwords or info in my databases but that a) I wouldn’t like to put all eggs in one basket and b) dedicated state actors like NSA might have access into cloudflare.
Very much appreciate the knowledge and responses! We’re definitely in need of systems administrators who have time and energy to expend in terms of helping us reach something sustainable both financially and technically speaking.
Echoing thetruelinuxdev, I’d be willing to help however I can. I also have a personal server just sitting mostly idle I could try to use for this stuff, but that of course depends on you all as the admins of the instance trusting some random on the internet vs a VPS provider like DO. I do have Linux Sysadmin and other technology experience. Giving money is easier but might not be as impactful as helping with infrastructure, to me.
Thank you! We will definitely take any and all suggestions and offers into consideration 🥰
Just PM for resume 🤣 but seriously, while I know I can’t donate as much time as some of you already do, for what I can do, I’d rather support a community effort like this with my labor rather than that other-site.
I’ll throw my hat in here too, I have capacity on my in-home infrastructure and will also be colocating a couple servers in a proper datacentre over the coming months. I back up everything to disks, minimum 2 week on-disk retention, then duplicate it onto tapes (don’t laugh - they’re rated for 30 years cold storage). I keep yearly backups infinitely, monthlys for 1-2 years, weeklys for 6 months iirc. Based in Australia if that helps. 10+ years experience as a sysadmin/cyber sec professional. Feel free to DM if I can help the cause as I’m happy to donate capacity
While I may not be able to provide financial assistance, I’m more than willing to lend my expertise in system administration and development, should you find it beneficial.
I did that in my garage a few years back to tinker around with linux servers. Just don’t make the same mistake I did and buy the cheapest SSD on eBay. My friend group still hasn’t forgiven me for letting our minecraft server get corrupted lol.
Absolutely, and at a minimum, I’d suggest implementing the 3-2-1 backup strategy, widely regarded as the “gold standard” for maintaining data integrity. This involves keeping a total of three backups: two stored locally, but on different devices, and one off-site backup (ideally kept in a secure location such as a bank safe deposit box, with hard drives rotated every two weeks).
That’s definitely where I went wrong. The sad part is I did make an effort to backup the mc server weekly. I had a cron job that would bring the minecraft server down, export the SQL db’s for plugins, and copy the world / other config files into a tar.gz file and copy it to a hard drive raid also running on the server. I knew having raid wasn’t a backup in itself, and also running it on the same server wasn’t doing me any favors. The problem was the archive files were incomplete, but I didn’t realize that until the SSD died and I went to restore the archive file. Still not really sure why they weren’t complete archive files. I still have them, just can’t open them. It’s weird though, when I first set it up they totally worked.
Indeed, one challenging aspect of data backup is the necessity to confirm that the backup is restorable. This adds another layer of complexity to system administration. However, with the aid of a well-crafted script to handle backup verification, reliability of backup restoration can be ensured, potentially through a method such as hash comparison.
Backups can sometimes give you a deceptive feeling of safety. If you neglect regular verification, you might find that your backup fails just when you need it most for a restoration.
Hashing! Why didn’t I think of that when I created the original backup script haha. Low key might go write a new backup script and incorporate that. I’m running another private mc server for some friends, this time in the cloud cause I didn’t want to deal with another crash. I would like to bring it back to running in my garage sometime.
I really recommend you to use Borg Backup instead. It has incremental encryption which is really useful, and generally does its job well.
For what it’s worth, I’ve had issues with data corruption restoring with Borg. Defeating the purpose as a backup. I’ve used Restic happily and restores very well.
I’ll check it out, thanks!
In your situation, I’d recommend creating a script that performs the following steps sequentially:
- Accumulate the hashes (ideally SHA-256 or SHA-512) of all files you plan to backup.
- Create a text file that lists the file paths and their corresponding hashes.
- Initiate the backup process.
- Upon completion of the backup, restore the data to an auxiliary drive.
- Compare the hashes from the text file to those of the restored data.
- Alert you about whether the process was successful or not.
Regarding Borg backup, while it offers robust verification and security mechanisms, I’ve personally found its performance lacking, though your experience might differ. I would recommend thoroughly reviewing its documentation and conducting some trial runs prior to integrating it into a production environment.
Ahh okay. I do enjoy writing programs myself where I can so I’ll probably still do it myself, but good to have options. Since it’s just the MC server I’m truly worried about, I don’t care too much about the performance within reason. I like the idea of a text file with corresponding hashes. Keep it simple! Or as my old shop teacher used to say, KISS, keep it simple, stupid.
What about just autoscaling on AWS
Auto-scaling comes with its trade-offs. It can lull users into a false sense of security, leading them to believe that all configurations will seamlessly scale up. However, even with managed databases, there are inherent limitations and the need for architectural changes to support a larger-scale infrastructure. While auto-scaling provides a robust system for managing overhead and mitigating the efforts involved in scaling, there are potential risks and hidden costs.
One significant risk is the misjudgment of traffic or the inadvertent scaling up caused by a software bug. This could escalate the demand for more computing servers indefinitely, limited only by your budget. In contrast, self-hosting, despite its higher administrative complexity, offers more opportunities to cut costs associated with cloud usage.
As with most things, different approaches have their pros and cons. I am simply offering alternative ideas for those interested in exploring other options.
I’m not sure they want to depend on Amazon services?
Ah, fair enough.
deleted by creator
It is pretty funny to watch the funding go up by a factor of four and see the website immediately become more stable. Web-hosting services are nice like that.
Started a small monthly contribution, hopefully will help scale to what will likely be a lot of increased usage
It’s going to be entertaining seeing the new numbers after you update this post.
That’s… surprisingly low! Nice! Now i find myself wondering about the financials of something like the biggest Mastodon instances…
I’m now contributing a little bit. :)
Thanks for the quick approval!
Really hope this becomes the new reddit!
super happy I set up a recurring donation last week - I had a feeling you guys were going to get hammered at some point!
I’m coming in at about 78$ a month for mydroplet; a s-4vcpu-8gb and a singular 250gb volume, including taxes. I still think paying someone to host it remotely is a better deal - but I also live on an island and we lose power multiple times a year and I don’t want my inability to power things impact my users.
Is it possible to donate Cryptocurrency? I’d love to give XMR (Monero) so I don’t have to reveal any details about myself, etc.
deleted by creator
Is it possible to donate Cryptocurrency?
at least currently we’re tethered to OpenCollective (and not really in a position to extend ourselves across even more stuff to maintain) so if they don’t accept it then no, that’s just a limitation we’ll have to live with. that said, i believe OpenCollective allows for anonymous donation–we have many Guest contributions–so that may be suitable for your purposes if you can do crypto to fiat conversion
deleted by creator
Thanks! I’ll try to reach out to them and see if they still support that method.
Edit: Sent OCF an email and will report back later if I’m able to donate crypto through them : )
deleted by creator
That’s great to hear! I’m still trying to wrap my head around the federation concept itself so if you don’t mind, I have a question for you. In the event the site did go under (which it seems like is not the case, yay!) what happens to our account? Will we have to create a new account on another instance? If so, will we be able to use the same username?
I like the email analogy.
When you want to use email, you sign up for an account. It doesn’t matter if you go to gmail or hotmail or something else, you can still communicate with anyone else that has email.
Now if your username for email is fred@hotmail.com and then that server went down, you would no longer have access to your account, people couldn’t contact you. If you then decided to set up a gmail account, you would only be able to get the account fred@gmail.com if no one else had taken it. The name “fred” is not reserved for you across all email providers.
Federated platforms like Lemmy (which Beehaw is part of) work the same way. Your account is not @CobolSailor but @CobolSailor@beehaw.org. If Beehaw went under and no longer existed, you can go and sign up on another server, e.g. lemmy.ml. But you could only use @CobolSailor@lemmy.ml if it was not taken by someone else, it’s not reserved for you across all servers - and can’t be, because there is no central server keeping track.
Does that explain it?
The email analogy just made the whole federation click in my brain. Thank you for the well explained response!
@Mersampa @CobolSailor
As someone who has like 1 hours worth of experience here, that cleared a lot up for me about how this works actually. Cheers!Quick question since you explained things so well! Can you read/comment on what’s posted on other servers from the server you’re on? Can I see what’s happening over on lemmy.ml (for instance) from here? If so how does that work?
Yes! As I understand it, I’m not sure how you would see specifically lemmy.ml from Beehaw, but you can definitely see the content.
The easiest way to start is when you are on your home page in Beehaw, you can choose between “Subscribed”, “Local”, and “All”.
“Subscribed” is as it says on the tin, any community you have subscribed to.
“Local” is any posts on Beehaw, the local server you are a member of.
“All” is where it gets interesting. This is any post from any Lemmy instance that Beehaw knows about. Lemmy learns about other instances when users subscribe to communities on those instances.
So have a look at the “All” feed, and pay attention to which communities posts are being made in. You can click on the community to see more info, and from there you can subscribe.
For example, you could subscribe to the lemmy.ml Memes community by going here: https://beehaw.org/c/memes@lemmy.ml
Thank you! The “local” versus “all” distinction makes a lot of sense!
Quick question since you explained things so well! Can you read/comment on what’s posted on other servers from the server you’re on? Can I see what’s happening over on lemmy.ml (for instance) from here?
while i can’t explain the how, the answer is yes and yes. we have a fair number of people who have accounts on other instances who comment here with their accounts on those instances; and it works vice versa for here and other places.
Thanks!
In the event the site did go under (which it seems like is not the case, yay!) what happens to our account? Will we have to create a new account on another instance?
yes, although luckily doing so is… pretty trivial compared to some other circumstances. i believe most instances also have open registration, so it’d be less rigorous than our registration.
If so, will we be able to use the same username?
provided it wasn’t taken on that other instance, yes (which for most people shouldn’t be the case, since lemmy is very small relative to other fediverse alternatives and its instances even smaller)
Hopefully it doesn’t ever come to it, but good to know I can create an account with the same name on another instance if I had to. Appreciate the response!
