Kaity A

Oh goodness, why is this so funny? lol.

I would like to acknowledge here that Ada and I, while being part of a marginalized community are both still very privileged.

I also want to make clear that I’m not saying that you need to have decades of experience, or the means to cover the costs by yourself, but rather wanted to point out it’s a hard job and you need to have (multiple, qualified) people to do the job for a larger instance. There may also be times when the donations don’t 100% cover the costs, and you’ll need to have a plan when this happens.

Unfortunately multiply intersectional marginalized people have a lot harder time getting that experience and to have a means to fund such instances. It can additonally be a lot harder for them to find opportunities to gain this experience.

If you do find yourself in a similar situation, and would like an opportunity to gain some experience/training in a LGBTQIA+ / neuro-diverse / marginalized people-friendly community project, please message myself or @ada and we may be able to help.

So, one thing I’d mention is the systems and admin work involved in running an instance.

This is on top of the community moderation, and involves networking with other instance admins, maintaining good relations, deciding who to defeferate from, dealing with unhappy users, etc.

Then there’s the setup and maintenance of the servers, security, hacks, DDoSing, backups, redundancy, monitoring, downtime, diagnosis, fixing performance issues, patching, coding, upgrades etc.

I wouldn’t be here doing this without @ada. We make a formidable team, and without any self effacement, we are both at the top of our respective roles with decades of experience.

Big communities also magnify the amount of work involved. We’re almost at the point where we are starting to consider getting additional people involved.

Moreover we’re both here for the long haul, with the willingness and ability to personally cover the shortfall in hosting costs.

I’m not trying to convince you to stay here. But in addition to free hardware, you’re going to need a small staff to do these things for you, so my advice is to work out if you have reliable AND trustworthy people (because these people will have access to confidential user data) who are committed to do this work long term with you. Where will you be in 3 years, 5, 10?

We have already begun moving, and this announcement was to warn our users about future downtime as we migrate more critical services like databases etc.

To be clear, $3k is an accurate, but unacceptable amount.

As in that’s what it’s actually costing us, but it’s not what it should be costing. I’d imagine more like $250 is what we should be paying if I wasn’t using AWS in the silly way I am.

I’m admitting up front that I’ve been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.

Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.

And so I didn’t notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that’s what got me.

About half that $3k is just data transfer costs.

Anyhow the notice was just to let our users know what is going on and that there’ll be some maintenance windows in their future so it doesn’t surprise anyone.

We have a plan and it will all work out.

Don’t panic or have any kneejerk reactions, it’s just an FYI.

Just want to say, I don’t blame anyone else but myself.

I certainly don’t blame anyone at 196.

I hope I’m really clear about that. It’s one of the reasons I specifically didn’t name 196 in my announcement.

We’ve got a solution planned, we’ve already started to implement it and have the image transfer issue solved already.

We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn’t continue because I can’t continue to line Jeff Bezos’ pockets like this indefinitely.