The attackers found another slow db query to exploit, so had to put some temporary measures in place.
Will post more details later!
This time they actually took down the instance for several hours.
Thank you mrmanager for your service! o7
I’m on other instances too, but federation of posts has been sooo wonky lately. Like posts show up on one account, then they finally show up on another hours later. Or I’ll comment and it won’t show when viewed from another instance. Idk what gives!
Was it like that before the attack as well?
I can look into it and make sure we are federating properly. During an attack or after, it wont work, but it should federate as soon as the attack is stopped.
I’ve noticed issues with federation tonight, but not before this attack. I just made a bunch of comments and posts and was wondering why none of them were gaining any traction. I checked from other instances and the posts do not appear on them.
Thanks for all you do!
Send me a link to a comment or post and what instance you are looking on. :) Thank you!
Sure! For example, I made a bunch of cross posts with this post as the OG: https://lemmy.today/post/42851505 (posted to lemmy.world, but I can’t see this post if I navigate to the instance)
Another example, a cross post of the above to programming.dev: https://lemmy.today/post/42851619 (cannot be found in the instance)
Then a few comments which haven’t seemed to federate:
- https://lemmy.today/comment/20705378 (lemmy.zip)
- https://lemmy.today/comment/20685555 (lemmy.world)
- https://lemmy.today/comment/20695612 (lemmy.ml)
I made more than that though, and I’m pretty sure none of them have federated.
I think I figured it out. Has to do with my mitigation of the attack yesterday.
Now new comments and posts are federating to other instances again, but the old comments and posts, im not sure they will federate.
Attackers were hammering a certain endpoint to cause the database to be 100% busy in their attack. So i blocked that endpoint, thinking it would not have an effect on federation. Chat gpt also agreed that it wouldnt.
Chat gpt was wrong. :) So I changed from blocking it to caching it, which is at least going to help prevent further attacks to a degree.
Thanks for letting me know!
Thanks for the fix! I’ve reposted some of the content and it seems that federation is working on these. Hopefully I don’t end up spamming by accident!
Edit: it seems like I could have just edited the posts and forced federation that way. TIL!
Ah, thats a good idea actually, just edit them. :)
Im just looking at new comments or posts in those instances and it all seems to be delayed from the entire fediverse, not just lemmy.today:
https://programming.dev/c/opensource?dataType=Post&sort=New (last comment 13 hours ago, last post 17 hours ago)
https://lemmy.world/c/foss?dataType=Comment&sort=New (last comment 18 days ago, last post 18 days ago)
https://lemmy.zip/c/technology?dataType=Post&sort=New (last comment 5 hours ago, last post 2 hours ago)
Lets wait and see when these instance communities gets new posts and comments and see if the ones from lemmy.today are still missing. Will keep an eye on them. :)
I see some comments not being federated yet, here is one example:
https://lemmy.today/post/42824004 (112 comments, last comment 1 hour ago)
https://lemmy.world/post/39592788 ( 108 comments, last comment 3 hours ago)
https://sh.itjust.works/post/50825191 (105 comments, last comment 3 hours ago)
In this example, lemmy.world and sh.itjust.works has less comments than lemmy.today on the same post.
Normally comments show up right away so perhaps this is a sign of a problem. Will keep looking at it during the day.
Maybe it was related to the attack bc everything is working smoothly now! I detect no problems:)
Excellent :)
Same here. I just noticed my posts from this account haven’t been federating since the DDOS kerfuffle.
Tell us when you know the culprit. I can write one hell of a strongly worded letter.
Keep up the good work :)
Man, people really suck. Is there anyway we can help or assist? Maybe donations to help with server costs? Thanks for all you do to keep it running! This place is fantastic and I hate seeing it being targeted.
Its ok but thank you very much for asking!
Server costs are small and its not that many problems with the server. We had almost a full month of constant uptime without doing anything, and then this attack ruined that, but thats life.
Keep using the instance, its enough motivation for me to keep running it. :)
Thank you!
Why is someone actually doing this? Government interests?
I dont know. The only thing that makes this instance different is that we dont block any instances and we dont moderate opinions we dont agree with. I dont think we should do those things.
I guess that upset someone. :)
Oh well. Best of luck! Back to my own job.
Thanks! See you around :)
The CIA!
