Screenshot of the F1 Bluesky feed at 2000 likes
Amazing to see the F1 feed reach 2000 likes! I've really enjoyed seeing it grow. Thank you everyone who uses it and gives feedback.
If the api is down then the health endpoint itself would be a normal REST error. I use UptimeRobot to check it regularly and it emails me if there is an issue. Though based on passed experience and how @fly.io services work, my apis and processes are coded to restart themselves in most cases.
I forgot to add that I have a help endpoint that says how old the last post it looked at is and return an error after a certain gap (like five minutes). This error indicates that there is something wrong with with the processing of the posts from the firehose.
As bluesky has grown that is where performance has hurt the most. And then my biggest concern with a change is that I break the logic that puts posts into the right feeds. Hopefully that addresses your questions, I tried to catch all of them though I did not call out the numbers.
The thing the cache does provide is the security you mentioned. It prevents DDOS and such with the micro-cache. The real challenge is making a process that runs separately to read the firehose and update the feeds in redis based on the config. That was most of the testing I mentioned before.
Performance is negligible, even for a semi-popular feed like mine is less than 1 rps. The proxy is there to add a cache, but that is overkill with that rps since one of my feed reqs is to update within a second of something posted (for live skeeting on race day). The cache is 1s but it rarely hits.
Mine apis are really straight-forward since they just pull from a Redis ordered list that matches the cursor. TBH, I have not changed that code really or tested it in years. It's simple enough I haven't really done more than manual tests when I have changed.
The custom feed interface was designed by Bluesky to be pretty simple. It is mostly just a list of items and a cursor/next-page value. All you need is a REST endpoint. docs.bsky.app/docs/starter...
And I either test that by creating a new "beta" feed which I have done in the past or in this case swapping the versions for a short period. In this case both feeds are running and I have a proxy in the middle that I can config to point to one or the other.
The reason I tested it in prod a few days ago is of course I have no control over the last mile. That is when my api turns into the feed that bluesky shows. I can test all the api stuff locally but there is always the potential something is not right with how it talks to bluesky.
For more specific things, like why does this post show/not-show in the feed. I also have a way of running the feed matching manually against any post and the output is specific information about what rule allowed it or rejected it.
I also can run the api and feed matching service locally and compare it to production. Using the firehose you can go back a whole day of posts, so locally I can just run through the last hour or more and see what that looks like.
Sure, I have a few ways that I test the feed. The first is that I have unit tests for the feed matching, ie mock configs and mock posts. This tests the config system works as expected to match posts.
Now that I am back at my computer for couple weeks, I am going to switch over to the new implementation permanently, but still running the old version in the background if needed. #F1 That being said over the last 5 days, the NodeJS feed had a 1h gap on 1/3 that the new one did not, so that's good.
Ended this test for now. Looking good and will switch over for longer next week when I have more time to monitor it.
I am hoping this will put me in a better place to start experimenting with different ways to automate updates to the feed logic as well. Including adding support for user feed back (more/less like this, etc) that you see in other feeds. The goal is to have these changes ready for the next F1 season.
In addition, I've added a automated scaling process that scales the worker process up and down in @fly.io when it gets behind the firehose. This helps keep the costs pretty low.
The feed occasionally sees performance issues and I've had to scale the app manually from time to time. This rewrite takes the same functionality from my original NodeJS app and ports it to golang. There I am better able to take advantage of multi-threading and a lower footprint.
I've been working on a complete rewrite to the F1 feed (and all my feeds), and will be testing it for a couple hours today. If you see any issues with any of my feeds please let me know.
This looks just like my Animal Crossing house (positive)
As we see a new wave of users, I am happy to see this still works.
What autocomplete soup is this?
Doctor Who fans are the Radiohead fans of those other breads no one fit
Apologies for the F1 Feed outage. I had some issues with my hosting site, and it should be catching up now.
Y'all telling me there is no team bias and then the next minute you're saying they are all about landoscar. Which is it? #f1
Rhymes with Skibidi
Zoomed in screenshot of the date for a post on the F1 feed. It says "now".
We're back!
Anyone watching my feeds and seeing old content (~2hrs), they have been affected by some internal Bluesky issues. AFAICT it is affecting all the feeds I follow (other than Blacksky, shoutout to those guys). It seems to be coming back and should catch up soon.
Likes are the only respectful way to save a post. Saves are an act of shame.
It's one of the few levers we have so I am going to use it.
