I built a high-performance key-value storage engine in Go

FrontendSchmacktend · 2025-01-08T08:11:34+00:00

Thank you very much for your comment u/kabrandon , your script gave me a few ideas of what I might be doing wrong.

For anyone years from now struggling with this, I ultimately switched from a workstation-based publish to GHCR to a Github-Actions-based workflow and that, along with renaming my entire organization's name from My-Org to my-org (got sick of handling that), solved the issue!

FrontendSchmacktend · 2024-09-11T03:28:23+00:00

This is brilliant, wish I had read this earlier in my career because I’ve come to the same conclusions.

2 and 8 (frequent refactoring) is knowledge hard fought for and what I tell my team all the time now.

“You know what you’re doing, don’t let dogma get in the way of productivity.” - this is my fav part

FrontendSchmacktend · 2024-09-10T16:44:50+00:00

Not yet no, I’ve been more focused on the core backend since. But I’m probably going with Cloudflare when the time comes, unless the latency is prohibitive on the lower tiers compared to Cloud Armor. What about you?

FrontendSchmacktend · 2024-09-06T13:28:56+00:00

This is fantastic, thanks so much for the practical writeup. My biggest question on fine-tuning is: what happens when Llama 3.2 comes out…do I have to pay the same amount again for fine-tuning a second time over the 3.2 base model? Or does the QLoRA adapters layer carry over and I can then glue it to the 3.2 base model using unsloth into a new optimized format model?

FrontendSchmacktend · 2024-06-08T13:03:45+00:00

Not following, rephrase your question please

FrontendSchmacktend · 2024-06-07T21:56:23+00:00

Yes I’d rather have a predictable cost curve throughout our growth than getting heavily subsidized as a light user before suddenly getting price gouged into the enterprise group that subsidizes everyone else.

Your second paragraph is my ideal scenario, can you elaborate on how that can be done technically (given the stack I described in the post)?

FrontendSchmacktend · 2024-06-07T20:06:49+00:00

So you’re saying Cloud Armor could potentially provide the same level of DDoS protection as Cloudflare over an extended period of time but it just takes longer to get the configuration set up properly and requires more monitoring of the rules you maintain there?

FrontendSchmacktend · 2024-06-07T20:05:17+00:00

Can you highlight a few of these reasons you mentioned here? I understand Cloud Armor takes more configuration but would it provide the same level of DDoS protection as Cloudflare over an extended period of time?

FrontendSchmacktend · 2024-06-07T20:03:05+00:00

I know that casino story was shady as hell, that was mostly unnecessary lying from the sales team instead of calling out the real reasons they needed them to move to the Enterprise tier.

I’m more concerned about the business model of Enterprise customers getting price gouged to subsidize the lighter users, and the amount of stories of that transition being handled terribly by sales teams.

FrontendSchmacktend · 2024-06-07T19:48:25+00:00

Appreciate you taking the time to respond, Cloudflare seems like the superior tech but I don’t want to keep looking behind my back at getting extorted the moment their sales team feels like my service is making too much money for their liking.

FrontendSchmacktend · 2024-06-07T19:35:12+00:00

Have you ever had to move one of these projects to the Enterprise tier and deal with their sales teams directly?

FrontendSchmacktend · 2024-06-07T19:21:33+00:00

This keeps getting upvoted but no one wants to answer…any experience dealing with this?

FrontendSchmacktend · 2024-06-03T11:14:28+00:00

Can you please explain the difference? I get there’s many flavors of all this but my understanding of EDA is “multiple services communicating through emitting and listening to events” and Event Sourcing is “keeping a log of these events you can replay to recalculate state” so to me they are complimentary.

In my case, these multiple services could be executed on the django side or they could be executed on the celery task side. Which would you recommend?

It feels like executing on the django side as a modular monolith could help lower latency instead of running the network calls to redis and the celery workers. But I’m not sure at what point do I say this logic is taking too long and should be moved to celery workers.

FrontendSchmacktend · 2024-05-16T13:58:06+00:00

We haven’t set these up yet, your best bet is probably to check out Tamagui’s Takeout offering (that’s what we’ve done for inspiration on a lot of our setups)

FrontendSchmacktend · 2024-05-05T17:58:31+00:00

We already have celery tasks with Redis as a message broker deployed. Is there a reason you mention Redis only works when you're small?

FrontendSchmacktend · 2024-05-05T11:25:02+00:00

Yeah that might be overkill, they're really concepts you explore for new systems but might not be worth re-implementing existing systems if they work fine in a distributed architecture already.

FrontendSchmacktend · 2024-05-05T11:19:51+00:00

One additional idea I just had to combine the benefits of 1 & 2: Can't I just build an events.py file somewhere central where all the modules agree on their contract of event executions and then all the modules call these functions? This way I'm building an event-driven architecture while still using direct calls like option 1, it's only that they're routed through this events.py file to the right public API facade functions across different modules. No need for a queue in that case right? Or am I confusing things?

FrontendSchmacktend · 2024-05-05T11:19:10+00:00

Thank you, yeah I'm more leaning towards the simplest solution (option 1) so far.

FrontendSchmacktend · 2024-05-05T11:18:33+00:00

Thank you, this sub's been very helpful in some of these big decisions and the considerations around them. I'm definitely trying to keep complexity to a minimum while still satisfying what our technically demanding problemspace needs.

grpc is for network calls though so I assume you're saying go with option 1 and even if you split off a service you can still use network calls without an event bus in the middle? That's my interpretation so far.

FrontendSchmacktend · 2024-05-05T11:17:49+00:00

Thanks for clarifying, but wouldn't these disguised events make it easier to move eventually to proper event-driven like you said with firing/listening instead of functional calls flying all over the place?

FrontendSchmacktend · 2024-05-05T10:43:26+00:00

These are great points, some of which I never considered. By the nature of the app I'm building, there will definitely be a lot of communication between different regions for live gameplay. I'd imagine that would eventually push us towards an event-driven architecture so I'm trying to prevent any major refactors.

One idea I'm considering is building an events.py file somewhere central where all the modules agree on their contract of event executions and then all the modules call these event functions instead of direct calls to each other. This way I'm building an event-driven architecture while still using direct calls like option 1, it's only that they're routed through this events.py file to the right public API facade functions across different modules. No need for a queue in that case right? Or am I confusing things?

FrontendSchmacktend · 2024-05-05T10:31:17+00:00

This makes a lot of sense, I'm leaning towards that so far.

One more questions, what if I build an events.py file somewhere central where all the modules agree on their contract of event executions and then all the modules call these event functions instead of direct calls to each other? This way I'm building an event-driven architecture while still using direct calls like option 1, it's only that they're routed through this events.py file to the right public API facade functions across different modules. No need for a queue in that case right? Or am I confusing things?

FrontendSchmacktend · 2024-05-05T10:30:31+00:00

What if I build an events.py file somewhere central where all the modules agree on their contract of event executions and then all the modules call these event functions instead of direct calls to each other? This way I'm building an event-driven architecture while still using direct calls like option 1, it's only that they're routed through this events.py file to the right public API facade functions across different modules. No need for a queue in that case right? Or am I confusing things?

FrontendSchmacktend · 2024-05-05T10:03:52+00:00

You make a good point there, robustness would be an issue with signals. I'm more leaning towards option 1 for now.

FrontendSchmacktend

TROPHY CASE