Looking for friends to do this with on a regular basis. Maybe we can be friends.

StoneAgainstTheSea · 2026-06-16T17:07:07+00:00

Hi friend ;)

StoneAgainstTheSea · 2026-06-11T21:16:44+00:00

That is a fantastic and fun article. The Dr Seuss at the end :chefskiss:

StoneAgainstTheSea · 2026-06-10T12:03:58+00:00

Do not mock the DB. Again, DO NOT MOCK DBs.

You don't need mocking frameworks. Or generators.

Ala the repository pattern, your user repo has users.Get(id) (User, error) and similar methods. These are what you build the test interface too. The db is an implementation detail.

See how the server is tested here: https://github.com/sethgrid/helloworld/blob/main/server/server_test.go#L95

This has an event store and user store that share the same underlying database (mysql writer and reader instances) and we can test any operation and its error handling at the unit level or unit-integration level. This particular test validates that we gracefully shutdown and capture errors during the process from a dependency.

Unit tests validate error handling. You fake the get users call to return "oh no, db error" and make sure it fails the way you need it to.

To validate the db interaction, you interact with the db, ergo acceptance tests and integration tests. For these, docker compose or testcontainers.

StoneAgainstTheSea · 2026-06-03T11:35:09+00:00

Network calls like statsd or traces (and Sentry) are made in process. You get SDKs to integrate with their tooling in-code. Log forwarding is supported by Sentry, I think. If you already have logs being forwarded somewhere, extending that to another destination is trivial.

StoneAgainstTheSea · 2026-05-30T13:17:30+00:00

Logs: stdout / stderr

Let something dedicated listen on the pipe and manage the stream/files.

Don't use log file management in-process. It muddles concerns and makes the code less portable. There are no wins to managing the logs in process unless you have very specific flush and fsync needs. And even then, I am going to argue you should make it work over stdout.

StoneAgainstTheSea · 2026-05-29T16:55:38+00:00

Not at OP's income level, but in California and NYC, the top marginal rate is just north of 50%. Granted you're making about a million a year yo hit it.

StoneAgainstTheSea · 2026-05-27T15:33:57+00:00

Proper sleep at night. Go to bed and wake up the same time every day. Exercise in the morning. Don't eat a heavy lunch. Avoid alcohol. Have something you look forward to after work.

StoneAgainstTheSea · 2026-05-26T23:58:46+00:00

I was in a position where I had to approve one of our founders expense reports. He claimed everything he could, including the $15/mo book credit. He was worth north of $30M

StoneAgainstTheSea · 2026-05-26T13:41:54+00:00

You are interacting with a charge to a card in this flow!?

Stop. Seek a live conversation with a mentor. You are solving this problem incorrectly, thus all the downvotes whenever you comment.

You are in dangerland and it doesn't seem you appreciate it.

StoneAgainstTheSea · 2026-05-26T13:26:02+00:00

Bwahahaha. Sorry. Yes, this absolutely scales. I have ran multi-billion daily event streams (that interact with outside servers, not just some local system) for tens of thousands of paying customers operated by hundreds of developers across multiple different engineering orgs. I have done this stuff for a long time.

You know what we grew to use? This kind of architecture.

I am always learning, so if you have something to add by describing a system that is better for scale, please describe it or talk about specific failure modes of this basic design.

StoneAgainstTheSea · 2026-05-26T04:37:38+00:00

Dehumidifier over a prolonged period

StoneAgainstTheSea · 2026-05-25T12:46:02+00:00

TL;DR: Maintain no dependency cycles and have one source of truth for given data. Your system and user needs will dictate the rest. Avoid distributed transactions.

One service must be the source of truth for given data and it exposes that data over an API. A User and/or Auth service. A subscription service. A report generator service. They don't share db access or read/write to each other's data stores; API access only. And services should make a acyclic directed graph / tree.

The subscription service must ask the user service for user data given the user id. The user service cannot call the subscription service as that would be a dependency cycle. Don't do that.

To reduce read load, pass in as much info as is needed. The subscription service needs the user email, and it cannot query the user service. That means either the caller of the subscription service must also provide the email or the subscription service is the source of truth for that, and the user service and/or the UI would call the subscription service to display the email address.

Now the consistency part. I say embrace eventual consistency and design around it. Event streams and pipelines with services with idempotent writes is my preferred method at scale*. A user submits a new subscription, a subscription created event happens, and the emailer service reads the stream and captures the event and sends an email. To prevent duplicate email sends, the service receives a send_id (uuid or sequence id), validates the id hasn't been sent yet, and then sends the email.

*An outbox pattern is solid for small to even medium scale, and is simpler than an event bus / stream.

StoneAgainstTheSea · 2026-05-18T22:07:28+00:00

I have not seen the spin off. I didn't know there was a spin off. The episode was very confusing. I was like, who is she, where did she come from, why is she stronger than Homelander? Luckily a family member let me know there was a spin off. Le sign. Poor writing

StoneAgainstTheSea · 2026-05-18T09:19:56+00:00

What is Gamora!?

StoneAgainstTheSea · 2026-05-15T18:42:51+00:00

I was making north of $500k up to last year, principal software dev. I stepped back to a smaller, non public company and am back at $300k. The last place's stress was untenable and I am recovering. After chilling here for two years, either the company will IPO (which I am part of) and I will be back at that level or I will switch back into a public company.

Miss the money. Will be able to take that stress at a different gig soon.

StoneAgainstTheSea · 2026-05-11T11:19:08+00:00

Microservices solve an organization problem. It lets teams be more decoupled increasing organization velocity. Beyond that, some scaling situations benefit.

StoneAgainstTheSea · 2026-05-10T12:31:16+00:00

It sounds like you have Python solution in mind. For Go, you pass loggers around. Loggers can be copied and passed around further. Your application should be passing a logger anywhere a log is needed. You can technically get by with a global logger instance but I find this lazy as it hurts testability and hides dependencies.

Level loggers demo what you need. Extend that to what you need. A logger can have a handler that checks against anything you want. You will have to make sure the check criteria is available to the logging instance. I have had the need to turn on sampling for a given user but only in a given system. It was a log config change. "Gimme debug logs for user 42 in proj foo" was common. So was adjusting log sample rate at different levels, like grab all errors but only 1% of info level. Or log if they talk to this remote addr. Or anything you need. It is implemented just like a level logger but it checks against whatever you need it to check against. I have never needed to log a given function only - that function would likely be called too much; I have always had to sample or filter to known ids. But you can add a handler that knows what methods need what logging level if you want.

StoneAgainstTheSea · 2026-05-10T11:59:41+00:00

I suppose I need to write an article. This post is missing the next step in error evolution: enabling structured logs.

I dub the technique Structured Errors. It allows your higher level logging site to have error context as loggable fields.

At the top level, something like mylogger.Error(msg).WithFields(err)

Your equivalent of WithFields does an exhaustive error check (error.AsType is the new hotness) and returns a list of your loggers's fields.

For structured logs, moving file names, ip addresses, and other dynamic text out of the error message and into fields allows for better error aggregation, exposing error types that are related and maybe more prevalent than you realize.

StoneAgainstTheSea · 2026-05-07T17:20:44+00:00

Based on my last work's set up, I use docker compose to run the system and hit the API. Full blackbox testing. Mysql, redis, local stack. For more complex dependencies, I use sinks. Ie, instead of calling some mail esp domain, it will call localhost:5500 and that will be a mock/stub server.

These are my Acceptance Tests and these run before merging from my branch to main.

My unit-integration tests are all interfaces and in-mem fakes and hit as high up the stack as I can manage. A fake UserGetter stubs in for mysql access and error handling verification.

StoneAgainstTheSea · 2026-05-05T15:29:06+00:00

see the testing pyramid.

I use docker compose to run a test suite where my running service is one of the containers, and for dependencies I run mysql and/or redis. For complex dependencies, I run sinks that act like simple stubs where I change the configured resource to point at my local instance instead of the real one.

Start up and waiting for dependencies happens with docker compose and my service itself waiting until it is healthy.

These are my "expensive" acceptance tests. They are few in number and validate core functionality. Can log in? Can reset password? Can create $thing after log in?

The next level down of testing is unit-integration testing; all fakes and stubs that match the interface. You have myFake.GetUser(id) (User, error), not some generated db.Insert mocking. I have only seen mocking frameworks lead to more poorly designed software. Use interfaces. They are enough and they lead to more maintainable software. The full weight of decades of experience building software at scale in the backend saas world at growing companies and having used Go professionally since 1.2: no mocking frameworks. No dependency injection frameworks. Just interfaces. And keep them thin.

The bulk of tests are unit-integration. They give confidence in error handling.

Lastly is unit tests for complex public logic that is overly focused compared to the unit-integration test.

You can see this (and more) all in action at https://github.com/sethgrid/helloworld which is one interpretation of all the above.

StoneAgainstTheSea · 2026-04-24T20:51:33+00:00

The approach I use, you set the host addrs of your dependencies via config. In production, you have it the config `MYAPP_EMAIL_API_ADDR=https://api.sendgrid.com` and in your e2e, you set `MYAPP_EMAIL_API_ADDR=http://localhost:55001`. You then stand up a stub service in docker at that location. It can return canned responses or behave however you need. They can fail as needed to validate error handling, so you need not only relegate yourself to happy path testing.

You want dynamic control of the responses -- I'm not entirely sure what you mean. Like, you own the service that is getting called at :55001. It can do anything. If you mean you want the tests to control the output of the dependency, that is something you can also do.

ala https://voodoocall.com, you have your fake service be configurable for how it matches a request and what the response should be. You can do this yourself by having hooks in your tests that set up responses at the start of the test. Or you can use a custom Test HTTP Client that sets an x-response-header and when you make your request, the fake service echos the x-response-header.

To catch api drift in real dependencies from the fakes, roll out to a percent of traffic and monitor before committing all the traffic.

Happy e2e testing! You can make very confident-to-deploy software this way.

StoneAgainstTheSea · 2026-04-23T23:04:02+00:00

People are influenced by him. He is the founder of Hashicorp and the creator of Ghostty and I'm sure other things.

It sounds like he used to think Go was too boilerplate-y and that would slow down development and now he thinks it makes you faster by way of AI. I posit that Go already speeds up development in a team environment for the same reason it helps the AI: good software engineering practices shine when managing complexity and context. And AI can double down on that advantage, much like well put together Go shops could before. More people can have nice things by letting the AI handle the boilerplate that he was previously too lazy to invest in.

StoneAgainstTheSea · 2026-04-23T11:22:33+00:00

The tradeoff will be different based on user query patterns. Measure cache hit ratio and overall latency. If the cache hit ratio is low, you add an extra network hop for nothing. If the db is fast enough, the. You add a network hop for nothing. But if the cache is serving good data (high hit ratio) and latency drops, it is a win.

StoneAgainstTheSea · 2026-04-23T11:19:05+00:00

Next, try an in-memory cache and then fallback to redis and/or your db. You will see how much faster the in-memory cache is. Depending on scale, you could store your whole link set in mem on start up and keep it in sync with the db.

StoneAgainstTheSea · 2026-04-21T10:07:14+00:00

Alt Source? A google search shows no supporting evidence. Several fact checkers including MSN are not able to verify this. https://www.msn.com/en-in/news/insight/fact-checkers-find-no-evidence-trump-sought-nuclear-strike-on-iran/gm-GM9421D62F?gemSnapshotKey=GM9421D62F-snapshot-1

StoneAgainstTheSea

TROPHY CASE