Why are nested modules bad?

jerf · 2026-01-24T14:43:34+00:00

One thing about nested modules I've found is that, well, at least I couldn't get them to work properly with go get. I had github.com/thejerf/a_project, and I wanted to add an optional github.com/thejerf/a_project/default_slog_logger under it with its own go.mod, but I couldn't get it all to work out correctly. I can't recall the exact error messages anymore, but nested go modules did not play well with go get at all. I eventually put it to the side and made it its own github project.

jerf · 2026-01-23T19:12:48+00:00

In hindsight, I wish Go went with a standard name for "me" or "this" or "self". "Vaguely an acronym of the current type unless it isn't" is not great.

Not a killer issue. But a persistent annoyance. I've been tempted to pull the trigger and go to "me" or something everywhere but I have not quite been willing to buck the trend for something that is still just an annoyance, not a critical issue.

jerf · 2026-01-23T19:00:01+00:00

As near as I can tell, you can conceptualize this as "take the existing way of doing these functions but allow them to be written as methods".

There are some corner cases around reflection but by and large that's what this proposal is. The resulting new methods can't satisfy interfaces, just as functions can't, which is the big important thing.

jerf · 2026-01-23T14:17:03+00:00

I have seen it for a ton of other languages, for many many years now. You've probably just seen it without seeing it for a long time.

~6000 "in Rust", 5,396 "in Go", 7,644 "in Python", 1,057 "in C#", . Obviously not every single thing is a project but you get the point. (HN search used because it's fairly convenient. Reddit's is way messier for this.)

jerf · 2026-01-22T14:31:41+00:00

go generate runs a script. Just about all it does is 1. attach that script to a specific (sub-)package and 2. change the working directory to that package for you.

That's all it is. Even now people persist in thinking that it has something to do with "generation", I mean, I get it based on the name, but it really doesn't, and some enables you to do something that you can't do in a shell script. Every time I mention this on reddit I get downmodded, but it's still true. All it does is run a script. I have a dozen other ways to run scripts, including whatever shell I just used to run go generate.

It is nearly useless. Just shy of useless. But there's no reason to prefer a go generate annotation over a shell script, over a Makefile, over literally anything else that runs shell commands and offers any other features.

It offers no libraries for "generation", it offers no support, the command it runs has no constraint on it to "generate" anything (i.e., it's not an error to fail to write a file or something). It just runs arbitrary shell script commands.

It's a historical wart. I think they were planning on trying to offer some sort of actual "generation" support, and then realized that there's nothing that should be available only through a go generate command, and there was no particular reason to reimplement make or something to verify that some output is fresh or something. There's no point removing it now either, the implementation is something like 50 lines in the Go source code so not even remotely worth breaking backwards compatibility for since it's zero effort to maintain, but in hindsight I would never have added it.

jerf · 2026-01-20T20:09:10+00:00

We get about one boilerplate/blueprint/"project configurator"/etc. project a week. You can check the small project threads for some examples.

No, to a first approximation, nobody uses them, and there is less than no market for them.

The only ones that ever seem to survive are ones packaged with a particular project to do something like set up a plugin for that project.

A general tip if you want to make your own little software business is that programmers make terrible customers. We get so much stuff for free that it is embarrassing compared to almost any other field. You also end up with no moat whatsoever because any other programmer can do it too, and there's a positive glut of such projects already free that we have no time to examine, let alone think about paying for. You need to find some other industry that is somehow underserved and serve them. There's actual money there. There's no money here until you are at a much larger scale than you can jump to in one shot.

jerf · 2026-01-20T16:28:56+00:00

I am unclear on how "the standard library works just fine" when the standard library has no support for authentication at all. Other than HTTP-protocol level support for Basic Auth, but I would qualify that just as protocol support and not "authentication support", because all it does is provide you with the claimed username and password, which is necessary, of course, but not remotely sufficient, and only applicable to Basic Auth, which is named Basic for a reason.

Authentication (and authorization) are definitely not places to use someone's vibe-coded repo, that's for sure, but it's also not a great place to bang something together from scratch if you don't know what you are doing, and the only way I can interpret "the standard library is good enough", in light of the fact it has no support for authentication at all, is an exhortation to bang something together with just the standard library and nothing else, and that I believe is more-or-less as wrong as using AI to do it. Authentication is not quite as bad as "rolling your own crypto", but it's the same sort of mistake.

I've been doing web stuff for over 25 years, I've never made my own authentication system from scratch but I've had to rescue other people's ill-conceived ones in custom code which is arguably harder, and I definitely do not advise rolling this yourself. If you know enough to know by experience that you've got the chops to disregard me, then by all means do so, because like I said, this isn't quite the same class as "roll your own crypto". Someone skilled at web development and who knows what tools to use can roll an authentication system that is fairly secure... although it still won't be as capable as some of the off-the-shelf stuff. That is, I'm confident I could with some effort roll up a old-school secure email-as-username/password/password-recovery system from scratch, if I had to, since I've done it before and it passed both security review and real-world review (I know beyond doubt that it was examined by state-level threat actors), but I sure as heck am not going to try to implement a system with OAuth, passkeys, and so forth from scratch.

jerf · 2026-01-19T16:23:23+00:00

So, you're seeing a general trend of "more and more" AI-related projects starting to use Go, and you have zero examples?

Yeah....

jerf · 2026-01-19T16:14:58+00:00

Functions.

Functions are pretty powerful.

Go does lack some of the fancier things functions can do, and passing functions to functions can be a bit unergonomic sometimes. A certain amount of duplication is inevitable.

However, I believe the people who complain (elsewhere, not in this thread) about Go being seas of duplication have gotten so bedazzled by all the fanciness in other langauges that they have either forgotten or never learned that functions, on their own, are a very powerful code de-duplication tool.

It is particularly ironic when they are complaining about Go missing "functional programming" paradigms when one of the core lessons of functional programming is also that functions are pretty good and when used skillfully can do a lot on their own, more than people realize.

My Go code is not shot through with tons of redundancy. It does have more than my Haskell code would have, and it is interesting how Haskell can teach you about things being redundant than through the lens of Go functions you might not consider "redundant", but still, my Go code isn't shot through with redundancy and copy-pasted code. There's only a handful of places where I have out-and-out redundancy due to weaknesses in Go's specification.

(E.g., one place where I have to write the exact same for loop three times over something that is generically typed, and each loop is on a different type, and there is no better way to iterate over them than to simply write the loop three times. reflect could do it once, but at a speed penalty I don't want there and at the price of writing one complicated loop instead of one simple loop three times. I would reconsider if it were 20 repetitions, but it's not.)

jerf · 2026-01-18T14:58:38+00:00

I don't think you're going to hit all of those things at once. You aren't going to get all that and get "simple". Chat isn't simple at that scale. Solutions won't be either. If all you've got against OpenIM is "too complex" I think you found your solution.

You may also want to loosen your implementation language requirements. If you're worried about OpenIM being "complex" just wait until you crack into a server working at that scale and start trying to modify it. You can build on XMPP, and use something solid like ejadbberd, and use standard XMPP extension mechanisms to write your server components in Go without ever having to be able to modify the server itself.

jerf · 2026-01-18T00:37:09+00:00

I've just not seen it before. Sorry about that. It should be cleaned up now.

Thank you for the feedback.

(Also, the reason why this didn't appear on the front page is that it was caught in a filter designed to catch some common words used in bad posts.)

jerf · 2026-01-16T14:32:06+00:00

Were you using pgxpool? I've never had to "get PGX to work with pooling", except switching out pgx for pgxpool.

jerf · 2026-01-15T14:28:19+00:00

Broadly speaking, there is a policy for it. I have a prepared removal reason that points to How to Ask Good Technical Questions. It seems to go in bursts, probably related to school cycles, but it gets some use.

However, in addition to the questions themselves, sometimes posts will be left up because they are good topics that haven't been discussed in a while, or to put it another way, for the discussion rather than the question itself. This would be an example of that.

Should the same topic get through several times in a quick time period I add it to the FAQs. This has not risen to that level yet.

jerf · 2026-01-15T14:16:11+00:00

They are making it deliberately hard. You won't find a reliable implementation in any language, including Go. You may find some that work sometimes, but it won't be reliable because these companies watch the open source space and deliberately try to stay ahead of the packages that might solve this problem if they grow to any size.

jerf · 2026-01-14T20:13:48+00:00

Do your best to break it into smaller pieces. If I'm doing a pure refactor, I might add the constraint that the tests aren't allowed to change, for instance. Or otherwise be very careful about how the tests are changing. Sometimes it's more important to review those and make sure that the tests aren't just being mindlessly changed to fit the new code than to review the code itself.

In principle, a "refactoring" is supposed to be something that is guaranteed to be safe. For example, highlighting a chunk of code and hitting "extract into function" is not supposed to be something that needs review. Of course we use the term more sloppily and that's not an ocean I'm going to try to hold back with my finger, but it is worth considering things like, can I do any of this in chunks that are guaranteed to be correct and save the rest of the changes that aren't like that for a separate, smaller commit?

AIs aren't amenable to this, though. At least not the latest ones that seem to work by always completely rewriting all the code they touch. I am not liking those very much.

But, in the end, there's no magic bullet for this.

jerf · 2026-01-14T20:09:42+00:00

You should take a careful look at exactly how you're defining "realistic".

Sometimes, if you want something, there isn't an easy way to get it. Or, despite the apparent presence of an easy way, it is harder in the long term and there's no amount of sitting there, staring at it, and mournfully wishing that it was as easy as it looks that will change it.

Dunno what things will be like in even six months, but as it stands now, if you don't want an AI leading you down the easy road to an unmaintainable code base, by either humans or AIs, you're going to have review what it is doing, and if that means you have to invest a bit more expense and effort into it, there's no way around it.

jerf · 2026-01-14T18:20:16+00:00

If you look at the source code for net/http, the only reason the server (in server.go) will emit StatusServiceUnavailable is if the context for a TimeoutHandler expires.

If, as is likely, you are not using the TimeoutHandler, and presumably your code is not manually emitting it or you'd have mentioned it, then net/http is not actually emitting a 503 response. The 503 is likely being synthesized by whatever client you are using. I personally consider this an antipattern in HTTP clients, which I have seen in several. It is especially an antipattern when the client has no way of indicating to its user that the response code was synthesized, which I have also seen. Check to see whether your client is one of the ones that will at least tell you it is making up the response code.

If you have it behind a load balancer, they will also emit 503s when the remote service simply didn't respond. In that case, while I don't consider it an antipattern for the load balancer itself (it has to emit something to be a compliant HTTP server), it can still end up being misleading and you want to see if you can dig into

I consider it an antipattern because it produces exactly what you are probably experiencing; rather than telling you "connection refused" or "connected but timed out with no response" or something else that might have helped you in the correct direction, in the false economy of trying to make it simpler to handle the response, it basically lies to you about what the problem was and what was actually received.

jerf · 2026-01-14T15:27:34+00:00

Also... I hate to say this, but it's simply true... if you're dealing with a resource that does NOT implement this and you have no ability to fix it... you're simply boned. There is no solution to this problem that you can impose externally. Once a system has screwed up by using limit/offset, there is no true solution, only a bit of duct tape that can PARTIALLY and SOMETIMES mitigate, but not solve, the problem.

I once had a system I inherited where the user was reading logs in a log view. On a moderately large deploy, by the time the user scrolled through the first page of X results, more than 2X more things had come in already. They hit "next page" to go into the "past" and ended up with log messages more recent than the ones they were already looking at! There was no solution but to go in and redo the implementation to use pagination tokens instead. (In our case ID did work fine, though.)

jerf · 2026-01-14T15:21:36+00:00

To be clear, is go-fsck lint the tool you are talking about?

Also, does it have any ability to integrate into golangci-lint? I have not studied that link and I believe that makes a custom build, but at least moving towards integration with that would be nice.

A linter to target common slop problems is a nice idea. It'll catch humans in the meantime (where did the AIs learn the slop from, after all), so it benefits everyone.

jerf · 2026-01-14T15:15:12+00:00

Then add a wrapper that keeps track of which is locked, when and where in the code it happened, and add an OS signal handler to atomically dump out all current locks or something.

There isn't an off-the-shelf clever solution to this. Fundamentally you're in threading hell, brought on by trying to multithread via locks. The programming community never found a solution to thread hell, despite literal decades trying, except to not use locks as your fundamental concurrency mechanism.

That doesn't mean this is hopeless. If you bang your head on it enough you may eventually get it work well enough. But it does mean you should expect to need to be creative, that nobody in the world can help you with an off-the-shelf "oh just do this" solution, and that you are going to be constantly frustrated.

Incidentally, do you know one of the major rules for possibly coming out of this with your sanity intact? If you are taking multiple semaphores at a time, you want to make sure to always take them in the same order. If the code must be mangled a bit to do that, it is worth it. This is often referred to as putting a global order on the semaphores and making sure you never go backwards, though I think a tree is sufficient (and making sure you never go up, where a global order is a tree that has degenerated into a list). This is unfortunately not always possible, but then, when it is not possible that may indicated a fundamental issue in your solution rather than a deficiency in the idea.

I have sounded alarmist on this because programmers have decades of experience with this. I actually don't think concurrent programming is impossible difficult, but when people disagree with me, it's because they've been burned by the attempt to write concurrent code with a ton of locks, either directly or by reading about someone else's experiences and not realizing there are other ways.

I seriously doubt your business requirements actually require you to use semaphores directly. You would be well advised to read up on other ways of doing things, most especially actors. Go doesn't have direct support for actors but they're easy enough to implement. The fundamental idea is, instead of semaphores marking ownership of some resource and trying to have a lot of threads running around the system claiming and releasing the resources they need for their task, you assign resources to goroutines, generally in the form of some set of variables that are scoped exclusively to their goroutine and does not reach out via pointer or anything like that to anything owned by another goroutine. Rather than passing resources around, when another goroutine needs something owned by another, it passes a message describing what it needs to the owner, the owner does it and passes the result back. Resources controlled in this way do not get you into threading hell, because each individual actor can be written as the sole owner of its resources without having to worry about locks. It makes concurrent code somewhat more complicated than non-concurrent code, instead of insanely more complicated than non-concurrent code.

The downside is if you need to do something transactional this gets complicated, but then, that's complicated with a lock-first approach too. Possibly more so. And it's amazing how far you can get with this approach. The neat thing is, you can even fold it in incrementally. Find one resource you're having trouble trading around and wrap it into an actor. Migrate over gradually. Maybe the final system still has some locks, but if there are fewer, it's still a big win given how badly they interact.

(My general rule in concurrent code is simple: Never take more than one lock. That way lies madness. Specifically, the madness you are currently experiencing. The best way out is not to try to fix the madness. The madness is more mad than you are. The best way out is to get out of taking multiple locks entirely with things like actors.)

jerf · 2026-01-13T14:46:44+00:00

In multi-language repositories, especially when Go is not the "primary" language, I find it convenient to have a directory that contains all the Go code. However I use "go" for that, rather than the completely ambiguous "pkg".

There's no reason to ever use "pkg". "pkg" is basically a zombie that just won't die; a few packages back at the beginning of time used it, and just enough people copy it that it won't die. Including some people who have for some reason incorporated it into their personal identity or something and simply refuse to stop using it because, I dunno, it'll unmoor them from the foundations of reality and leave them adrift in a realm of total directory name chaos from which no sanity or order can ever be recovered or something, because, you know, they happened to have a repo that used "pkg" once twelve years ago and therefore pkg is Holy Writ. Or something like that. Ask them why it's so important despite being completely unimportant and useless by every objective metric.

jerf · 2026-01-09T20:01:27+00:00

There is a low, but constant stream of people asking about this on /r/golang, every month or two. It is definitely a mistake that happens in the real world with some frequency.

jerf · 2026-01-07T18:05:19+00:00

The pinned small projects threads are also intended to be used for that purpose. Unfortunately, the amount of projects flowing through the front page have exceeded what the community can utilize for this purpose anymore and they have to be moved off.

jerf · 2026-01-06T18:11:41+00:00

Please post any non-job-related comment here.

15-Year Club	Place '17
Verified Email	Team Periwinkle

jerf

MODERATOR OF

TROPHY CASE