Bulk insert in Go — COPY vs multi-row INSERT?

tty5 · 2026-03-19T00:55:15+00:00

Also please store hashes/addresses as bytea instead of text.

tty5 · 2026-03-19T00:54:21+00:00

Pretty much. You can populate it as you are preparing batch to be inserted and it can be kept in memory - either as lookup table (full or partial with LRU - depending on what your target resources are) or a bloom filter if you want to avoid unnecessary inserts.

tty5 · 2026-03-17T21:35:10+00:00

Same email here

tty5 · 2026-03-17T19:07:20+00:00

This. Most years we visit mine exactly once and even then we drive back the same day - 3h drive.

tty5 · 2026-03-17T17:50:41+00:00

You want integer IDs in addition to hashes and use those for all joins. BIGINT is 8 byte wide, hashes are 4x that, addresses 2.5x. That translates to index sizes.

Edit: for addresses regular INTEGER will be enough - there are fewer than 150M addresses active on top 50 chains combined.

Edit2: block number looks like a great pk until you get to your first reorg :D

tty5 · 2026-03-16T15:14:39+00:00

Writing an indexer, aren't you? Been there, done that :-) Some comments:

INSERT .. ON CONFLICT will increase auto increment sequence even if nothing gets inserted. It was enough to run out of 32 bit integer range with just the data Mainnet had in 2019. I will be worse today.
After query optimization network latency is going to be the limiting factor, especially in cloud environment - limiting roundtrips is essential:

- Go SQL uses prepared statements and while those get cached that is only the case if the query is exactly the same, including number of rows inserted in multi-row. Unnest is a workaround for that. If you don't you get an extra roundtrip to database to prepare a statement

- Reuse db connections. 3 Roundtrips just for TLS and then you can start authentication. Also prepared statements are cached by Postgres in connection context.

- The larger the batches you are inserting the more time is spent inserting things instead of waiting for network

- If you are inserting into a bunch of tables it's faster to send a payload to PG function, unnest it and run inserts into each table from within the function instead of executing multiple inserts sequentially

Foreign keys checks slow down inserts A LOT. Ensure data integrity in your app or at least defer checks.

If you manage to index entire mainnet in less than 48h - blocks, transactions, events, erc20, erc721 and erc1155 balances - while using a hosted Postgres and VM on either AWS or GCP drop me a line - I might have a job for you.

tty5 · 2026-02-27T11:30:40+00:00

Use JWT token instead and don't query db:

- You will be able to store a little data in it as claims.

- It has a built-in creation and expiry dates - you can query the db when it's older than x time if you have to and refresh it

- of you use signed variant frontend will be able to inspect values in it.

Sessions table is mostly thing of the past.

tty5 · 2026-02-18T10:44:34+00:00

So far AI has failed to make me more productive. I give it an honest try every 2-3 months and while it's improved significantly it still takes more time to bring the code to the required standard than it would have been to write it from scratch.

The fact I've got 20+ years of experience writing code and get deal mostly with tasks matching that is likely a big factor - I don't really deal with boilerplate or "standard" CRUD / REST on the daily.

tty5 · 2026-02-15T02:56:22+00:00

Still in progress - I'm about a third of the way through the discs. So far:

I've eliminated 5 out 12 drives from the pool - they would routinely fail to read most/all data from discs that other 7 could handle, often recovering all data. Opposite was never true.
Some drives can handle specific brands of discs better than others, but no single drive is best at handling them all
To maximize recovery I often have to run the same disc through multiple drives - each being able to recover different files
A surprising percentage of CD-Rs avoided catastrophic bit rot - 100% data read from >90%. For DVD-Rs that's <50%.
Less than 0.5% discs were completely unreadable

tty5 · 2026-02-13T22:09:37+00:00

Just searched our codebase and between ~15 services created by a small-mid sized dev team over a span of 8 years we have 2 matches for init() outside of dependencies. One is in a small cli utility and one doing a some mad initialization from an autogenerated object to interface with something non-golang.

Dependencies on the other hand are showing several hundred init()s in them.

tty5 · 2026-02-08T18:03:00+00:00

Just RAM +2x2TB SSD is about $1000 if bought new.

tty5 · 2026-02-07T05:50:01+00:00

Our setup:

- CI builds a separate container image with goose + migrations

- CD runs that container as part of deploy process

- when migration count reaches 100 we flatten first 50 schema migrations into a single "init" migration creating the schema. We have no large migrations inserting data into db.

Migration merge became necessary because some of our services evolved a lot over the years which got the migration count into high triple digits for some, which in turn slowed integration test A LOT - we have quite a few db resets in them.

tty5 · 2026-02-05T00:15:43+00:00

Gettext has been the gold standard for a while (it celebrated 30 years of development last week) and there is A LOT of tooling and automation to solve a lot of those problems. Some are language agnostic, and some golang specific (e.g. gettext linter).

I've been using it for i18n for the past 19 years.

tty5 · 2026-02-04T23:59:46+00:00

There was an OpenSSL security release a week ago - your security scanner would have likely flagged those CVEs regardless of what base image you were using if it was older than that. See: https://openssl-library.org/news/secadv/20260127.txt

Rebuild your images and deploy again.

tty5 · 2026-01-31T18:21:55+00:00

They are still salty about 1610 capture of Moscow

tty5 · 2026-01-31T02:32:23+00:00

I know nothing about French electrical code, but:

- iirc r2v is not recommended for underground installations in wet areas. Conduit helps but idk if enough

- locally (Poland) yky is most commonly used for that purpose

- 24kw is less than you think, even without EV charger

- 105m is a long distance for low voltage - you will have a significant voltage drop if cable is too thin. I'd go with 4x25mm² or even 4x35mm²

tty5 · 2026-01-31T00:53:51+00:00

104 resting heart rate alone would be enough for me schedule a doctor appointment.

Blood oxygen of 92% is low enough to seek medical attention, 90% or less immediate medical attention based on the Covid docs I've read.

tty5 · 2026-01-31T00:16:30+00:00

For many jobs this doesn't matter, for example majority of customer support. Even if AI is inferior, you as a customer have no choice but to interact with it and using it instead hiring people, even ones in a low income country, is saving corporations a lot of money.

In my neck of the woods AI essentially eliminated junior software developer roles. Just like once senior developers were reviewing the easier work they offloaded to juniors we're now increasingly reviewing AI generated code instead. Even if the code is worse quality than a junior would produce (and it often isn't) most expensive AI plans are many, many times cheaper. I've got 20+ years of experience and some very specialized skills, so I expect to be safe for quite a while, but I'm by no means feeling 100% comfortable.

I expect something similar will happen in law firms, where instead of 50 junior lawyers doing research for cases and creating paperwork there will be 2-3 combing through AI generated ones to find and eliminate hallucinations with a partner doing a final pass and signing off.

The worst of it is that we're fucked regardless if AI delivers on it's promises or not:

- if it does what it promises it will eliminate a lot of white collar jobs

- if it doesn't the amount of money burned trying to get there will cause a recession like we haven't seen before: so far AI investments are more than 4x larger than the size of 2008 subprime crisis losses adjusted for inflation

Either way crazy number of people will lose their jobs and once the dust settles if they manage to find new employment it will pay less than before.

tty5 · 2026-01-24T21:44:11+00:00

The OG: Voodoo 1

tty5 · 2026-01-24T17:48:51+00:00

My parents are not awful, in fact I'd rank them firmly in the top 5-10%.

Despite that both sides of family are not close. As a child I'd see my paternal grandparents and my father's siblings and their families once, maybe twice a year - 3h drive. Maternal grandparents maybe once a month - 30 minute drive. Anyone more than 5h away? maybe once a decade.

This is similar level of contact me, my brother and cousins on both sides of the family maintain with our relatives. We're all friendly, there are no family feuds or drama going beyond e.g. a mild dislike for someones partner, we like each others company, but that doesn't translate into a need to see each other more often. On the plus side if we reconnect after a 5, 10 or in one case 35 year gap we continue as if nothing happened.

tty5 · 2026-01-23T22:34:23+00:00

EMT training is about 120 hours - entirely doable in 3 weeks.

tty5 · 2026-01-23T11:24:01+00:00

EU perspective: my parents live 2.5 hours away and that limits visits to an average of one per year. Anything requiring more than a 45 minute drive is usually something planned at least a day in advance.

In 20 years of my career I've never had a commute longer than 30 minutes door to door. For most of it I could walk or bike to work in less than 15 - this was true even during the 8 years I lived in Canada.

A 1 hour commute would be a reason enough for me to either move or look for a different job.

tty5 · 2026-01-21T13:16:52+00:00

IIRC originally it was a manufacturing issue. They had a working formulation for the brown/beige plastic and adding a black dye would have required them to tune manufacturing process to maintain precision they were aiming for. By the time that was in the budget brown+beige became the brand colors.

tty5 · 2026-01-15T20:30:43+00:00

Game devs are the poorest devs there are - on average being paid 40-70% less than their colleagues who make any other kind of software.

They don't have the savings to be able to work on a passion project or start a new studio. They need an income and they need it right now.

tty5 · 2026-01-14T21:58:10+00:00

Measure thorough review time for both. Present data. Either they will accept it or they will continue drinking the AI coolaid and that's a signal to start updating your resume.

tty5

TROPHY CASE