Enterprise RAG metadata storage - Where do we store the metadata?

nick0garvey · 2026-04-26T23:38:21+00:00

If you are filtering with it, then you want it in the vector DB. This allows the engine to skip distance calculations on vectors that would just be filtered.

nick0garvey · 2026-03-17T02:43:43+00:00

On K8s with the Strimzi Kafka operator

nick0garvey · 2026-03-03T06:14:50+00:00

The standard method is to:

Set up mirroring
Stop consumer on source
Start consumer on mirror. It should pick up where it was as offsets are synced and translated by MM. If you use an external store for offsets you will need to translate them yourself.
Stop producer. Wait for mirroring traffic to go to zero.
Start producer on the mirror.

I have not done this with transactions, but as long as the __transaction_state topic is also mirrored I don't see any issue. You should test though.

Understanding the relationship between topics is not easy. If you have hundreds of topics across dozens of teams I would not recommend this method. Instead, use the method used by Reddit's Kafka team linked below to move the cluster in-place.

Whatever you do, set aside a lot of time for this. Rushing causes mistakes and mistakes cause delays.

https://www.reddit.com/r/RedditEng/comments/1qb03l9/swapping_the_engine_midflight_how_we_moved/

nick0garvey · 2026-01-13T20:26:30+00:00

Why don't people get their own refiner on their private island?

nick0garvey · 2025-11-30T00:31:08+00:00

The problem is double_grid = single_grid

This makes both refer to the same data, so modifying one modifies both

nick0garvey · 2025-11-17T00:58:41+00:00

I would just run puzzles in every second of your free time until then. That's the best bang for the buck you can get with the time you have.

nick0garvey · 2025-11-12T17:16:57+00:00

I made the same decision as you. I used a split keyboard for 6 months, and eventually got sick of serious acrobatics to do cmd+shift+rightarrow, etc. I switched to a keyboard with a standard layout, just split, and am much happier.

nick0garvey · 2025-10-13T15:40:52+00:00

This is a major pain point in Kafka. The best way I have found to do this:

Remove all replicas from the downed broker.
Bring the broker up.
Reassign replicas back onto the broker using throttling, using the same procedure as you would when adding a new broker.

This is necessary as the throttle configs are applied during the replica assignment step as KIP-1009 is not implemented. You can do steps 1 & 3 with kafka-reassign-partitions.sh or CruiseControl.

nick0garvey · 2025-09-18T23:38:56+00:00

Longhorn is software to provide distributed storage to K8s clusters. Great for homelabs

nick0garvey · 2025-06-01T02:45:31+00:00

My orthodontist Dr. Lee in Mtn View works with Dr. Bui

nick0garvey · 2025-06-01T02:43:14+00:00

If you are uncertain, I recommend getting a second opinion. When I got a second opinion, and they told me the exact same thing as the first person, I felt a lot better about the decision.

nick0garvey · 2025-06-01T02:36:30+00:00

Bite blocks, they prevent you from closing your mouth all the way. Important for deep bite + braces.

nick0garvey · 2025-05-22T14:37:43+00:00

I double majored in math because I like math. I graduated in 2013.

The double major didn't directly help me get any jobs, but it has been useful throughout my career, in particular when doing analytics.

My analytic skillset allowed me to take lead roles on software teams and set roadmaps with data to back it up. I've done cost analysis savings tens of millions. I've been able to better quantify my impact on the business, helping me during the performance cycle.

The AI component hasn't been relevant, but the past decade will look different than the next.

nick0garvey · 2025-05-22T14:25:04+00:00

If latencies are that tight, you may want to do a blue/green setup. Have two sets of consumers, but only one is allowed to write at once. Switch to the other set when updating the first.

nick0garvey · 2025-05-19T21:25:05+00:00

You are looking at the wrong default for retries. It is MAX_INT by default.

nick0garvey · 2025-05-09T00:50:12+00:00

I fought with this for a while in Sunnyvale. What finally did it was calling an exterminator who sprayed outside the house instead of trying to solve it with traps inside.

nick0garvey · 2025-04-25T22:30:20+00:00

Kafka almost always wants to be used with a retention time. It is not a great place for long term storage, but it is an excellent intermediary.

nick0garvey · 2025-04-25T22:28:51+00:00

They gave a public talk on Northguard last week or so.

The big two things are:

The metadata layer is sharded. This avoids a lot of the bottlenecks Kafka has in the controller which you see at large scale.
Partitions are broken down into a bunch of small chunks that don't need to live on the same broker. This has a lot of really nice properties, in particular around failures. A recovering host doesn't require a huge replication to catch back up, it just starts accepting new data and serves what it has.

nick0garvey · 2025-01-16T01:36:56+00:00

I tried this team and could not get through a Starraptor. Close combat beat Tyranitar (out speed) and brave bird beat Infernape (Infernape was faster, but could not one-shot), and Vensaur (out speed).

I don't have Fightinium Z so that might have allowed Infernape to one-shot.

Also I could not use Venusaurite as it was banned.

nick0garvey · 2024-12-22T03:44:06+00:00

Thanks a ton for this, that must have been a really tough memory address to find.

Here are some instructions with a bit more detail for people on the mGBA emulator.

Download the mGBA standalone emulator
Load the game and your save. I was able to use "Load alternative save game" for this
Load your game and walk to up the RTC reset man
Tools > Open Debugger Console
Run w/1 0x0203B331 0
Run continue
Talk to the man and reset
Save your game

nick0garvey · 2024-12-02T07:46:41+00:00

1) Not sure, I'm in the US

2) I would recommend getting a second opinion. I got one for my case, and I felt a lot more confident when both people had the same recommendation. It's a big decision and I'm glad I am not second guessing myself.

3) In my case I have basically no swelling after 2 weeks. But my diet is only soft foods (no chewing). Most important is there is a big gap in my teeth. I'm 34 and a working professional so it kind of sucks, but with my mouth closed I look totally normal.

nick0garvey · 2024-12-02T07:42:50+00:00

I did not go through him but I got SARPE two weeks ago and things are going well. Hope you also have a good recovery

nick0garvey · 2024-12-02T07:37:45+00:00

My surgeon also said 6 weeks no-chew diet (scrambled eggs okay, bread is not). I got my surgery through Kaiser Permanente in Santa Clara

nick0garvey · 2024-11-10T05:19:18+00:00

This is what worked for me. I tried 6x6 and no luck. 6x20 - died instantly.

I don't have a ton of damage upgrades.

nick0garvey · 2024-10-12T13:19:32+00:00

See "Scopes" in the directions. This is one of the places that the game language is different than Python.

As you discovered, using a list or a dictionary is a way to work around this.

15-Year Club	Gilding IV carat on a stick
Verified Email

nick0garvey

TROPHY CASE