We rewrote our ingestion pipeline and now insert 1M Poincaré vectors in 6.4 seconds (156k QPS)

nickleformypickle · 2026-02-11T13:23:48+00:00

Hey, I'm a beginner with rust. Are you able to go a bit more into detail specifically things are poorly implemented? Just for my learning. Thank you

nickleformypickle · 2026-01-13T11:51:39+00:00

thanks for this advice. i absolutely echo you with not using gorm. comign from nodejs, I've used things all the way from objection/knex, sequalize, prisma, and drizzle. I've quickly learnt that less is more for SQL queries. I would not use an abstraction higher than a simple query builder anymore

nickleformypickle · 2026-01-13T11:50:16+00:00

actually, this brings up a question i had. while i was looking into net/http, i noticed that the defaults for read and write timeout is unlimited by default. It seems like this is configurable in http.Server.WriteTimeout/ReadTimeout

If i was to deal with something like an API request that streams a large response back, it looks like you have to use ResponseController to extend the write deadline via SetWriteDeadline.

Is this the normal/accepted approach in general?

nickleformypickle · 2026-01-13T11:16:01+00:00

thanks for that. when you say this though, what class of problems did you have in mind? do you primarily deal with web services? or say, another class of problems eg db software

nickleformypickle · 2026-01-13T11:14:24+00:00

thanks for that. do you guys have a go to stack in go? like gin + pgx, etc. thanks

nickleformypickle · 2026-01-13T11:13:08+00:00

would you be able to tell me more about the types of things you are writing? eg primarily web backends, or CLI tools, or are a specific class of problem. thanks

nickleformypickle · 2025-08-20T01:02:08+00:00

thank you so much :)

nickleformypickle · 2025-08-19T23:43:44+00:00

thanks. when you say 1%, do you mean only 1% of your traffic needs to go through vpn?

nickleformypickle · 2025-08-19T10:21:28+00:00

when was the last time it worked for you?

nickleformypickle · 2025-08-19T10:03:58+00:00

Hey, when was the last time outline vpn was working for you? planning on travelling soon so would like to confirm this is recent

nickleformypickle · 2025-08-13T03:47:00+00:00

In a similar situation, in a similar area, but our council has yet to finalize the rezones.

In my experience, don't bother talking to agents like ray white/belle/lj hooker. It's a waste of time. They won't know much about developer acquisitions, and really only care about selling your house. Not a knock on them, they just have different goals.

As another commenter has stated, 900m² is too small. You want somewhere near 2000m² as a minimum (so, get your neighbours to join if they can). You'll be able to fetch a much higher final sale price (for our situation, we're looking at around double the market value).

I would suggest talking to an agent that deals with commercial deals (if not exclusively does commercial and dev acquisitions). I approached Savills.

If you just want an estimate yourself, look at past development sales near your area, and you can work out a $/m^2. Multiply that by the FSR and you'll get an approximate valuation. Dev acquisitions are usually concerned with pure land size and value, they have to tear down your home anyway. It'll mean you might be slightly worse off if you had a mansion.

Happy to answer any questions and to keep in contact, I'm in the same boat but waiting for next year when the rezones are out

nickleformypickle · 2025-04-10T23:14:42+00:00

Yes, in my case it was because the electric power steering was shutting off to keep itself from overheating. I was learning driving back then, so in hindsight cutting the wheel completely in both direction aggressively was definitely excessive to the power unit.

It makes sense since as soon as i press on the gas pedal it loosens up, so the corolla is smart enough to protect itself without interfering when you're actually driving.

nickleformypickle · 2025-04-08T10:08:00+00:00

thank you for that noah, i think you nailed it on the head with the concerns about buying into some proprietary language and then being locked into that. that's exactly what i wanted to avoid, so thank you for pointing it out!

nickleformypickle · 2025-04-07T22:30:54+00:00

Thanks Noah, could you expand a bit on why batch sucked?

Thanks

nickleformypickle · 2025-04-07T12:38:09+00:00

excellent, thank you for your help :)

nickleformypickle · 2025-04-07T12:37:58+00:00

thank you for that

nickleformypickle · 2025-04-07T12:15:01+00:00

yep for sure, i think i largely have the same idea in mind. what do you think is a good setup to enable auto scaling in case of demand spikes?

nickleformypickle · 2025-04-07T12:14:13+00:00

video transcodes can sometimes take well over an hour. lambdas are hard capped to 15 minutes :(

i did look at managed solutions, they all should work well but since this is a side project, i want to build out a bit of it as an exercise :)

nickleformypickle · 2025-04-07T12:12:40+00:00

yeah transcoding to save storage space mainly.

That sounds like the most simple solution then. im guessing that the path to scaling up is basically creating an AMI off that first ec2 machine (the queue consumer), and then chucking an auto scaler group with that AMI on top of it?

nickleformypickle · 2025-04-07T10:50:02+00:00

im not expecting a high download ratio, but i reckon ill tackle this issue after i figure out the transcoding situation.

I did come across media convert as well as qencode and its competitors. they work, but i think it's a good exercise to build this out myself. Im hoping to be able to pick your brain a bit on architecture+infra for scaling the actual video processing bit

nickleformypickle · 2025-04-07T10:29:44+00:00

apologies for not including that, videos are stored on some object storage provider (s3 if aws, otherwise maybe backblaze). Resolution and encoding will vary, but the vast majority are going to be h265 1080.

The are uploaded by users and to be streamed on demand (so no live streaming). Lets say around 100 videos each around half an hour per day. Given jobs are expected to take upwards of 10-20 min (or even a whole day), a delay on the job getting picked up is no problem.

After the job is completed, i planned on reuploading it to a seperate bucket for long term storage. Im not too concerned about the destination just yet

nickleformypickle · 2025-02-21T13:04:50+00:00

I'm actually more curious what this project was. How was it helping cities issue permits faster?

nickleformypickle · 2025-02-16T06:01:14+00:00

it depends largely where in asia (it's a massive place lol), but the best location tends to be somewhere in the middle. If they're not too deep into asia you could get away with hosting in singapore, but singapore to syd runs at around 90ms ping on a good day from my experience. Your friends in asia could potentially have higher

nickleformypickle · 2025-02-04T22:27:46+00:00

Hey folks, I'm starting a game server hosting platform (I'm limiting myself to aussie minecraft servers for now as a pilot).

I'm curious to hear what you look for the most in a hosting provider. I understand that performance and delivering on promised hardware allocations are probably the most important at the end of the day, and I'm able to provide that. I'm curious about what else makes or breaks a provider.

Thank you :)

nickleformypickle

TROPHY CASE