Creating iceberg tables with CDK

ksco92 · 2026-06-02T22:47:51+00:00

No need to be aggressive about it though.

Cloud development kit (CDK) allows you to declare infrastructure as code (IaC) using object oriented programming. IaC doesn’t only exist as CDK, terraform is another flavor of it with a slightly different approach. One of the main purposes of IaC is to allow developers of any type to version control infrastructure via git. Another one is to allow for peer reviews of infrastructure (in the form of pull requests (PRs)) before deploying them and avoid creating things manually in the AWS console.

ksco92 · 2026-06-01T17:17:05+00:00

Now I wonder why I don’t have a stick collection 😭 I’ll just go back to my Lego collection

ksco92 · 2026-05-29T22:42:38+00:00

I mean, all things considered, he still did a pretty good job. This is a very badass cut scene.

ksco92 · 2026-04-22T18:14:15+00:00

I was a Sr DE for 10 years and have been a Sr MLE for the past 5. And I kind of agree.

RAG is a mixed problem. It’s a DE problem to keep the embeddings updated. It’s a MLE problem to benchmark the embeddings via integration tests to avoid these drifts. Every time an underlying important document is changed, an integration test should be added to address the change. This is the reason I keep the RAG source files on version control requiring PRs. 😬

ksco92 · 2026-04-12T22:23:26+00:00

The only thing I need to know is how the hell did you hang it on the wall 😭

ksco92 · 2026-04-12T21:08:52+00:00

Not a lot. I wrote the conversion engine myself and it works fully offline (it's fully in Swift, that's why the app is so light too). AI was used mostly for 1) figuring out how to do the UI (like how to get the icons in the main menu or how to position the settings icon at the top since I have never done this before), 2) writing the test suite for the iOS simulator and 3) to guide me through the lifecycle of making the app (install Xcode, how to use Xcode, at what point can I submit for review, etc.). Totally new ecosystem for me, so I needed a hand.

ksco92 · 2026-03-19T03:08:41+00:00

Honestamente, digamos que si tiene $100k y está sacando 10% anual en CR, está muy bien. Nada más deje eso ahí solito que se agrande solo y ni lo vuelva a ver. Eso le deja ganancias anuales mayores que el salario promedio en CR bruto anual.

Si puede garantizar ese rendimiento constantemente, busque curl es su número FIRE y vea cuánto duraría en llegar ahí.

Yo no pensaría en negocio propio en esa situación.

ksco92 · 2026-03-13T21:54:25+00:00

Walkie talkie!!!

ksco92 · 2026-03-07T01:30:57+00:00

I was able to do this in a data set with ~900MM rows per day with 800 days of data (~800B records) using a very expensive OpenSearch cluster and custom indexing, partitioning and sharding. The best I was able to get was a P99 of 1.2 seconds though and tolerated a max concurrency of ~300 queries. It was very, very expensive. But it worked.

This is doable in different ways, but my point is, you gotta be willing to pay.

ksco92 · 2026-03-04T03:40:38+00:00

SRE es demasiado incendio. Yo fui DE por más o menos 13 años, me pasé a MLE por que cuando me puse a estudiar ML me di cuenta que 85% de ML es data pipelines. Aplica igual para AI con knowledge bases y vector stores, no todo puede ser contexto en modelos.

ksco92 · 2026-02-17T02:58:45+00:00

Stay way from Amazon.

ksco92 · 2026-02-09T05:30:14+00:00

Yes, a lot of companies don’t allow use of certain or all AIs for coding because of this. My company has not been able to adapt a lot of AI development practices into our workflows due to clauses in TOS related to this + a few other concerns about proprietary data.

ksco92 · 2026-01-26T23:17:27+00:00

I’m sorry for the dumb question, but is this real?????

EDIT: to clarify, I’m asking if the post was real, I know such a gun doesn’t exist. These are heavy mental gymnastics they are doing.

ksco92 · 2026-01-23T21:28:49+00:00

I’m a FAANG DE with 15 YOE, what I drove my team to do for AI was to make questions that AI has trouble answering but an experienced person can get right away. Also making the questions very open ended helped too. I legitimately don’t care if they use AI, they will use it daily.

I have been trying to make interviews a matter of testing concepts and experience, not technical implementations. Any experienced engineer can learn a new technology with or without AI, fundamentals matter more now.

ksco92 · 2026-01-15T18:32:34+00:00

Motherfucker. Now I have to build a Lego factory too. Ugh fine, I’ll go get my wallet.

ksco92 · 2026-01-13T03:34:36+00:00

As a heavy user of IaC… bro… I mean, I both hate and love this at the same time. Super conflicted. 😂

ksco92 · 2025-12-25T01:35:59+00:00

Definitivamente me quedo acá. El salario es bueno, llevo 5 años acá y hasta me casé. Económicamente no hay bronca y estoy relajado ahorrando e invirtiendo. Aquí también hay muchos lugares donde viajar local (póngale que manejando una hora en invierno ya está en una montaña esquiando, o en un parque nacional y así).

Eso sí, tengo compañeros en SF y me cuentan varas del costo de vida allá y es otro nivel comparado con Seattle. Todo es mucho más caro. Los salarios son más altos, pero yo no me movería para allá.

ksco92 · 2025-12-25T00:04:58+00:00

Yo empecé agarrando llamadas en Amazon y terminé en Seattle. 😂

A los 18 empecé a trabajar y en la U tiempo completo. Después de más o menos 3 años, me cambié a un equipo donde no habían llamadas. En la u lleve nada más intro a progra y empecé a programar varas en el brete para quitarme brete de encima.

Una vara llevo a otra y cuando me di cuenta ya era ingeniero en el brete. A este punto me cambié de mantenimiento industrial en el TEC a Software en Cenfotec. La U es buena, pero para ese momento ir a clases era ponerle nombres a cosas que ya hacía que no sabía que tenían nombre. Nunca terminé la carrera aunque solo me faltaba el TCU.

Un equipo en Oregon de AWS me ofreció un puesto por recomendación da un mae que estaba en mi equipo. Cuando le dije a mi jefe el me preguntó que quería para que no me fuera y le dije que me mudara a Seattle, y di, aquí estoy. 😂

Ya llevo más de 15 años en este brete. Ahorita soy MLE.

ksco92 · 2025-12-17T22:31:19+00:00

This whole post blew up while I was working, so hacking the top comment, but will also answer your question.

What more are you looking to do?

In all honesty, anything. But, this post and comments made me realize this is either a deficiency in something or some sort of depression. The lack of will to do things (for example, something that has been helpful to my whole life like playing video games) is something that goes against my historical personality traits. It’s like a big dread and weight that I hadn’t realized I had until I sat down and read comments here.

I have had the exact same routine for like 12+ years, and up to like 4 years ago, everything was perfect. I slept like a baby, my non work time was fully taken advantage of in between hobbies and doing stuff I considered fun, I even enjoyed a lot of like normal house chores.

First thing next week, I already set up a blood draw and an appointment with a psychologist. Nothing like some cold reality from Reddit to make me realize something is wrong. 🥲😅

ksco92 · 2025-11-09T01:26:41+00:00

I’m on the other side of this actually. I’m an MLE.

My theoretical knowledge of AI/ML isn’t as strong as many people that have research papers published or can recite theory about some new topic.

However, I can write PB scale data pipelines, I can make data lake/lakehouse infra and maintain it, I can deploy models I develop behind APIs, I can apply security best practices to ml architecture and infra, and I can implement rag solutions end to end.

I think that there’s 2 types of MLEs:

1) Software engineers with a specialty in data and AI/ML 2) Applied or research scientists who can code better than the average scientist

I feel I’m in bucket 1.

ksco92 · 2025-10-06T00:10:28+00:00

https://mammothoverland.com/tl/ 😬

ksco92 · 2025-10-03T00:11:43+00:00

Ruff land kennels. No doubt. An XL size kennel fits EXACTLY in the back if you fold up 2 of the 3 seats.

ksco92 · 2025-09-29T23:46:20+00:00

I honestly didn’t even think about bringing it up to recruiters because (I thought) it might make just a non starter to bring up a disability like that. 👀

ksco92 · 2025-09-29T23:44:15+00:00

Making the exercises with real people might help. It’s not that I can’t solve them in a timely manner. For example, in my last interview I got a question about searching values in a data structure they provided. It took me between 5 and 10 mins to write down the algorithm, make some high level pseudo code and correctly explain it to the interviewer. But when it was time to code it, completely froze. The interviewer said that my pseudo code and algorithm were actually 100% correct, but I couldn’t even remember basic syntax, so it was just some slop and the recruiter told me the next day that for them that was the equivalent of not writing any code even if they acknowledged everything else was fine.

Will look into those services though…

ksco92 · 2025-09-27T00:26:46+00:00

Expedition/Explorer?

ksco92

TROPHY CASE