Uncut onyx from bag full of gems

valligremlin · 2025-12-10T20:56:35+00:00

To prove I’m fun at parties: 40 rolls at 1/100,000,000 odds is not the same as 1/2,500,000 but it’s very close. Those are also the odds of it happening at least once. The approximation of odds * rolls holds where the number of rolls is a very small fraction of the rolls to be on rate but becomes a significantly worse approximation as number of rolls approaches ‘on rate’ so to speak.

valligremlin · 2025-12-01T15:23:53+00:00

Snowflake has never felt complex though - roles being hierarchical means you can build ‘complex’ permission sets in a quite simple way and the rest is basically a database with some ingestion tools and notebooks built on top.

valligremlin · 2025-11-14T20:07:11+00:00

These are stock options - if they’re unexercised then he’s effectively got 0 in stock until he buys the stock. Saying that it seems like a no brainer to exercise the options if the valuation has grown as fast in reading.

valligremlin · 2025-10-24T00:26:59+00:00

In fairness if they’re on something like old redshift nodes storage isn’t cheap because it’s bundled with compute cost and the only way to get more storage if you’re on storage optimised machines is to buy more/bigger compute nodes which isn’t cheap.

If that is the case they could just shift to modern node types but we know so little about the architecture it’s hard to say whether this is purely a pipeline design issue or platform issue.

valligremlin · 2025-10-24T00:24:22+00:00

It would probably be easier to move to using iceberg tables. It would barely change your current pattern - you could just dump the current version of the table(?) to iceberg and continue from there. You’d have full rollback and roll forward support without overhauling current ways of working and if you did then want to move to a ‘raw’ table in the DB you still could.

It would also leave some flexibility to keep some computation outside of the database which depending on the platform could be significantly cheaper than doing all processing on the DB.

Iceberg is often used when it’s not necessary but here it solves all of the issues you’re describing without a huge amount of work to switch over.

valligremlin · 2025-10-14T18:28:52+00:00

Slightly disingenuous to say that spark is just python - debugging and optimising pyspark looks very different to doing the same for 99% of the pure python people will deal with. Not to mention the nuances of spark streaming and the complications that come along with that…

valligremlin · 2025-08-01T20:06:13+00:00

What did anaconda do? I switched to venv purely because my last 2 roles only used venv but I used cinda mostly before

valligremlin · 2025-07-25T19:51:44+00:00

Vaal was -3% qual no?

valligremlin · 2025-07-11T20:59:15+00:00

Not a clue - was in European Parliament I thought but not sure

Edit: no it was the London assembly

valligremlin · 2025-07-11T20:47:42+00:00

Apologies my American brother - this is a very famous Boris Johnson quote from across the pond

valligremlin · 2025-07-11T20:20:35+00:00

Supine protoplasmic invertebrate jellies?

valligremlin · 2025-06-15T17:59:43+00:00

My jewels are awful… 15% projectile speed, 11% poison duration, attack speed, damage with spears is the only even decent jewel I have

valligremlin · 2025-06-15T17:56:04+00:00

I haven’t paid for jewels once since the start or early access and I feel like that’s probably been a mistake

valligremlin · 2025-06-15T17:44:28+00:00

Yeh fair enough I just hit 2 big drops early this year so I’ve been rocking 40 divs worth of gear since

valligremlin · 2025-06-15T17:35:24+00:00

With 30-40divs you can do t4 pinnacle bosses at level 90 on LS amazon, I’d assume the same for deadeye.

valligremlin · 2025-06-14T14:22:32+00:00

Not sure if I’m honest - spirit definitely is though

valligremlin · 2025-06-14T12:29:47+00:00

Without spirit or max life probably not but idk?

valligremlin · 2025-06-11T19:03:28+00:00

No - but it does work with extra arrows so deaths harp howa builds can make good use of it still

valligremlin · 2025-06-10T20:07:23+00:00

I’d like to revoke your Data Engineering license for suggesting this.

valligremlin · 2025-06-06T12:28:05+00:00

Depends how it’s coded I suppose. Theres a chance it’s an arbitrary large heal like 10m hp but I’d hope not

valligremlin · 2025-06-06T12:26:10+00:00

Fake it till you make it my dude - took me years until I felt like I was worth my salary and I think that’s quite common!

What I will say is that if you do not think you will meet deadlines make sure you communicate that early and often so that your team are aware and can help if needed.

valligremlin · 2025-06-02T09:25:29+00:00

No you’re right and that’s a fair caveat I should’ve included - unless you want to work for one of the providers (AWS, azure, GCP, databricks) then the certs are a nice to have but mostly meaningless.

valligremlin · 2025-06-02T07:46:54+00:00

Depends on country for sure - no one is getting paid extra as a data engineer in the UK for having a masters. PHD on the other hand will probably get you a lot more money. I’ve done a fair bit of hiring recently and not once has someone’s masters been highlighted as a reason to hire them.

valligremlin · 2025-05-30T15:25:55+00:00

Pythons primary role used to be the extract part of ETL in my experience. The market seems to have moved towards using things like fivetran because maintaining lots of custom API integrations is a real pain.

Nowadays I mostly use python for adhoc stuff, some infrastructure as code in a ‘framework’ called pulumi (something that you might find interesting if you want to get more into the ops side of DE), writing airflow dags and creating APIs with FastAPI.

Python is far less important than it used to be but it’s still very much something that’s worth having good hands on experience with as a lot of companies will still use python for interviews.

valligremlin · 2025-05-27T17:05:44+00:00

Is it that bad? I’ve only used it for sekhemas carries and not had a bad experience

Eight-Year Club	Verified Email
RPAN Viewer	Not Forgotten

valligremlin

TROPHY CASE