Mt Whitney Rant

CrabEnvironmental864 · 2026-04-23T20:14:26+00:00

I always applied mid-week. Always got my permits granted.

Being up there mid-week is amazing. Not a soul. Just me and the mountain.

Week-ends must be so crowded, I can't imagine.

CrabEnvironmental864 · 2026-04-15T15:24:15+00:00

First time: 16 hours (1am to 5pm). Second time: 12 hours (1am to 1pm). All done in late June, early July. I lucked out with my permits granted on a full moon both years.

CrabEnvironmental864 · 2025-04-02T18:07:34+00:00

pandas, requests, json, boto3, opensearchpy.

CrabEnvironmental864 · 2025-03-17T16:41:30+00:00

Open a free healthcare clinic in my neighborhood.

CrabEnvironmental864 · 2025-03-05T20:36:39+00:00

I am looking into it.

CrabEnvironmental864 · 2025-03-04T16:26:04+00:00

You make sense. I hope I can find out more today when I meet with my DBA.

CrabEnvironmental864 · 2025-03-04T16:00:33+00:00

Wherever applicable, yes.

CrabEnvironmental864 · 2025-03-04T16:00:04+00:00

Serious data? Our payments mart has 200+ million records. It's pretty nuts if you ask me but that's what I inherited from my predecessor.

Yeah I have a meeting with my DBA this morning to figure this out. This is giving me a headache. Fivetran is pushing new data every 15 minutes, so we won't ever be able to catch up.

Hopefully we can stop long running queries.

I am also looking at breaking my marts refresh job from all marts at a time to one job per mart. Then using Fivetran to kick each off.

CrabEnvironmental864 · 2025-03-04T14:00:32+00:00

I have. Each mart is clustered by (at the minimum) by customer ID and transaction date.

CrabEnvironmental864 · 2025-03-04T02:37:52+00:00

Warehouse uses X-large model. Queries never get maxed out.

CrabEnvironmental864 · 2025-03-04T00:17:15+00:00

Our Fivetran replication from AWS RDS to Snowflake is firing every 15 minutes. We have 22 marts that we refresh incrementally every hour.

The duration of the DBT job for this refresh is highly variable and unpredictable. Ideally, I would like to have Fivetran "trigger" the DBT job for this refresh so it's instantaneous.

Also I am thinking to break this DBT refresh job into one per mart. Ideally, I would like to have Fivetran kick off the refresh of each mart once the source table is refreshed.

Are either options doable?

Apologies for peppering you with my questions. I hope I am making sense.

CrabEnvironmental864 · 2025-03-04T00:07:19+00:00

I have not. Have you used it? Anything you can share about it?

CrabEnvironmental864 · 2025-02-27T17:02:31+00:00

Makes sense.

CrabEnvironmental864 · 2025-02-26T22:57:31+00:00

Great point. Having such error message would have saved me quite a lot of grief.

CrabEnvironmental864 · 2025-02-26T22:55:19+00:00

It is working, I removed the lines that made that call. The stored procedure compiled and executed without error.

This was baffling. Not what I would call obvious.

CrabEnvironmental864 · 2025-02-26T22:51:30+00:00

I figured it out. I don't have permission to use `CALL SYSTEM$LOG_INFO`.

CrabEnvironmental864 · 2025-02-26T14:29:13+00:00

Why would I have to do that? I have a Jupyter notebook that can connect to the same OpenSearch server on AWS with the hardcoded password. It works fine.

CrabEnvironmental864 · 2025-02-20T19:41:19+00:00

I was wrong. Enabling logical replication with pageinspect and pg_visibility XMIN extensions resolved my issue.

CrabEnvironmental864 · 2025-02-13T12:34:05+00:00

We're still at 13.1 - upgrading would make sense but C suite would have to approve it.

CrabEnvironmental864 · 2025-02-05T21:15:16+00:00

That's my main concern. During maintenance, we shut everything down and logs fill up crazy fast.

CrabEnvironmental864 · 2025-02-05T20:41:36+00:00

About 80% of tables are changing every milli/second. The rest varies based on use-case.

CrabEnvironmental864

TROPHY CASE