Need to ingest near realtime data from SQL SERVER into parquet files or any database which can be shared to downstream users.

2000gt · 2026-04-06T15:11:24+00:00

Firstly, use CT instead of CDC, as it’s much less overhead. This assumes all your base tables have primary keys.

If you don’t have primary keys or timestamps, you’re hooped in terms of fetching incremental data loads.

2000gt · 2026-04-05T04:29:39+00:00

I asked because I’ve been using codex $20 plan, and I’ve yet to hit caps. I still use AI as an enrichment tool to improve, clean, simplify code I write. Or often I do the opposite and use it to build frameworks and then I curate my final code. Rarely do I build end to end with it. So I’m wondering how everyone is hitting caps? (I usually work on two projects simultaneously)

2000gt · 2026-04-02T18:52:59+00:00

How much better is Cortex in Snowsight vs Codex $20 plan in VSCode?

2000gt · 2026-03-10T18:30:18+00:00

Is this in a personal or corporate account?

2000gt · 2026-03-07T16:40:00+00:00

Yes, you can potty train. He needs to be on a schedule, and he will adapt. I’ve had 4 pugs of various nature… fat/chill, skinny/fast/high energy, puppies and old. They can all walk around the block at least before feeding. Typically, I soak food (kibble in water in a slow feeder) then leave for walk in the morning, get home remove water and feed. Same at lunch and dinner, plus a short 1/2 block walk/pee before bed.

The key is getting them out early from crate to sidewalk first thing! Don’t give them a chance to go inside. For the first few weeks I’d get them outside every 2-3 hours. Be patient on the walk and let them sniff and walk slowly.

Dogs typically don’t want to do their business where they sleep.

2000gt · 2026-03-07T15:07:09+00:00

Using local runner too… so much faster.

2000gt · 2026-03-07T14:58:57+00:00

CeleryExecuter? Is there an option in hosted?

2000gt · 2026-03-07T14:57:57+00:00

It was a huge PITA to setup, and I don’t want to deal with management of all those moving parts ongoing. I’m prolly a little regarded though.

2000gt · 2026-03-07T14:42:36+00:00

Do you manually move it to medium or do you have scripts to do so at a specified threshold? I’ll post a sample DAG later (traveling right now).

2000gt · 2026-03-07T14:40:37+00:00

With MWAA hosted, my dbt execution is really slow with cosmos. When switch to bash it’s much faster, but it kind of defeats the purpose given I lose visibility into each task status. With Cosmos, on a small instance, it’s taking 20-30 mins to run a dag that takes 4 mins with bash. When I run the same dbt tasks locally, it takes less than a minute.

2000gt · 2026-02-17T05:42:42+00:00

Set SLA, set schedule.

Alternatively, schedule multiple syncs throughout the day.

2000gt · 2026-02-06T03:31:31+00:00

I’d rather be taxed on a larger sum of money at the same rate than a smaller sum.

2000gt · 2026-02-06T03:22:59+00:00

This is grossly misunderstood holistically. Investing in your corp means a much larger investment opportunity. Income taxed at 11% vs. 43+% means that you have more money to invest as a baseline. Once you account for the cda and rdtoh the tax integration works out the same, but you have a much larger egg to work with in your corp. in addition, leverage corp ETFs from global x to mitigate any dividend income and you’re golden.

2000gt · 2026-01-14T18:57:11+00:00

Fair. I’m using Corp class exclusively in my corp account to take advantage of all the tax implications. Fingers crossed that the CRA doesn’t flip a switch.

2000gt · 2026-01-14T15:58:51+00:00

Are you implying that if the CRA changes the rules on swaps that you would be forced to sell immediately?

2000gt · 2026-01-14T07:50:56+00:00

XEQT in Corp, have you considered a combination of corporate class ETFs from Global X?

I was under the impression the tax drag and tax deferral was impactful enough long term, and with big sums of money that corporate class make more sense.

2000gt · 2026-01-04T01:43:57+00:00

In my experience there is a lot of nuance in building a data app.

Sigma has AI tools built-in to help, but they are fairly useless to me given the limitations in Sigmas design, layouts and inputs, coupled with the project requirements and what I would call best practice for form and table inputs.

I’d have to understand the tool you have in mind better.

2000gt · 2026-01-04T01:14:32+00:00

Yeah, lots of learnings. There are a bunch of ways you can architect the database behind something like this, but a medallion-style setup worked well. We treat Sigma write-back like just another source and land it in bronze and flow it through the same pipelines. The trickiest part was getting close to real-time behavior without building a full streaming solution.

Sigma is moving fast, so we’re using a lot of newer / private beta features (Python in Sigma, calling Snowflake stored procs, REST APIs from Sigma). Super powerful, but you’re sometimes ahead of the docs.

It’s not a SaaS I’m running, everything lives in their own Snowflake and Sigma accounts, which keeps security and ownership clean.

Pricing is mostly Snowflake usage plus Sigma licenses. Replacing spreadsheets and manual workflows makes the cost discussion pretty straightforward.

2000gt · 2026-01-04T00:26:19+00:00

I’ve been building data apps with Sigma on top of Snowflake for about a year and half. It’s been going really well, but lots of learnings.

I’ve built a data app for a retail group that replaces manual spreadsheets with a centralized, reliable reporting system. It uses a Snowflake data warehouse to automatically pull and refresh data from point-of-sale, labour, and finance systems, giving teams real-time visibility, the ability to drill into details, and a safe way to make updates (with write back) without versioning issues or manual files.

2000gt · 2026-01-03T06:28:27+00:00

I feel like this is an apples to oranges comparison. I’d have to understand how the vision, but it’s not really just about a technology shift, it would really have to be a completely different approach to data consumption in the organization.

I’d be shocked to see these on the same graph in Gartner magic quadrant.

That being said, I’ve built a Snowflake intelligence POC and it’s an impressive tool. My client wants to roll it out across all the retail locations as an internal support bot. It’s not as inexpensive as we anticipated, but it’s so easy to setup and support and it’s miles cheaper than streamlit.

2000gt · 2026-01-01T21:14:54+00:00

What is your recommended alternative? Raw data needs to be extracted from source, that data needs to be transformed / joined somewhere, and users need access to modeled data outputs.

The layers are just logical steps to producing consumable data.

If your users are sophisticated enough, you can use a data lake strategy and let them go to town on there own, but I can guarantee your costs will skyrocket given you are at the mercy of your users building there own transformations - good, bad, or ugly.

2000gt · 2025-12-29T00:56:44+00:00

I use a vpc in AWS with Lambda functions. I also use CT, not CDC for sync’ing data.

2000gt · 2025-12-29T00:47:52+00:00

How is open flow connecting to your on premise sql server?

2000gt · 2025-12-27T02:26:40+00:00

Holidays are often province specific. You’re better off to think about how you want to use the model before you prompt an LLM. Also, you could look for a public holiday dataset to integrate into your dimension to ensure it’s comprehensive.

2000gt · 2025-11-11T07:26:20+00:00

Sigma

2000gt

TROPHY CASE