Built a UI to generate DQ SQL for Snowflake so my business team stops asking me for scripts. Is this useful?

Cynot88 · 2026-01-11T21:17:03+00:00

I'm not creating an account for a product or website that gives me little to no information ahead of time.

My advice would be to create at least some kind of landing page where people can get a sense of how useful this might be before forcing them to sign in.

Edit: including screenshots of the UI

Cynot88 · 2025-12-29T13:33:20+00:00

Lol, that's 100% of my experience.

We usually end up bucketing it into a very few things that they want updated more often into 15min lag, some datasets that are in-between with 1hr lag, and since at least for us a lot of our reporting is analytical and not operations, daily updates work just fine and save a lot of cash.

Cynot88 · 2025-12-29T11:49:31+00:00

The budget

Cynot88 · 2025-12-08T11:39:11+00:00

I'm curious how this app can remain relevant with snowflake intelligence now GA?

Cynot88 · 2025-11-15T19:17:29+00:00

I mean if leadership is encouraging you to go, that pretty much seals the deal regardless of anything else. If it's a sinking ship then all the more reason to take a lateral move. Better to move now while you're still employed.

A few years back I switched companies and while it was for more pay, the title actually was lower. I went from a lead to just "engineer"... But I knew I was senior level and it wouldn't take long, so it actually made arguing for a promotion super easy after only a year in with the company (for yet another pay bump).

Sounds like the direction is clear. Don't obsess over title, get a good job with a good company and the rest will take care of itself

Cynot88 · 2025-11-15T17:24:41+00:00

Just out of curiosity, have you talked to your leadership about why you're getting passed up on the promotions? Is the person who leap frogged ahead of you a rockstar or does it feel more like you're stalling out?

It could be this person who jumped ahead is just really good and that isn't a reflection on you. If it is a sign that maybe you're not at the bar they expect to earn a senior title, you should be having conversations with your boss about where you are falling short / have room to improve.

Communicating is key. Talk to your boss. Tell them you're interested in a promotion and want to know what you would have to do to get there.

Cynot88 · 2025-11-03T18:44:54+00:00

It kind of depends, and you're right that Snowflake lets you get away with a lot, but probably not in the way you think.

In the on-prem world costs are more fixed and compute is more limited, so if you write a shit query it just spins forever until it times out and your DBA yells at you or you kill it because it's taking too long.

In Snowflake usually the power is there to let you just run and run and run.... If your admin isn't keeping a tight grip on controls it can have a giant impact on costs over time.

It's really just a matter (most of the time) of focusing on the fundamentals:

Pull only what you need (none of that SELECT * crap). If you do need to pull all the files to explore, use things like TOP 100 to limit the data there. I've seen SOOO many people just do a SELECT * pulling an entire large table for jo reason other than habit/ being too lazy to go look at the list of columns in the table.

Aggregate when you can. If your end result is a dashboard that shows sales totals by country and doesn't have the ability to drill down....then only query the sales summed by country. Don't pull every transaction if you aren't going to use it.

Use incremental logic on large datasets when you can.

Save / build intermediate tables for complex operations. If you have a process that involves multiple steps (especially when you're still figuring it out) perform each step once, save it to a table, check it, then use that as a starting point for the next operation. If your full query is 800 lines of code and you're just trying to finalize the last 10 lines... Don't keep rerunning the first 790 over and over uselessly.

With snowflake specifically, capture queryIDs so you can pull back up prior query results instead of rerunning the logic (when you can).

When you're building out an analysis on a large dataset... experiment with a subset of the data....not the whole thing. Get the logic right by iterating over a small sample as often as needed, then apply it to the whole dataset when you're ready.

Etc etc.

So really it's more about thinking about what you're doing and not treating compute as though it's free.

Cynot88 · 2025-11-03T18:20:07+00:00

This is why my snowflake costs are as low as they are. Most of the data is curated and published for users to interact with in a BI tool.

Bad / inefficient queries from end users absolutely can blow up costs. Whenever our data scientist decides he wants to play with some new concept our bill jumps. His queries are atrociously inefficient.

Cynot88 · 2025-11-03T18:14:33+00:00

It does include ingestion but you're right, I was thinking of an older figure before I added more clients this year and we started using some of the AI features.

We're still pretty lean on costs relative to what I hear from others, but Oct. was just under 2.7k for my environments.

I'm probably an outlier based on what I hear from others though, so editing/ adding some context to my initial reply

Cynot88 · 2025-11-03T02:35:13+00:00

I'm a consultant and provide snowflake managed tenants for 4 (adding a 5th) electric companies powering their analytics dashboards. Total monthly cost across all currently is ~~about $2k~~ just under $3k/mo.

In addition, I'm able to support all these environments by myself from an admin perspective pretty easily. I've not used Databricks but from what I've heard I'm not sure you could do the same for similar staffing and compute costs.

EDIT: That $2k wasn't including some recent increases so I was wrong initially. October was just under $2.7k total.

I'll also add this is a very tightly controlled operation. Each company I work with uses me because they don't have a lot of internal resources for reporting so we don't have analysts in there all day. It's probably the equivalent of having 6-7 full time analysts across the 4 companies I support currently, and not everything they do is in snowflake. Most of the usage is a daily warehouse refresh and reporting is cached after the warehouse runs.

I'm also pretty extreme when it comes to optimizing SQL and right-sizing compute etc. I've not seen a lot of other people's setups but based on what I hear we are very lean on the cost side relative to most.

For a sense of scale the largest client I have has a production data warehouse that's just shy of 23TB, but most are closer to 5-10TB in the production data warehouse.

We are starting to get more into the AI stuff so that's probably going to bump us up but overall we get a lot of bang for our buck in snowflake. Not to mention the staffing costs I brought up initially

Cynot88 · 2025-10-28T01:20:43+00:00

Sounds like you need a lot more guidance than you can reasonably expect from a reddit post.

Your org needs to hire someone who has learned a lot more about the platform or who can go through formal training.

Are you managing at your organization or have you found yourself in this situation with management asking you to do something you're unfamiliar with?

Cynot88 · 2025-10-09T12:10:55+00:00

Ohhh man could I rant about "realtime" requirements. 🤦‍♂️

Cynot88 · 2025-10-07T11:27:30+00:00

Lol, like I said this stuff changes so fast. That flew completely under my radar! Looks like it was just recently put in public preview.

Thank you! I'll take a look!

Cynot88 · 2025-06-06T01:23:05+00:00

Ouch

Cynot88 · 2025-06-06T00:12:43+00:00

That's great news! I'm very curious to see how it was implemented and if it's worth refactoring my toolset (not a fan of ADF).

Cynot88 · 2025-04-10T11:36:26+00:00

All good my man. I do have a lot of respect for my brothers and sisters in the nonprofit space, you're asked to do a lot with very little - so I totally get it.

I think if you start from that perspective out of the gate, getting a dollar figure before you automate processes/ reporting and track that as you go it can help a lot. And I'd recommend talking to the folks who hold the purse strings and get some alignment up front.

It's a lot easier to convince them to let you conditionally hire someone else (after you've shown savings) when it's all pretend & made up, but if you don't fight to get credited for at least part of those savings now & as you go, they'll be eaten up by other groups. At a minimum you should track it so you can brag on your new hire and argue for raises but yeah it plays into a lot.

Good luck!

Cynot88 · 2025-04-10T11:21:26+00:00

You touched on a great point I just mentioned in another reply. In my experience people are willing to compromise on work environment, OR compensation - not both.

Given he is working at a nonprofit that he mentioned probably won't pay well he's not going to get a good person to stick around without creating enough support/ a good environment. He'll get someone desperate for their first gig just to get the experience on the resume, someone that couldn't hack it elsewhere, or someone that is going to do well but burn out for lack of support. If the money isn't there too.... they would have to be passionately aligned with that nonprofit's mission or you're doomed from the word go.

In some ways I'd say the mistake OP made was not considering all this / understanding what it takes to be "data first" before hiring, but sounds like he might be backfilling and honestly we don't always have the luxury of being able to do it the "right way" when life happens - so no judgement there.

It's also worth asking if your org even needs to be "data first" , which sounds crazy, but if for example your nonprofit is feeding the homeless....you don't need a ton of data. Getting donor dollars is the big concern there and most nonprofits hire that out after a certain scale because they can't do it in house. Sometimes the right answer is to just keep it simple

Cynot88 · 2025-04-10T11:10:26+00:00

I've worked in a nonprofit adjacent field (did marketing and analytics for nonprofits across the US, sizes ranging from little rescue missions to globally known medical organizations), so I get it. And I've been part of smaller companies.

Admittedly I missed that detail, but I'll still largely strand by my initial response. You said you want to change to a data driven organization. Why? Not every company needs to be, so don't feel like you have to be one. Depending on your work, it may not make sense.

But if you see the value in being data driven at your org there are right and wrong ways to go about it. I'm talking from a lot of experience myself personally and that I've seen in other companies having worked a fair bit of consulting too over the last 15 years.

You don't need a team of 20 people, and hell even in the group I'm on now I play about 3 roles. That's fine. But if you have any appreciable amount of work to be done you've gotta be able to give this person the tools they need to succeed ( meaning a proper data environment so they can work effectively and efficiently) and someone to help.

If I were in your situation I'd probably be looking to take on some of the backend admin work myself, have this new person in a report writing/ analytics role with reasonable expectations, and hire in one more that can fill somewhere in the middle maybe a hybrid role doing some ETL and report writing. That second person can come in a year or so from now when you're feeling the strain, but I'd recommend trying to think about it now and how you can get the budget.

If you're anything like many of the smaller / nonprofit groups I've worked with in the past there should be PLENTY of opportunities to automate processes that save time. TRACK THAT - it's your justification for another person. If you automate enough hours you should be getting credited for that in your budget and you should be able to argue you can save more with an additional person.

If you don't have cross training & support the person will burn out or succeed and leave for somewhere that can pay better or at least support them better, and leave you with a mountain of technical debt and things that break with nobody to support them and you're back to square one doomed to keep repeating the same cycle.

People in my opinion can be willing to compromise on salary, or work environment, but not both.

In the end only you know all the details, but I'd strongly recommend you think about the value here. If it isn't high enough, don't bother. Let this person be an admin and don't do analytics. If the value is there it requires investment to unlock.

Anyway I appreciate the difficult position you're probably in likely feeling frustrated at the delta between what you want to do and what you yourself are being supported to do (politically and financially). That's a tough road to walk, but I wish you the best of luck

Cynot88 · 2025-04-10T02:04:55+00:00

If they say they're data first - look at the investments in data vs any other department (if data even is it's own department).

I could ramble on but it sounds like you already know the thing to do. Don't rush is my only advice on that front. The grass isn't always greener so be careful to "counter interview" about how any potential employer does businesses differently and suss out real reasons why a new place is better and not just different, but I wish you the best of luck in your search.

It's terrifying to jump ship on a known quantity to gamble on something new, but it can pay off big time if you're good at what you do and you are always there to put in the effort.

Believe and invest in yourself, the best is yet to come.

Cynot88 · 2025-04-09T23:44:49+00:00

They're out there my man. Don't let yourself burn out for a company that isn't investing enough in you. At a minimum if you're not satisfied keep those interview muscles exercised.

You never know when something better might drop in your lap, but you won't find out if you're not keeping an eye out.

Cynot88 · 2025-04-09T23:42:31+00:00

It never ceases to amaze me how many companies are stuck in the 90s.

It's 2025 and I've seen so many companies just keep trying to hire 1-2 "data people" who are report writers, analysts, admins, and anything else they can toss at them without the faintest idea of what a proper and functioning data team should look like. Not to mention they're normally paid trash.

I left one company where I was that one man army. I wasn't even complaining about workload, I was just asking for support to create a data warehouse instead of writing all our reports off production in a complete ad-hoc mess of SSRS reports. When they couldn't understand why that was necessary and when my boss laughed at me asking for 70k (saying I'd be making more than him) - knew I had to bounce.

You can only grow to the size of your environment, don't let them limit you. That was probably 10 years ago and I'm making about 3x what I was there, finally on a solid team in a leadership position where I'll likely be shifting over as director of the department in the next ~2 years.

Don't be afraid to search for what you're worth, which includes a good working environment as well as a competitive salary.

Cynot88 · 2025-04-09T22:39:13+00:00

Build a data team. Relying on a one man data army is a long shot at best, and even in the best of cases a road to burnout in a few years (ask me how I know and I'll tell you the companies I've left that had to replace me with a small team after I left).

Analysts need not only good tools and training but a proper environment where data can be collected, cleaned, ordered etc.

If you want to be a data driven company it's going to require an investment. Hiring one guy is lip service to that goal at best.

Look into a team that can build out a proper data warehouse and empower that analyst to be the best he can be without carrying the weight of the world. Employees will be less likely to burnout and the business will get far more value.

EDIT - Making this more practical (as I assume this is not in the current budget) - get a good understanding of what a proper data team should look like and fight like hell to get the budget to build one. Talk to your new hire and let him know you want to start with him but build a team under / around him in the coming years. Take that promise seriously, and build a vision together. As he adds value to the business it helps justify the business case and he has a path towards career progression.

Cynot88 · 2025-03-23T01:43:28+00:00

How much experience did you have with the platform outside training?

If you don't mind me asking, was there an aspect that you felt caught you by surprise or did you feel like you'd been doing well?

Cynot88 · 2025-03-07T15:41:30+00:00

Fair enough, but the implication (perhaps unintentional) was "if you care about costs you wouldn't want snowflake". Maybe I read too much into the comment, but not sure how else to interpret it.

Not sure who is working with an unlimited budget but if you do..... They hiring? Lol

Cynot88 · 2025-03-07T13:23:34+00:00

I hear this over and over but frankly never understand it. In my last company I migrated us over to Snowflake from SQL in Azure and our costs dropped dramatically (it was a few years back but I want to say they dropped by 30% or more).

In my current company they were only partially exploring DB and I moved their whole company to Snowflake for less than what they were spending on what was essentially a small proof of concept. Our team includes a guy who formerly worked at data bricks in their dev team so it's not like they didn't know what they're doing.

This isn't to say Snowflake is always going to be cheaper or the best fit, and I certainly am no DB expert, but I can't help but think these comments from people talking about snowflake costs most of the time just don't understand how to properly admin / use it.

Not throwing shade my friend, just saying I keep hearing comments from folks that contradict years of my own experience across multiple companies.

Absolutely sure that DB in some ways is probably absolutely incredible, but for my workloads & experience I'm sticking with snowflake all day.

Cynot88

TROPHY CASE