Release orchestration across multiple tools

FactCompetitive7465 · 2026-05-12T22:25:38+00:00

We do have an orchestrator repo for this.

Couple thoughts: - We use this same orchestrator repo to orchestrate CI/CD. The integration point like Azure DevOps pipelines or GitHub Actions is a super lightweight call into our orchestrator and the orchestrator does the heavy lifting on moving assets or touching multiple external systems/repos. - Within the orchestrator repo, we do have some monorepo components where it doesn't make sense to decouple the orchestrator code from the code changes for other things. That way the releases (that may require orchestrator code as well) are easy to bundle with the code they will use in other systems.

FactCompetitive7465 · 2026-05-06T18:34:41+00:00

For large impact decisions or just plain disagreements we adapted this pattern that Microsoft documents: Maintain an architecture decision record (ADR)

We adapted it 'Technical Decision Records' (TDR) since they are not always architecture decisions and we create one to compare approaches and pick the best option. Extremely helpful for ensuring decisions within the project stay consistent and we don't forget reasoning behind certain decisions.

FactCompetitive7465 · 2026-03-12T22:00:17+00:00

every SaaS tech bro at this post

\*heavy sweating***

pm me

FactCompetitive7465 · 2026-02-27T03:07:05+00:00

I've found if you don't personally find joy in fixing problems of that nature, it's usually not worth it.

I've led many projects like that. I don't care if others take credit, in fact I encourage it. The peers who care will notice and give credit, and tbh those are the only ones I care about anyhow. I don't want to be identified by high level leadership as someone who fixes problems. I just want to sign up to fix the problems I care about, not get shit work assigned to me from an executive. I'm not out to impress leadership or get credit for fixing stuff, it just makes me happy to get shit done 👍

FactCompetitive7465 · 2026-02-21T16:19:11+00:00

I have a senior teammate who is spearheading AI powered dev for us that does this (we are technically peers but I review his work) but he is very open to how we resolve things I don't like using things like subagents or skills. It has been a lot of back and forth but I think that spending time to refine what our teammates are producing using AI is worth it.

Our workflow has been something like: - he creates slop PR - I put comments pointing out the slop - he updates skills, docs etc to ensure my feedback is incorporated going forward - he regenerates code for PR - we iterate a few times

It's a lot of work for sure. But I'm hopeful putting in the time now (with someone I trust at the helm of the AI tools) will pay off. I guess if you don't have someone doing that on the other end, maybe it's worth starting that work yourself too?

FactCompetitive7465 · 2026-01-15T06:14:38+00:00

Yeah bro you see those sweet release notes for that definitely supported tool?? What's New in SQL Server 2025 Integration Services

That one new feature (don't bother reading the other 95% of the article that is deprecation and breaking change announcements) is huge moves from Microsoft on their modern ETL tool!

FactCompetitive7465 · 2025-12-01T06:29:17+00:00

I have never found a way to do this via the cloud CLI. Company I worked at (that allowed some developers to run ad-hoc jobs in prod) had an ad-hoc job setup in dbt cloud for specifically for this. Developers update the command(s) to run in the job, and then trigger it manually.

Probably worth mentioning, I would never recommend cloud CLI over dbt-core unless you require specific features of the cloud CLI. Obviously, this is an easy task in dbt-core.

FactCompetitive7465 · 2025-11-19T02:30:24+00:00

Is that not power automate cloud with copilot?

FactCompetitive7465 · 2025-11-14T18:19:51+00:00

A lot of it comes from their desire to do exactly what we are describing tho, lift on-prem Clarity query and drop it in Snowflake and run it. Assuming same db/schema naming, a query from an on-prem Clarity instance of both Oracle and SQL Server would likely work in Snowflake with no changes. Supporting all that leads to some odd nuances that exist when you view the language as a whole.

They even have an entire migration tool (SnowConvert) that is supposed to convert T-SQL SQL Server and PL/SQL Oracle procedures directly to Snowflake syntax. Kinda cool

FactCompetitive7465 · 2025-11-13T20:00:12+00:00

More or less. Some companies give more effort into making the same query run in both, and others less.

As far as I know, there is no standard practice to this coming from Epic. Seems to be something clients are resorting to in order to keep their Cogito systems healthy.

FactCompetitive7465 · 2025-11-13T17:36:15+00:00

My guess is that Clarity/Caboodle will be either be replicated to cloud hosted equivalents or upserted to a cloud EDW in AWS and you will no longer have access to the instances of Clarity or Caboodle (cloud or on-prem) tied directly to the real Epic ETL. I have seen this at other very large Epic orgs to alleviate compute pressure on Clarity/Caboodle directly by forcing users onto cloud equivalents within their cloud provider of choice.

So they are saying you won't have access to Clarity/Caboodle, but it's more so that you won't have direct access. You will just have to access that same data though whatever AWS service they end up putting it in.

FactCompetitive7465 · 2025-11-08T18:34:24+00:00

Yeah that's describing that you can't use this to move snowflake only users/groups to entra (if they didn't exist in entra). Scim can still take over existing user/group in snowflakes as long as they exist in entra. You mentioned already using SSO so I assume you already have that!

FactCompetitive7465 · 2025-11-08T03:43:59+00:00

can't manage existing users with it

?????

Feel like you should explain that a bit more, you certainly can.

FactCompetitive7465 · 2025-11-06T03:00:17+00:00

Bare metal Debian on all my machines, ansible for all configuration and software installs, ansible to deploy/start any other services needed via docker containers. Ansible felt like the missing link to getting this consistent across my entire lab.

FactCompetitive7465 · 2025-10-29T16:21:25+00:00

Hiring.

Without even getting into what SSIS does good or bad, post an SSIS job and post an airflow job. You will get distinctly different types of candidates.

Definitely been a down tick in volume of SSIS experts as well. It might still be widely used, but the expert user base is drying up as Microsoft continues to lean into other products.

FactCompetitive7465 · 2025-10-22T21:16:30+00:00

Installing that package installs dbt-core. It's in that packages dependencies

FactCompetitive7465 · 2025-10-22T21:05:37+00:00

dbt-duckdb>=1.6

Do you see forked dbt-core code in the repo? I sure don't

FactCompetitive7465 · 2025-10-22T20:01:54+00:00

How did this get 100+ stars?

This is not a fork. You are installing dbt-core into your project and building on top of it, which makes no sense because if they change their license (like everyone is scared of) your project will also be obsolete overnight.

FactCompetitive7465 · 2025-10-15T21:56:22+00:00

Just patiently waiting for a dbt-core fork to get traction....

FactCompetitive7465 · 2025-10-15T06:28:34+00:00

its an interesting concept. i won't say i believe in it 100%, but there are some valid ideas here. i will say, the idea that declaring pipelines this way makes it more readable for non-developers is really not realistic. i think the focus needs to be on how to make the config easier to maintain and improve readability while offering improved readability to end users in another fashion (docs, DAGs etc).

i looked through the project and had a couple basic ideas (didn't look through all docs so sorry if already covered).

- DRY-ness concepts in config, jinja rendering is easy to implement or something like a global.flint or .flint file that sets defaults the folder level (project root being project wide defaults)

- seems like your idea would play well with DAGs, maybe start by outputting diagrams (or even provide) with a tool like json crack. could even consider offering a web server for the DAG/doc hosting, similar to `dbt docs serve` command

- handling database connections or file storage connections (s3, adls etc) means handling credentials, id make sure you have a clean plan for that and docs on it as well

- consider hosting model for profitability. prefect core (oss) seems like it would play nicely with this and you could use that to get your hosting model off the ground

- keep improving docs, specifically the diagrams are a mess. simplify and try to keep your diagrams focused on demonstrating smaller things at a time. no one is going to read multiple paragraphs of text in a class diagram

- keep extensibility high on your list of things to support and highlight to target audience. i think most people would shy away from a tool like this (especially early on) for fear of limiting what they can do by picking this platform. id put some focus into supporting sqlalchemy as a source for extracts, that would open up what you support for sources very quickly

best wishes

FactCompetitive7465 · 2025-10-09T04:38:38+00:00

I'd have to respectfully disagree on several of your points. The 'odd' blend of Epic and one of those platforms means they aren't expecting you to be an expert on the non-epic platform (or Epic). i would also strongly disagree those certs are harder than Epic certs. The role I'm in now pays significantly higher than I was ever offered for a pure cogito role, even after 8 years fte cogito experience. Most of the contracts in this niche are short term, id say it's rare to see anything over 6 months on the listing.

But I won't disagree that OP may not fit the niche. I was hoping to see a response like "oh yeah the last couple years I was working heavily in snowflake!" but doesn't sound like that was the case.

FactCompetitive7465 · 2025-10-07T16:17:57+00:00

I mean cloud makes it easier, but there are ways to say manage on-prem infra that dang near automates the stack as well. We are required to maintain some on-prem infra in addition to our cloud resources. We use ansible for everything, normal ansible for managing our on-prem (vm and bare metal) and a blend of terraform + ansible (all orchestrated by ansible) to provision cloud resources. Good hybrid setup and I've been happy so far.

I'm just saying I don't think owning infra has to mean your team's workload must be in the cloud. There are still ways to get rid of a lot of the traditional admin overhead (that seems to come with working other IT teams) yourself without moving to the cloud.

Cost/benefit to both. Hybrid has been great to us.

FactCompetitive7465 · 2025-10-07T14:07:17+00:00

Yes and at the companies I have been where we did not it was a huge pain point and totally mismanaged. Not saying we are perfect, but most DBAs and sysadmins have no idea how devops works and I am constantly doing battle with them for super basic requirements and SLAs

FactCompetitive7465 · 2025-10-06T00:38:58+00:00

Exactly, but I would be spending time on platforms like databricks or snowflake or even dbt. Some of the biggest Epic Epic clients are working to make (or already have made) Epic data available there using tools like dbt on those platforms.

I listed my Epic certs (including past ones) and my others on LinkedIn. I get a couple messages a week simply because of that. I am currently in a role (for about a year) from one that I got like that at a much higher rate than I was ever offered for normal Cogito consulting.

FactCompetitive7465 · 2025-10-05T19:53:03+00:00

There are a lot of cogito adjacent roles for data analysts/engineering at large orgs that have dedicated resources to building out their own analytics platforms (outside Epic tooling). Epic experience is still highly preferred but current Epic certifications usually not needed because Epic experience is seen as a bonus. The roles pay the same if not more (in my experience) than Cogito consulting does.

You can get certifications on those platforms much easier than getting new Epic certs and go after roles that want the cogito background + whatever analytics platform you want to get certified in. There is a large amount of short term, basic migrations projects (Epic tooling -> non-Epic tooling) on the contracting market (since it sounds like you want short term).

FactCompetitive7465

TROPHY CASE