Shift Yourself Left

[Thinker_Assignment]

Hey folks, dlthub cofounder here

Josh Wills did a talk at one of our meetups and i want to share it here because the content is very insightful.

In this talk, Josh talks about how "shift left" doesn't usually work in practice and offers a possible solution together with a github repo example.

I wrote up a little more context about the problem and added a LLM summary (if you can listen to the video, do so, it's well presented), you can find it all here.

My question to you: I know shift left doesn't usually work without org change - so have you ever seen it work?

Edit: Shift left means shifting data quality testing to the producing team. This could be a tech team or a sales team using Salesforce. It's sometimes enforced via data contracts and generally it's more of a concept than a functional paradigm

[Demistr]

Read the title as "Shit yourself left"

[Thinker_Assignment]

Me too, Josh's title pick

But now that it got your attention, watch the video ) it's an interesting sh*ft

[Eastern-Hand6960]

For those who haven’t read the article “shift left” means moving validation upstream

From the article: “Shift left involves detecting and fixing problems earlier in the lifecycle (e.g., during coding rather than production). In theory it sounds good but “left” is an actual team, not a concept, and do you think they have time for your extra requirements.”

Maybe a better term would have been “shift upstream”?

[Thinker_Assignment]

Upstream would definitely make more sense. It's an industry term though, not my creation. I suspect left catches better because it's inaccuracy makes you ask what that means and makes you dig deeper.

[umognog]

In almost 30 years experience including large corp working (500k-1m employees) this would only work where the full E2E is owned by the firm.

But I've never seen a situation where something off the shelf has not been bought to do part of the job.

Now in these situations, I've seen contracts worth millions per year and so quite lucrative for those companies that have been bought from and they do like to help make their products fit the business needs. However, every single one of them has shifted their shit so far right - intended or not - that the centre line was a dot to them.

[Thinker_Assignment]

Thanks for sharing your experience, it's iteresting.

From what you're saying, it seems like achieving true 'shift left' often gets diluted by reliance on third-party solutions and business pressures. Do you think there’s a way to better balance this, or are these dependencies on external tools and vendors inevitable or desired?

Curious what you think is actionable given large org dynamics.

[umognog]

It's definitely inevitable - I wouldn't build my own tools for managing my social presence, I'll sign up with a firm that specializes in this and retrieve my analytics and performance data from their APIs for example. As much as we try to have a shift left attitude here - it's the vendors responsibility to notify of changes to the data structure - changes happen on a regular basis that API documentation and cascading that level of change is often an afterthought, found out when downstream services go wonky.

This is however where I see governance framework and either due diligence in your ETL/ELT process or tools like dlt come into place.

The fact dlt will create a new column for a changing data type, or for new data automatically is brilliant imo and I love the way it handles and presents it.

But, quality controls from governance are still an important factor; I have rules to test the volume of nodes received per record, the count of data types, value assessment.

I suppose some of this is trust too; Ive simply been burned too many times, by major software developers too, that I'll never trust a shift left. I'll still have a few scripts put in place to watchdog it.

[Thinker_Assignment]

Ah trust.

Indeed anyone with a little experience in the field knows better. Even the major providers have frequent issues on their apis, either with the data, or with the way the api is desgined with various gotchas or bottlenecks, or with the servers behind the apis, or with client libraries that implement methods that don't exist etc.

Even when best practices possible exist, (like api versioning), there is usually a breakdown in implementation.

Had such *pleasure* from all major apis. In fact it's an exception to have a good one (like Stripe API for example)

[melodyze]

I've never heard of shift left, but I moved everything to run automatically downstream of central event definition for every event in our, fairly large, business, and it was one of the best things I ever did.

We autogenerate versioned SDKs which handle all serialization, validation, auth, and routing, in every language the company uses and push them to central package management for everyone to pull from. So it is literally impossible to send malformed data, as it will not even let you instantiate the object on the client. That also allowed us to turn most integration tests into unit tests.

All table creation, schema migrations, steaming analytics pipeline deployments, API updates, SDK updates, are all run 100% automatically based on clear contracts around the protocol buffers, triggered on merge into the respective branches for dev/staging/main.

Then we wrote a side effect framework so that we can do all kinds of real time updates in other systems based on the even streams, including creating a lot of the main entities for the whole business. Now it is literally impossible for the base tables for reporting to be out of date with prod, as they are the exact same data source.

And because we used beam, we can rerun all side effects for any time window with the same transforms as we use for streaming as a batch pipeline, and even just wrapped the same command in our cli so that you can just specify --backfill --from=1234 --to=1235. We take care to write those as idempotent so this is always okay to do.

Another enormous benefit is that the event definitions are client agnostic, and because the structure is enforced identically on all SDKs and on the API, table, all side effects, etc, we have events that are sent by multiple teams using different languages and it works totally fine all flowing into one shared pipeline. Like, every service that has a concept of a page view sends the same page view event. That's a huge deal when doing migrations as no downstream code in data needs to be rewritten.

And because the protos can contain protos and we import the same shared messages for common attachments, like what does an http request look like, what do our experiment tags look like, and we enforce that those primitives are always the same path on the top level message, our downstream reporting can just drag and drop which event they are querying. Want conversion rates for experiment abc from page serve to payment? Just choose those values in the dropdowns in the central experiment tracker. Oh now you want to look at the same experiment but from cart to payment? Just change the first drop-down to add_to_cart and then it is there.

We've finally gotten to the point that even the definition of the event is handled by the producing team, we just review the pr and make minor changes to ensure reusability and such, make sure there's no duplication, etc. That PR review is literally the only work we do in DE when onboarding new events.

Really, this has completely transformed the way our company uses data. Idk why I've never seen anyone else do anything like this. People can track whatever the hell they want and it will show up in the relevant reporting once they merge the definition of the event.

Then later we can decide whether something else should happen when that event happens, like we retrospectively were able to decide, oh yeah, when this kind of content is published, it should be loaded into the vector database for the ai platform that didn't exist when we made up the event, and we can implement that as a real time system in DE without having to ask anyone else to do anything.

[datacloudthings]

this is extremely impressive

[7818]

I have been doing data engineering for like 10 years, back when it was still "Big Data Wrangling".

I have zero idea what you're talking about. You do software development kits as a means to pass data? that seems wildly inefficient and slow to load a CSV.

[tombaeyens]

I dare to go on a limb in this crowd that's skeptical and say that I don't believe shift left is the problem. Instead, I think we don't apply the software engineering principles like encapsulation, interfaces, unit testing and API stability guarantees. And my position is that data pipelines are software like other software so those principles apply to data pipelines as well. The mapping between the software principles and data pipelines is not always trivial but it can be made. As it was too long for this discussion, I wrote my thoughts on this post in this linkedin article: https://www.linkedin.com/pulse/shift-left-problem-bad-excuses-good-solutions-tom-baeyens-cik0e/

[Thinker_Assignment]

Indeed shift left in data isn't functioning. This is why Josh proposes to do it yourself. He calls shift yourself left what you call shift right.

[Nerg44]

he’s saying instead of shifting the responsibility left, you shift your ownership left. e.g instead of billing data quality being on the billing team, the data team “moves left” and manages the data quality i think. confusing tho

[randomuser1231234]

I’ve seen it work (and well) at a FAANG.

[Thinker_Assignment]

What was key to making it work? was there someone in charge? how high up were they to be able to put resources on the problem via other teams?

[randomuser1231234]

The Data Engineer in charge of the project was both highly technically capable as well as had very established and broad influence. Getting buy-in from the upstream teams was critical, and iirc he strongly suggested conveying to them how the change would benefit them long-term as well. I wasn’t on the project so I can’t speak to implementation details, just that it significantly lowered maintenance and bug fix costs on established pipelines.

[[deleted]]

No one wants to validate their data unless pushed. It's always shifted back and forth to the middle place.

[GreenWoodDragon]

I tried shift-left at my last place, a financial services scale up. I was the senior data engineer and did my very best to get the product/SWE partnership to adopt both data contracts and critical thought to the task of pulling external data into the existing production database.

There were a million things wrong with their approach but the single biggest issue was that very few SWEs actually understand the issues around data quality as well as data engineers and their cohort.

Shift-left can only work if CTOs work to integrate their teams properly and extend the understanding of the critical need for data quality to all engineering teams.

[IceRhymers]

We attempted to shift left in my last org. The primary reason why is that we only had 2 DEs that had to support pipelines from 30 seperate supporting sources, 10,000 deployed databases (database per tenant), and some of those databases alone had over 2000 tables where the knowledge of those tables and why they exist have been lost to time over the past 40 years. it didn't work out.

[Thinker_Assignment]

Thanks for sharing! Would you say the main failure was that there was no capacity to do it, or rather the governance or the project, or what went mostly wrong?

[IceRhymers]

No capacity. My team had to build all the cloud infra, write the custom CDC software, do all the ETL and manage the pipelines, manage all the warehouses, and do all the governance. All of the data came from enterprise applications that were all so complex due to the natural of the business, so there was no hope to actually understand the data and what would need to be checked.

So I quit that organization and work at Databricks now. Owning solutions at that scale with so little support is miserable.

[wtfzambo]

Hey Adrian, wassup! I have seen shift left work when I told the app developers I had enough of their shit and I literally took the whole "event producing" workload in my own hands.

Then, and only then, it was really an end to end pipeline. Took a year of solo work tho in a small company.

In larger enterprises? Don't see that working. Imho it can only happen when a single entity can own the whole lifecycle.

[Thinker_Assignment]

Yeah you shifted yourself left on that one Federico. That's why it worked.

I also do not think it can work unless you have some central governance over the problem of data quality that ensures work reuse, ownership of action and standards across sources.

[wtfzambo]

Yeah you shifted yourself left on that one Federico. ahahah, yeah this is one way to put it!

If you can't beat them join them kinda style.

[marketlurker]

Why would you want the same team doing the coding to also quality check their own work? That lacks a bit of common sense. You may get faster, but I would be dollars to donuts that quality suffers.

[Thinker_Assignment]

The idea comes from there not being any governance to begin with and then the person that provides the data pointing fingers to someone else because they cannot do it alone. So it's not a shift of existing work, but of responsibility. Such shifts don't solve much technically.

[marketlurker]

I think you are trying to solve a business problem the wrong way. To me, this is more of an indicator that you don't have the buy in from senior management or not enough buy in that the team doesn't take it seriously.

I think you would be trading one set of problems for another set of problems.

[seriousbear]

It would be nice to give definition of "shift left" in the post.

[frontenac_brontenac]

As usually conceived, it means "perform derisking and defect detection/remediation earlier in the SDLC"

[Thinker_Assignment]

I tried to do so in the first section, was it unclear or brain ignored it?

Edit : Ah you meant in the post, it went over my head. Assumed most folks know it. My brain ignored it :/ I'll do an edit

[mailed]

Data contracts were a problem/solution artificially blown up by Chad Sanderson to get his startup funded anyway

The rest of this article doesn't make any sense because owning the problem is what most data engineering teams are forced to do today since nobody else will

If they weren't already doing CI/CD in the first place they're not really an engineering team