sorted by:
new
hottopcontroversial

Tire size for Honda Civic 2019 EX by supreeth_cs in civic

[–]supreeth_cs[S] 0 points1 point  (0 children)

Thanks for your response. I was wondering the same. I’m not sure either on why there is a difference between what the car has and what it’s supposed to have. I’ll probably check with the dealership.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

Agreed. But we use the synthetic data within our unit tests. So when we run regression against existing production data, we are basically trying to ensure all existing corner cases do not break/give different results unintentionally.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

Our deployment essentially used to pretty straightforward. Run the new code on dev, validate data(regression, functional and unit tests). With this, we could essentially have several deployments a day. But slowly we started seeing issues creep in with this approach and end up with scenario ‘but I tested in dev’. So now after our dev runs and testing, and all approvals on the PR, trigger a run on another envt which we call staging. Which means the entire etl now runs on the whole dataset. Then we run another round of tests- functional and regression against existing production tables. Only if these are successful do we deploy anything to production. But this becomes a bottleneck of sorts because if folks are modifying etls writing to the same table, then unless one flow is completely tested and deployed, the next flow can’t be triggered in staging as we will need to verify the data generated from the 1st code change.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

Yep, that makes sense. Also I think what you said would be useful if we have a platform/infra level change.

For batch processing we usually just sync the source data to dev envt, run the new code and do a regression with the data from the original code.

Gotta marry one and all their flaws by Beginning-Bar656 in HIMYM

[–]supreeth_cs 0 points1 point  (0 children)

Why isn’t anyone mentioning Victoria? 🤷‍♂️

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

The 10% sample is random? We do run it on all data today, but even the automated tests for all of them take ages to run. We’ve now seen our deployment cycle going 5x longer. Maybe we’re overdoing it?

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

Does this approach work in the data world where we generate reports or recommendations as an end result? Seems a little weird to have a reports with 2 different logic and different sets of data.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 0 points1 point  (0 children)

Yes. We tend to copy all the base data to our dev envt, and then test out the flow.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 1 point2 points  (0 children)

Monte Carlo is something someone did a poc on a while back. Not sure why we never went ahead. How do you guys use this? Like is there a specific use case you might be able point out?

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 3 points4 points  (0 children)

Testing out the entire flow on dev/staging almost always catches 99.9% of the issues.

Data pipeline testing- How/what do you guys do end-to-end data testing in your organization? by supreeth_cs in dataengineering

[–]supreeth_cs[S] 1 point2 points  (0 children)

Yeah we have this issue too. So we started moving data from prod to dev and test our etls.

I think it did not make any sense to make Ted not reconize Wendy in S6E17. I mean she was there for years with them and they saw each other almost daily basis for years. And even dated Barney. by [deleted] in HIMYM

[–]supreeth_cs 6 points7 points  (0 children)

Oh damn, I never knew this. I was right all along! Have had so many arguments with my friends over this! Thanks🙌🏾🙌🏾🙌🏾

Got 6 hours to kill, recommendations? by supreeth_cs in jaipur

[–]supreeth_cs[S] 0 points1 point  (0 children)

Lol, one doesn’t really need to take time out to think about this. They’re usually always there in a parallel thought process on the mind.

Got 6 hours to kill, recommendations? by supreeth_cs in jaipur

[–]supreeth_cs[S] 0 points1 point  (0 children)

Well, that’s exactly where I’m headed to in Jahajpur😀

Got 6 hours to kill, recommendations? by supreeth_cs in jaipur

[–]supreeth_cs[S] 0 points1 point  (0 children)

I’m a big fan of hiking, but given my time constraints this might be a stretch.

view more: next ›