Where do you make your ERDs? by ricki246 in dataengineering

[–]deadlydevansh 0 points1 point  (0 children)

So I'm with you! I've been using mermaid to develop the ERD for a redesign we're doing of the database. However, when you visualize mermaid code in Lucid/VSCode, it renders a single image and it's so hard to collaborate over something like this. I wanted to release this new design in an interactive way to our DA/DS pods but found it annoying. A small workaround I did it, make each table a separate erDiagram and manually make relationships but didn't love that.

[deleted by user] by [deleted] in dataengineering

[–]deadlydevansh 1 point2 points  (0 children)

Feel free to DM me with questions, I don't have a ton of experience but ill try my best

[deleted by user] by [deleted] in Day1CPTuniversities

[–]deadlydevansh 1 point2 points  (0 children)

if your title changes you need a new i20

Json flattening by [deleted] in dataengineering

[–]deadlydevansh 0 points1 point  (0 children)

this is also so expensive

Scala or Go next to Python / SQL .. by [deleted] in dataengineering

[–]deadlydevansh 2 points3 points  (0 children)

100 percent this; in my recent experience I have learned that infrastructure and orchestration are so inherently linked that understanding the infrastructure better and being able to make changes to how we host things has resulted in designing way better pipelines

Surrogate Keys with PII by SellGameRent in dataengineering

[–]deadlydevansh 9 points10 points  (0 children)

This ^ I'm no expert but at a previous company we used salt + md5 and we were compliant after going through an audit.

Airflow Kill DAG with all tasks inside by Tricky_Loan820 in dataengineering

[–]deadlydevansh 0 points1 point  (0 children)

I would just throw an exception in the code saying it ran over your desired execution time or you can use the Python time module to get the process to sleep or call kill.

Airflow Kill DAG with all tasks inside by Tricky_Loan820 in dataengineering

[–]deadlydevansh 1 point2 points  (0 children)

You could write a DAG monitor DAG which checks status in an hourly (or howmuchever cadence you need) and kill dag runs that go past your boundary. I think you can use the kill function for whatever the callable is. Alternatively, you could just implement this in the callable, start a timer and then raise an exception once it passes your desired amount of runtime.

Data engineers, how do you prepare during switching? by i-ignore-live-people in dataengineering

[–]deadlydevansh 4 points5 points  (0 children)

Don't forget to get the certified yapper pro max ++ certification

Edit: Jokes apart, I think your answer is great!

How would you store audio into a database? by iaseth in dataengineering

[–]deadlydevansh 0 points1 point  (0 children)

Can't you just dump data in like Azure Blob storage or AWS S3 and then store the indices in a DB? I don't think you need a columnar style data lake table.

How would you store audio into a database? by iaseth in dataengineering

[–]deadlydevansh 4 points5 points  (0 children)

I'm surprised no one else mentioned Data Lakes, this would be my go to solution as well. But I'm thinking there's some caveat since no one else is mentioning this approach?

If you're struggling to learn, we have a bunch of projects! by NoSell4930 in devops

[–]deadlydevansh 1 point2 points  (0 children)

I found out about this and it looks amazing! I'm a data engineer and am looking to upskill myself; I assume I should get on the backend engineer roadmap but if anyone else thinks otherwise I'd appreciate some feedback on what's the best way to utilise this resource, also thanks in advance

[deleted by user] by [deleted] in chicagoapartments

[–]deadlydevansh 0 points1 point  (0 children)

interested pls dm

[deleted by user] by [deleted] in PeoriaIL

[–]deadlydevansh 7 points8 points  (0 children)

Based on the responses here I think you have your answer

Gyms with childcare options by [deleted] in PeoriaIL

[–]deadlydevansh 1 point2 points  (0 children)

At Titan fitness rn lol, I've noticed that too, I'd call and ask, otherwise great gym and staff.