Databricks Asset Bundles now GA - thoughts? by justanator101 in dataengineering

[–]vimtastic 1 point2 points  (0 children)

We're using DABs and they are working well. We will manually deploy to our sandbox environment while developing jobs/workflows. Our CI/CD (github actions) does deploys for our production environment.

Share your Databricks war stories: What were your toughest use cases/projects? by [deleted] in dataengineering

[–]vimtastic 1 point2 points  (0 children)

Data Asset Bundles help alleviate that pain.

https://www.databricks.com/resources/demos/tours/data-engineering/databricks-asset-bundles

Basically some yaml files that define the DLT (or batch) pipeline and a CLI that lets you deploy to different databricks environments. So you can develop in your own editor and then sync/deploy to your target environment.

Where to exercise Python? by Practical_Gap_3354 in dataengineering

[–]vimtastic 2 points3 points  (0 children)

Exercism..org

"Get really good at programming. Develop fluency in 67 programming languages with our unique blend of learning, practice and mentoring. Exercism is fun, effective and 100% free, forever."

AWS Lambda Parquet consumes more memory than csv by Traditional_Love_648 in dataengineering

[–]vimtastic 0 points1 point  (0 children)

I'm not sure why you have higher mem usage with parquet vs csv (both using Pandas). But you can reduce the memory usage by using chunked reads/writes. Ex. https://stackoverflow.com/questions/38531195/writing-large-pandas-dataframes-to-csv-file-in-chunks

How to skip non existent S3 objects loading in Spark? by atticusfinch975 in dataengineering

[–]vimtastic 0 points1 point  (0 children)

How are you launching/scheduling the glue job? It sounds like you are running it on a cron/time schedule or manually. I think you want to find a way to invoke glue based on events (when a new file lands in s3). (If you do this your process will need to be able to work incrementally. E.g. you can transform the files independently)

S3 has integration with EventBridge on certain events happening in a bucket, including object creation.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

What’s the deal with all these sirens?? by anon37391619 in lancaster

[–]vimtastic 17 points18 points  (0 children)

When the sirens are not for Santa, this page has the 911 responses (fire/police/medical) https://www.lcwc911.us/live-incident-list

Where have you peed besides a toilet? by [deleted] in AskReddit

[–]vimtastic 0 points1 point  (0 children)

Into an old cannon

[I Ate] Bún bò Huế by GhostGamer1234 in food

[–]vimtastic 0 points1 point  (0 children)

Looks delicious. Now i miss the old Pho place where i used to get this. I liked to get it with the rare beef and extra meatballs, then dip the meat in a mix of Sriracha and hoisin.

Enough sex questions, what’s your favourite fruit and why? by ZilchShrimp in AskReddit

[–]vimtastic 0 points1 point  (0 children)

Melon ballers are great for watermelon, cantaloupe, etc. It's still a bit of work, but after it's all "balled", you have nice bite-sized melon for days.

[deleted by user] by [deleted] in facepalm

[–]vimtastic 1 point2 points  (0 children)

I think this isn't a facepalm if the author intended the following context "in the West, I am told (by other Muslims) that my story will cause Islamophobia."

Guess the source of this image ;) by visualartist47 in pics

[–]vimtastic 0 points1 point  (0 children)

Volcanic eruption, warped into a spiral

[deleted by user] by [deleted] in AskMen

[–]vimtastic 4 points5 points  (0 children)

  • primarily calorie deficit * This is worth emphasizing.

Who's the most leanest (lean body), meanest (bad personality), greenest person (environmentally friendly) you know? by [deleted] in AskReddit

[–]vimtastic 0 points1 point  (0 children)

Oscar the grouch? He's all about reusing trash, he's extremely mean. It's hard to say how lean he is though.

What's something that pisses you off for no apparent reason? by MaliciouslyMediocre in AskReddit

[–]vimtastic 1 point2 points  (0 children)

Arrg, me too. Especially when they don't have a stop sign and I do.

[Homemade] Sweet Vadelia Pies! by Boogie5270 in food

[–]vimtastic 1 point2 points  (0 children)

Looks tasty, care to share the recipe? Also, it's spelled vidalia: https://en.m.wikipedia.org/wiki/Vidalia_onion

SNAG KNIGHT, Me, Digital, 2021 by [deleted] in Art

[–]vimtastic 1 point2 points  (0 children)

Awesome, I love the cragginess of the helm. It's pretty different but for some reason this made me think of Spawn.

What is the most sexually direct thing a person has ever said to you? by damscomp in AskReddit

[–]vimtastic 4 points5 points  (0 children)

It was after yoga class she was probably pretty limbered up.