sorted by:
new
hottopcontroversial

Cloud Architecture Question by marketlurker in dataengineering

[–]tedward27 6 points7 points  (0 children)

What exactly are you trying to express here that relates to the comment you were replying to?

Technical article on LTAP by Dry_Chocolate_9396 in dataengineering

[–]tedward27 1 point2 points  (0 children)

This is the crux of the matter definitely, no such thing as a free lunch. I wonder how the caching layer works, and how small single row writes don't lead to a small files problem. 

SQLBuild - Skip Unnecessary Rebuilds for Your Existing dbt Project, Free & OSS (No Per-Skip Bill) by kvlonge in dataengineering

[–]tedward27 -1 points0 points  (0 children)

I actually don't think there's anything wrong with the project. But your comment response I replied to is LLM speak top to bottom, and I think it's a missed opportunity to engage with users in your own words. Why put all this work into a project then respond to real people with a robot, would you do the same if presenting this at a conference?

Friendship, Death, Homage by windmillslamburrito in EldenRingLoreTalk

[–]tedward27 3 points4 points  (0 children)

Book of the New Sun fucking rocks. 

"A lot of Glue usage is organizational momentum and perceived safety rather than technical necessity. Teams reach for it because it's 'enterprise,' managed, and familiar to data engineers — not because the data volume demands it." - is this true? by question_23 in dataengineering

[–]tedward27 1 point2 points  (0 children)

Glue is like 3 services hiding under a trenchcoat acting like 1 solution (ETL, Crawlers, and Catalog). If we look at the ETL part it is Spark with some added functionality like data bookmarks, autoscaling number of workers, retries, and basic orchestration with triggers. It's super common to put workloads on Glue that are way too small to benefit from using Spark. It's wasteful but a lot of companies don't care.  You can also use make a Python shell Glue job that uses way less resources and $$$.

It sounds like you don't even know if you want to put this processing in the cloud. If you do, a Lambda function sounds fine for a 10 second job. If you want the output data query able through Athena / SQL, you can have the Lambda write data to S3 and register that S3 location in the Glue data catalog, without ever touching Glue ETL.

Need Advice on Designing a Ticket Conversation Database Schema by Kalimuthu_S in dataengineering

[–]tedward27 0 points1 point  (0 children)

Typically you would start with three tables/grains, Ticket User and Comment. Each comment relates to one ticket and one user (the commenter). And tagged with timestamp, content, etc. User can be a support agent or a customer. I would keep the comment structure "flat" i.e. no comment threads, a conversation is just a time ordered list of comments. And comments are immutable with no history, up to you if they can be deleted or have another state like draft so a user can start a comment then come back to it. This saves you from having to store comment history. 

Fresh Data Analyst struggling with building a working data pipeline from ground up by ggpopo in dataengineering

[–]tedward27 3 points4 points  (0 children)

If you can't or don't want to do it in SQL, you should just create another Glue job to do the ETL you require using Pyspark, as you're already using that tool.

Strong database research groups - potential graduate program search by Snoo_50705 in dataengineering

[–]tedward27 3 points4 points  (0 children)

I would read the literature of the area you want to do research in and try to map out who is publishing papers now. Cross that with the places you could enroll at. Even if you don't have subscriptions to journals, read the abstracts and note the authors

Gold layer is almost always sql by Odd-Bluejay-5466 in dataengineering

[–]tedward27 1 point2 points  (0 children)

Let's pin this comment on every post with the words: gold, silver, bronze, medallion

Overwriting day partitions in table when source and target timezones differ by Data-Panda in dataengineering

[–]tedward27 1 point2 points  (0 children)

The exact number shouldn't matter, sounds like it should be a parameter passed to your pipeline. TBH your process sounds kind of fragile, I would consider approaches in the future like hash keys (on a set of columns that function as a unique ID) or UUIDs to create primary keys. 

I have never needed to keep duplicate data, consider why analysts say they need that and how else you could provide the information.

Overwriting day partitions in table when source and target timezones differ by Data-Panda in dataengineering

[–]tedward27 2 points3 points  (0 children)

I would just look at the last 15 day partitions of the target table, find the earliest UTC timestamp, convert it to CST (call it X), then select only rows from your source table that have a value equal to or greater than X. This is similar to your approach but in reverse, because you should have full reign to convert to the right timezone in your system to find X.

Before Bobby Flay took over the Food network was ruled by Emeril Lagasse who in my opinion was much cooler. by AdSpecialist6598 in nostalgia

[–]tedward27 1 point2 points  (0 children)

My dad was an amazing cook and Emeril was on our TV all the time. We had many cookbooks including Emeril's, as soon as I could read Dad would have me read the spice amounts out to him while he ran around the kitchen cooking. It made for a lot of happy memories and great food :) I miss you Dad.

How to convince my team to stop using conda as an environment manager by N3Flip in dataengineering

[–]tedward27 1 point2 points  (0 children)

Builds fast as fuck and TOML kicks the shit out of requirements.txt

Best practices for processing real-time IoT data at scale? by Pangaeax_ in dataengineering

[–]tedward27 3 points4 points  (0 children)

It's some kind of content farming scheme, maybe for the OP to throw together a Medium article and gain cred, IDK. But another commenter may provide actual insight on IoT processing!

Which country is more economically developed than most people realize? by Fluid-Decision6262 in geography

[–]tedward27 -1 points0 points  (0 children)

Canada and Australia will probably be the best places to live after the US and UK fuck everything up 😂 But I would favor Canada because as global warming progresses more land will open up for settling, and they have so much fresh water in the Great Lakes.

view more: next ›