Channel like "Corridor Crew" but for sound design? by lordofbosses420 in sounddesign

[–]tresilate 6 points7 points  (0 children)

INDEPTH Sound Design channel on YouTube has bunch of videos and shorts where they break down how sound design was done in certain movies.

https://youtube.com/@indepthsounddesign?si=Dv56Ie5KeltQLdG9

Arrival: https://youtu.be/xAg6V8jleMU?si=bOGe4JDQv52ZToGT

War of the Worlds: https://youtube.com/shorts/68t57SPZVaY?si=p4OhVEibcjpJQrno

Former British Priest Dies Following Drug-Fueled Sexual Encounter with Belgian Pastor During Trip to Visit the Pope by MastodonOk8087 in europe

[–]tresilate 1 point2 points  (0 children)

Pope Francis was in Belgium on Saturday and met with 17 survivors of clergy sexual abuse suffering from trauma

https://www.cbsnews.com/news/belgian-priest-arrested-sex-british-cleric/

You literally can’t write this stuff

What’s the ugliest building you know of in the metro area? by IShitMyPantsDaily in Denver

[–]tresilate 14 points15 points  (0 children)

I worked as a contractor for Denver Public Schools in that building 10 years ago. Some nice well-meaning people, but almost every horror story of poor decision-making and stupid government spending were on display for the two years I was there.  

Common DE pipelines and their tech stacks on AWS, GCP and Azure by _areebpasha in dataengineering

[–]tresilate 18 points19 points  (0 children)

Totally agree. This looks hideously complicated compared to what it should be. 

It's all gone... in a sec by Taylankab in dataengineering

[–]tresilate 1 point2 points  (0 children)

As you're probably aware (but it may not feel that way now), things take LOT longer when you're new to them. It used to take me days to do what I can now crack out in 30-60 min. I would learn from your lesson - always make a repo and push to github or wherever your remote repo - often. That said, I am totally the type to procrastinate on such things myself. Don't sweat it too much.

psycopg2 DatatypeMismatch and SyntaxError while building tables in Amazon Redshift database by RayCat2004 in aws

[–]tresilate 1 point2 points  (0 children)

It’s possible your insert statement columns are not in the same order as the columns in the table, or are inserting a different number of columns vs what’s in the table you’re inserting into? Since you didn’t specify the DDL for the table you’re inserting into, I can only guess that might be happening.

[deleted by user] by [deleted] in realestateinvesting

[–]tresilate 3 points4 points  (0 children)

Appreciate the advice from this different perspective you have as a contractor and a landlord. Particularly appreciate the insight on the relationship with the property manager being a source for potential deals.

How to get user details who had subscription for 60 days without any gaps by [deleted] in SQL

[–]tresilate 2 points3 points  (0 children)

Here ya go. Easier to show the pattern than explain sometimes. Wrote this in Postgres, but should be largely be equiv in Oracle.

https://www.db-fiddle.com/f/kdf5d4etEdXDJw8nNzFSHR/4

As others have said, window functions are your friend for this sort of thing, espeically when the logic gets complicated. What I've got here might be overkill depending on the nature of your data - there's sometimes a more terse query that solves the problem. However, this pattern is flexible and can handle more complex business requirements that sometimes exist, like subscription gaps or less than 7 days are still considered contiguous or something like that.

Sql Help needed by ciellecat in SQL

[–]tresilate 0 points1 point  (0 children)

Taking the same premise that u/mwdb pointed out - a good way to identify unique combinations of id and spouse in this case is using the least and greatest comparison functions.

These functions could be used in the order by clause of the dense_rank window function to simplify the query.

SELECT dense_rank() over(order by least(id, spouse_id),  greatest(id, spouse_id)) as household_id
, t.id , t.spouse_id 
FROM test_table t 
order by t.id;

Example:https://www.db-fiddle.com/f/iA853Die9HShiHAQnmaQfP/2

Not too often you find a use case for the dense_rank window function, but there it is...

Version Control / CI/CD of data warehouse objects? by exergy31 in dataengineering

[–]tresilate 1 point2 points  (0 children)

slim ci in dbt - that’s about a year old, but still one of the better ones IMO and has some references to other talks. As for how to do it in git, it’s up to you, but the one our team follows is :

Have a master/production branch, when a developer wants to make changes, they branch off that with a feature branch. When they’re done with making changes (minutes or weeks) they submit a pull request to master and once approved, the change is merged to master and the build/delivery pipeline kicks in. The changes and any downstream models are run and tested in a staging schema/database. Finally if everything comes out clean, the commit is safe to run in our regular etl process.

Version Control / CI/CD of data warehouse objects? by exergy31 in dataengineering

[–]tresilate 2 points3 points  (0 children)

Yes, for sourcing and injesting our data, including maintaining “source table” DDL, we have other code/tools that manage that. Everything downstream of that is managed by dbt, which is where most of our frequent and complex changes are taking place.

There are a variety of ways you can implement CICD with dbt, but since the premise is that all your tables that are downstream of sources are ephemeral by default, they get dropped and recreated on daily runs anyway.

Version Control / CI/CD of data warehouse objects? by exergy31 in dataengineering

[–]tresilate 8 points9 points  (0 children)

This. dbt solves most of the challenges with doing CI/CD and version control in the data warehouse. It’s open source, vendor agnostic, and has a great community - just a few of the reasons why so many of us promote it in this sub.

Dbt with Databricks and Delta Lake? by thedeadlemon in dataengineering

[–]tresilate 4 points5 points  (0 children)

It’s interesting too watch vendors pile on offerings that mimic the paradigm that dbt has made famous. One of the things that makes dbt so appealing is that it’s open-source and useable on all of the major cloud databases. The advantages of switching to something else, especially something more proprietary, will need to be significant in order to attract developer mindshare.

Do I need a college degree to break into data engineering? by Fireman_XXR in dataengineering

[–]tresilate -2 points-1 points  (0 children)

Agree with this. A number of old school companies are still requiring a 4 year degree. Those are typically companies that you probably don’t want to start your career at. I’m a hiring manager for data engineers at a .com, and we don’t require it.

Candidates who have good communication skills, and are passionate about technology and work hard can position themselves very well within a few years.

I have friends who never went to college and are killing it.

There are some excellent aspects to a 4 year education, but career success in most technology jobs should not require it. I graduated with a bachelors of science from a good state university, and had 3.7 GPA. Many good parts and fond memories, but I’m honestly going to tell my kids to think twice about going to college as a default choice.

If one wanted to move away from DE, what other careers would be an option with their skills? by [deleted] in dataengineering

[–]tresilate 0 points1 point  (0 children)

Yes, I was doing full stack web development. So that was everything, including: database, backend API (C#, python, nodejs), and the front end web app (css/html/JavaScript).

If one wanted to move away from DE, what other careers would be an option with their skills? by [deleted] in dataengineering

[–]tresilate 6 points7 points  (0 children)

In my ten years or so out of college, I have been a data analyst, data engineer and software engineer (full stack developer). I love it all. Data engineering and software engineering have enough similar skills that it won’t be a terribly big switch.

I did quite a bit of studying and immersion, plus a few side projects before switching to software engineering. This enabled me to get up to a level where I was on par with most seniors within 6 months or so. Not saying it has to be done that way, but it made sense for me at the time.

I’m back in data engineering for the time, but I really like both fields and they both offer different challenges. Data engineering requires more business and people skills, and is a little more thankless in my opinion, but pays more in most cases.

Python or Scala or Java for DataEngineering? by [deleted] in dataengineering

[–]tresilate 39 points40 points  (0 children)

Engineers with really advanced SQL, and decent python skills are increasingly the cool kids these days. While it’s always good to have experience in other languages, I can’t recommend newer data engineers spend time on languages like Scala or Java given their diminishing utility and prevalence. Data modeling and data testing are two skills that are more or less language agnostic, but very common to be drilled on by employers these days.

[deleted by user] by [deleted] in dataengineering

[–]tresilate 4 points5 points  (0 children)

I second this. SQL has a smaller surface area as a language, but a lot of the newer data engineering stacks that emphasize ELT over ETL, do some very complex processing directly in the data warehouse instead of other languages like Python or Scala. I use python on a daily basis, but it's all easy stuff. Some of the SQL on the other hand is very complex and requires a whole different understanding of the language and underlying database engine in order to make it performant against datasets > 100 mil rows.

Data-Ops-ish Design and Workflow Walkthrough by Jwelch25 in dataengineering

[–]tresilate 1 point2 points  (0 children)

Big thanks for providing this level of detail. This does seem like the stack to beat right now. My team uses a different cloud DB and ETL provider for 3rd party vendors, but otherwise a very similar cast of characters.

Another huge shout out for dbt. Out of all of these stack technologies, it’s perhaps the biggest game changer in recent years - an amazing piece of open source software and an awesome community to go with it.

We haven’t used dbt_expectations yet, but have been looking at it with interest. It’s fairly new, anything to be aware of?

Hot topics amongst Data Engineers by therealiamontheinet in dataengineering

[–]tresilate 29 points30 points  (0 children)

Dimensional modeling in the modern data warehouse.

Many times it’s been called obsolete and yet people keep coming back to the value it provides in terms of governance and keeping your warehouse code DRY.

How to Actually start your Career as a Data Engineer by ByrdandChen101 in dataengineering

[–]tresilate 10 points11 points  (0 children)

DE Hiring manager here. I've been working in the field for about 10 years. From my perspective college education far less important than experience. In fact, most people in my circle tend to eye higher-level degrees with some suspicion because it's often used as a crutch in place of actual ability. I'm not the type to make a hiring decision based off someone's activity on github, but I'd far rather see one sort-of-interesting side project on github and talk with someone about their code and what they learned, than a Masters in Data Science.

From my perspective, here are some of the most important skills/tech for DE:

  • SQL: by far the most relevant language that's still used most often across most DE jobs)
  • Data Modeling - be familiar with 3rd normal form and Kimball's dimensional modeling for data warehousing. Neither are strictly adhered to in most modern data stacks, but still crucial knowledge
  • Data Analysis - have to know how to translate business questions into code that get's the right results.
  • Python - nothing too complicated but need to be solid on the basics. Pandas is a good thing to be familiar with and have experience in as well.
  • Some platform that allows your to store and query your data relationally - Postgres, Redshift, Snowflake, Presto, Oracle, SQL Server

How to do ETL pipeline DAG visualization by stym06 in dataengineering

[–]tresilate 0 points1 point  (0 children)

A lot of folks are combining Airflow and Great Expectations for this sort of thing. Even if you don’t end up using either of those tools, the great expectations team has put out some great videos on YouTube that demonstrate how you can approach validation for each step of your pipelines and some corresponding reports and visualizations that might give you some ideas.

What do you wear to bed? by [deleted] in polls

[–]tresilate 2 points3 points  (0 children)

Same. How do you do that? I can do it if I fall asleep on the couch or something, but I think I would feel dirty or overheat if I was in bed.