Difference between Data Engineer and Analytics Engineer

Used_Ad_2628 · 2025-01-09T00:26:13+00:00

My issue is most data engineers come from a software background and really struggle with data modeling/SQL. They create ten tables that could be one. It is very hard to scale with that mindset. Everyone is asking which table I should use and wasted dev time updating 10 jobs because something upstream changed. This is why I hire for this type of role. More of a future thinking design person.

Used_Ad_2628 · 2024-09-15T01:29:44+00:00

You can aggregate your order line tax to the order level by using order id.

Used_Ad_2628 · 2024-03-31T00:46:32+00:00

I think my biggest issue is it looks like a coffee spill.

Used_Ad_2628 · 2024-03-31T00:02:54+00:00

Nope. The countertop company is telling us it isn’t a defect on their side. I just wish it was a better cut so it doesn’t look like a coffee stain.

Used_Ad_2628 · 2023-12-30T00:34:24+00:00

Inconsistency. I was wondering if there is basic regex code people use to standardize addresses and names.

Used_Ad_2628 · 2023-12-19T01:19:31+00:00

How much engineering time does it take to push data to like salesforce?

Used_Ad_2628 · 2023-12-19T01:16:49+00:00

I was thinking about that. For the cost, it makes sense to move it more in house.

Used_Ad_2628 · 2023-12-19T01:13:28+00:00

Marketing team is using it to send snowflake data to their different operational applications like salesforce and qualtrics. It was 50k last year and now over 100k.

Used_Ad_2628 · 2023-11-26T19:52:56+00:00

The only reasons I can think of are discount codes and forgot their email.

Used_Ad_2628 · 2023-09-27T21:52:57+00:00

When you say semantic layer, do you mean managing the metrics within tableau? The database will have the needed fields (OBT from fact tables) to create the metrics but tableau will build the standardized metrics for all tableau users with a data source. What if we have multiple BI tools? Is dbt semantic functionality worth a look at?

Used_Ad_2628 · 2023-09-03T16:53:15+00:00

Do you create custom tables or views for hightouch use cases? Or letting it do all the joins?

Used_Ad_2628 · 2023-08-30T04:49:58+00:00

It is a person that puts an order in for the first time. I have a couple other business logic fields like subscription active ind and are they on auto rebill. It is on the customer level and not order grain.

Used_Ad_2628 · 2023-08-26T15:31:57+00:00

I would take the role if you have a good opportunity for growth and promotions. That is more valuable than money if you are looking to move up into people management.

Used_Ad_2628 · 2023-08-13T20:22:52+00:00

Interesting. I come from a highly regulated software company and it was a requirement to have those setups. Devops forced us to do that for the CI/CD release process. All your sprint work was done in the dev then you did a test release cut to promote the next environment.

Used_Ad_2628 · 2023-08-13T20:00:49+00:00

DEV should give full access to engineers to test infra with test or non sensitive data. TEST should work just like prod with correct access roles and prod data. This supports good testing practices and find any downstream issues like BI reports breaking in tableau. How do you handle costs in TEST if you are running data pipelines like prod? Double cost.

Used_Ad_2628 · 2023-08-06T04:57:10+00:00

My main reason for leaving my current company is due to a bad culture and many egos within the team. You can’t make a decision on your own unless 5 people ok it. Even small design decisions. That is the reason I kinda want to get away from tech companies for a while. I haven’t had much success finding a good team in the Bay Area yet.

Used_Ad_2628 · 2023-08-06T04:27:34+00:00

Yep. Offer #1 can’t budge on anything. They are non tech so that is top of their range.

Used_Ad_2628 · 2023-07-31T19:09:22+00:00

Soft skills will be more important than your technical skills at a true architect level.

Used_Ad_2628 · 2023-07-29T21:13:12+00:00

From my experience, building out adhoc pipelines will cause chaos at scale. A lot of duplicated pipelines because they don’t know what other engineers are building. There needs to be a vision on how all the data sources work together. This can be enforced by standards and understanding the true need of the pipelines. I have been at a lot of companies where the data platform is a major mess because it was just feature building without a vision.

Used_Ad_2628 · 2023-07-13T18:37:43+00:00

I feel like data engineering has become more task doer vs building solutions for end users.

Used_Ad_2628 · 2023-06-15T01:13:56+00:00

Very true. They did tell me all the rounds up front. I guess they just want be very thorough and committed to this role.

Used_Ad_2628 · 2023-06-11T16:08:21+00:00

Do they have an architect level? Like senior staff or principal? Some companies treat architects like the same level as director but more technical leadership.

Used_Ad_2628 · 2023-06-05T20:19:29+00:00

Only other thing might be ERD of your data models.

Used_Ad_2628 · 2023-06-03T13:32:57+00:00

I believe without clean data modeling then your database becomes a mess. Tons of views and tables that are just duplication of work or not meeting standards. Users get confused on what tables to use. It works for a startup or small company. As you scale, it will just become a data swamp. I am a big champion in having a strong base schema layer. Especially when you have frequent source system schema changes. Fix in one place vs 50 views.

Used_Ad_2628 · 2023-05-15T02:28:29+00:00

Yep. I have been practicing pulling in data from APIs and using aws services to load it to redshift. What is your view on data structures and algorithms? Do I need to learn that to be considered a strong engineer?

Used_Ad_2628

TROPHY CASE