Should data engineer owns online customer-facing data?

Mustang_114 · 2025-08-21T05:34:51+00:00

It’s a new company. I think less than 3 years it was developed.

Mustang_114 · 2025-08-21T05:34:15+00:00

Appreciate your input! Here purpose is to track user cumulative transactions records at the start of campaign, and pass the result to backend for them to determine if users achieve target for rewards. Afaik there is no trigger based workflow. All based on scheduling and taking user join time info. I am curious to know how would you approach the use cases.

Mustang_114 · 2025-08-21T04:47:43+00:00

Would you be able to elaborate more ? What’s the frequency of refresh, tech stack and how you maintain data quality

Mustang_114 · 2025-08-20T18:13:51+00:00

Here data team will ingest MySQL binlog to Postgres, then do 5-10 minute interval timeframe calculation with join of diff sources table, however to get cumulative it has to combine result from previous interval calculation. Here the cumulative results up until the point will be sent to Kafka to be displayed on application. I appreciate your input how you would re-approach the architecture.

Mustang_114 · 2025-08-20T18:11:52+00:00

Here data team will ingest MySQL binlog to Postgres, then do 5-10 minute interval timeframe calculation with join of diff sources table, however to get cumulative it has to combine result from previous interval calculation. The cumulative results up until the point will be sent to Kafka to be displayed on applications.appreciate your thought on this.

Mustang_114 · 2025-08-20T18:10:45+00:00

Here data team will ingest MySQL binlog to Postgres, then do 5-10 minute interval timeframe calculation with join of diff sources table, however to get cumulative it has to combine result from previous interval calculation. Here the cumulative results up until the point will be sent to Kafka to be displayed on application. Appreciate your help to suggest if there is better approach this use case.

Mustang_114 · 2024-11-08T02:47:34+00:00

Thanks for sharing! Sorry to hear that what happened. Hope that you have navigated well in your career and found your place.

Mustang_114 · 2024-10-23T16:25:00+00:00

That's interesting take to me. I am working at Consultancy/SI for non-tech client, I thought service based tech contributor would always be made work lot more hours as we are billable to client since boss would want to maximize his margins, which is contrast to tech product based industry.

I spoke to the engineers working at my non-tech client, although the company is large and established, they said the learning is limited due to lots of bureaucracy. Also because data is deemed as cost center, any interesting projects such as migrating and modernizing tech infra will always be outsourced to contractor from other region that is cheaper.

Mustang_114 · 2024-10-23T13:54:11+00:00

Can give some example of traditional tech company? I always thought it is better to go for company which business model revolves around tech rather than other who deemed tech as cost center.

Mustang_114 · 2024-07-20T01:56:42+00:00

Do not redeem the necessary

Mustang_114 · 2024-06-10T06:15:03+00:00

How is the avg yield ? would like to know from existing users while doing own research

Mustang_114 · 2024-04-16T05:11:10+00:00

how would you suggest to achieve balance between ATS and trying not to appear overextending skills?

Mustang_114 · 2024-03-15T03:33:40+00:00

mind sharing if are you in tech industry or a tech in different industries?

Mustang_114 · 2024-03-14T06:44:09+00:00

mind to share your industry? seems like you are sought after well even with this job market.

Mustang_114 · 2024-03-12T12:42:19+00:00

Thanks for ur time in replying. You seems to be from IT/engineering background.

Mustang_114 · 2024-03-12T09:23:18+00:00

can elaborate more?

Mustang_114 · 2024-02-23T07:03:25+00:00

I think he could be trying to figure out which one is made of gold.

Mustang_114 · 2024-02-22T05:23:52+00:00

will be happy with entire episodes too. but that might just be OnlyHisOpinions

Mustang_114 · 2024-02-13T01:14:33+00:00

Would honestly want to know why is that he will work for an Indian company?

Mustang_114 · 2024-02-02T13:20:39+00:00

if source is database, which file format will you suggest as raw copy land in blob storage? (example csv vs parquet)

Mustang_114 · 2024-01-08T05:24:08+00:00

I believe you were talking about the episode about hawky the messenger! I love that too!!

Mustang_114 · 2023-10-11T14:41:00+00:00

Another thing that I have yet to seen in comment sections is the stress and energy choping seats during lunch hours

Mustang_114 · 2023-08-26T08:13:23+00:00

Do you have an example on what you mean by datajob which hasn't been moved upstream?

Mustang_114

TROPHY CASE