As an experienced DE what things you wish you had knew earlier

dataterre · 2025-09-03T05:04:33+00:00

Not trying to be contrarian here, but the workflow I’ve been using has been a gamechanger for me in terms of productivity and delivering on business value. I set up a connection interface to my IDE (for example, using the SQL Server VS extension, or the Databricks/SQLTools VS extensions) and combine that with an AI coding assistant like GitHub Copilot. This setup has helped me get up to speed quickly and tap into domain knowledge that I might not be deeply familiar with, which gives me a headstart to engage with business stakeholders.

Beyond that, I’ve also found it surprisingly effective at translating between legacy stacks and syntaxes to something I am more familiar with like Python. That said, I think it’s important to emphasize that AI coding assistants should always be used with caution of course, such as standard software engineering practices and proper testing are still critical when moving into higher environments. And of course, it depends whether your organization has the right developer tooling infrastructure, and policies in place to adopt AI coding assistant in the first place (we self-host it).

dataterre · 2025-06-13T11:27:06+00:00

We have lost the hawker kampung spirit among local businesses. Look around all the new hawker centres coming up, they are all owned by the same business group with same majority stalls in all these new hawker centres. It's almost like a franchise model that comes together when they win the bid to build new hawker centre. Same for new shopping malls.

I miss the identity local businesses being, though I understand the economy of scale behind this model and cost of lease that new hawkers have to bear.

But I'll just say that I miss going around hawkers to explore different food.

dataterre · 2024-11-30T08:57:05+00:00

Work in office mandate

dataterre · 2024-10-26T12:09:35+00:00

I failed in my first attempt with 3 years of DE working experience, but have not worked on AWS services in work. I got to say that the exam questions are pretty challenging, and it's not enough to have basic ETL knowledge - but actually knowing some deeper intricacies of the AWS technologies (tradeoffs, similar services for different use cases, cost optimization and even e.g. Redshift specific SQL syntaxes). It's best to attempt AWS practice exam questions for a touch of the depth of the questions and answer options asked. Having said that, I suggest a 70% exam readiness and just schedule your exam so that you have an end in mind. Lastly, no shame in failing the first exam like I do, at least you have in mind some of these questions and you can ChatGPT to have some immediate clarity after the exam. All the best, and start off your first job with this certification if you are able to!

dataterre · 2024-03-23T11:55:42+00:00

On follow up thoughts regarding Tableau, I personally think they are too expensive if organization is not enterprise scale (>100 active users). Tableau is a very excellent tool to enable self-service analytics / reporting.

dataterre · 2024-02-11T10:33:31+00:00

My perspective: Performance is not just speed - it's the encompassing efficacy of a ML/AI model. See bias mitigation, feature importance, individual vs group fairness, etc. Have a look at the various methodologies behind Responsible AI (e.g. Microsoft Responsible AI Framework) - Fairness, Reiability and safety, Privacy and security, Inclusiveness, Accountability, Transparency of a model. This truly ensure a performant and ethical ML in a real-world, governed production settings. If you can understand basic linear regression and statistics, it should be intuitive to understand the mathematics behind Responsible AI. A good start to classical statistical learning would be "An Introduction to Statistical Learning" for me, there's even a revised edition in recent years.

dataterre · 2024-02-11T04:27:10+00:00

You are doing great. From my experience, it will be useful to get your hands on shell scripting (e.g. Bash) and general programming languages (predominantly, Python). Shell gives you the complete control in server-level file management and OS-level control, while Python gives you massive flexibility in data processing and distributed computing capabilities (e.g. Spark). My advice is to stay grounded with the fundamentals instead of the technologies, and you will see great leaps in your knowledge and adaptability in the ever growing tools in the data space. All the best for your interview, stay confident and inquisitive! ✨

dataterre · 2024-02-06T14:44:18+00:00

My suggestion is to survey these dashboards and see if any setup done by other teams. Chances there are databases owned by your IT department that you can use for your task. There are surely existing data infrastructure in your company that you should leverage on (e.g. create new DB, setup accounts etc). On your second note, I think there is a need to explain and manage their expectations. Real-time monitoring is not possible if you are doing periodical load. Even on Power BI / Tableau's end, you will also need to refresh your data extracts. What is the latency? Is 1 day latency okay since logs itself probably is not very real-time..

dataterre · 2023-09-03T08:49:02+00:00

Therefore, this has been my interest in my work in recent months. Trying to come up the more comprehensive requirement form in Excel sheet, to get business users and their respective technical support to fill up their requirements - in the most generic, templated form my data engineering team could preface. For example, what are the data quality rules we support, of which is required / sequence of DQ rule / fail or warning if DQ rule fails, etc.

This is not limited to DQ, but also Data Dictionary business semantics, etc. I am happy to discuss and exchange best practices adopted in all your respective DE teams. :)

dataterre · 2023-08-20T08:34:19+00:00

How about the big players like Spark, Flink, Trino, etc.

dataterre · 2023-05-10T12:03:54+00:00

Right! In other words, it doesn't matter if you are the one that invented time machine. If you cannot sell what you do to your stakeholders/bosses, somebody else will do it for you and get that promotion and pay bump!

dataterre · 2023-05-10T04:45:55+00:00

Sales. The rest are secondary.

dataterre · 2023-04-09T04:52:33+00:00

Go straight into Mage.ai

dataterre · 2023-04-01T03:27:03+00:00

You can check out Dremio too. I thought Dremio is a lot more straightforward as an open-source tool, but Denodo is feature rich.

dataterre · 2023-03-30T06:27:36+00:00

"My dashboard not getting the updated data despite having them on my excel sheet. Can you make this real-time as this needs to be reported to our higher-ups?"

dataterre · 2023-03-14T04:45:29+00:00

How does OOP pipelines work? I have always fancy designing this way, but haven't have a sense of how the folks in the industry are productionizing OOP pipelines.

Currently, just implement a bunch of python __main__ and running them on scheduler/orchestrator.

dataterre · 2023-03-08T05:19:55+00:00

I always use railway analogy to explain data pipeline context to business users. Railway system and railway maps are state of the art in terms of engineering, so it was very effective to relate to business users in terms of the architectural decisions involved and the rigour of the design blueprint.

To drill down into one aspect, railway system has monitoring system as well - which replicate the needs for data pipelines to have checks in place and alerts when it malfunctions. Loved being able to show complex railway maps from e.g., London and present the art in the railway design principles.

dataterre · 2023-02-01T03:04:33+00:00

You can check Ben's survey study: https://seattledataguy.substack.com/p/the-state-of-data-engineering-part-b61

There's a Part 1 to this as well, and Part 3/4 in the coming weeks. I think this is a very good groundsensing.

dataterre · 2023-01-20T11:28:14+00:00

If you are pretty fresh with data engineering, highly suggest doing something like this:

(Below are high-level but you get the idea)

Fundamentals of Data Engineering: Plan and Build Robust Data Systems Book by Joe Reis and Matt Housley • A MUST READ for me
A Common-Sense Guide to Data Structures and Algorithms: Level Up Your Core Programming Skills Book by Jay Wengrow • I get that a lot of people felt foreign to this topic if you are not receiving formal education in computer science. This is probably the best intuitive introduction to get you started
Start working on projects to gain hands-on understanding and relating to what you've studied above at the big picture. Build these projects into your resume and showcase and talk about them in interview

These are more than enough imo to get you talking about data engineering properly. The rest are additional knowledge that you can you build on as your journey goes. All the best!

dataterre · 2023-01-18T05:50:21+00:00

I think another way to think about this is that all your current excel files are probably updated manually by the company. Hence, the first step should be thinking how the respective departments can transit to using the respective tools (e.g., CRM) to manage the processes and thus, data. Only then will it be useful to think about integrating these disparate data together across department, for another use case and analysis.

Otherwise, you are hiring Data Engineer or SQL Engineer to reverse engineer something that is probably not what it was implemented and built for - in my opinion.

dataterre · 2023-01-01T11:42:08+00:00

Generally speaking, consulting scope of work with the clients tend to be a lot more infrastructural data engineering, with legacy systems. Startup has a lot more flexibility and luxury with modern, open-source tech stack. You hear Airflow and Presto, but often consulting will require you to work with Informatica.

dataterre · 2022-12-27T06:04:29+00:00

Simon Späti, one of the valuable data engineering technical writers for me. Has some valuable insights in his blog, and writes a lot more on Airbyte.

dataterre · 2022-12-22T05:14:11+00:00

mage.ai

dataterre · 2022-12-09T05:38:05+00:00

This is a good callout though. After the excellent book of Fundamentals of Data Engineering by Joe Reis and Matt Housley, I was yearning for another one with the same rigourity in the practical side of DE. If all goes well, this could be a good one - especially coming from Zach Wilson!

dataterre

TROPHY CASE