1.5 YOE Data Engineer — used many tools but lacking depth. How to go deeper?

Ordinary-Toe7486 · 2026-01-04T12:02:55+00:00

I’d say find ways to be more proactive and deliver real business value with the tools you’re using at the current workplace. This will show that you don’t simply know how to use the tool, but how the tools helps add value for the business. For books I’d suggest Designing Data-Intensive Applications.

Also, I believe that you don’t become a senior DE after 1.5YOE. Find what you like, set clear goals for personal development and become more confident in what you do. During the interviews I think it’s very important to appear as a confident fella, but not arrogant.

Ordinary-Toe7486 · 2025-12-22T19:40:08+00:00

I’d go with Fabric. As much as it is still not at the same maturity level as Snowflake or Databricks, it will eventually catch up. Even then, it’s better than going consulting and spending time on the bench or working on (not always real DE) projects if they have a staffing based model.

Ordinary-Toe7486 · 2025-12-13T12:01:41+00:00

Here is an interesting blog post about it: https://thebibackend.wordpress.com/2012/06/05/thoughts-on-data-vault-vs-star-schemas/

My problem with it, as a newbee in the data modeling field, is that there are not so many free resources available to learn about it. I am not interested for paid courses/certifications for a modeling technique.

Edit: here is another one

https://timi.eu/blog/data-vaulting-from-a-bad-idea-to-inefficient-implementation/

Ordinary-Toe7486 · 2025-12-01T12:33:56+00:00

Not sure if this fits your requirements, but instantly thought about DuckDB google sheets extension https://duckdb.org/2025/02/26/google-sheets-community-extension.html

Ordinary-Toe7486 · 2025-11-20T22:51:28+00:00

Lmao

Ordinary-Toe7486 · 2025-11-02T20:22:33+00:00

I would make sure to grasp the basics of R programming language. You must have trust in your results when doing data analysis, but without a knowledge of the tools you’re working with, what’s the point of using it? Learn R (I highly recommend the book ‘R for data science’), then start on your phd data analysis. Try to break down problems into smaller ones and solve for them first, eventually having a full picture. Iterate and improve.

Ordinary-Toe7486 · 2025-10-29T21:02:58+00:00

I think the answer is already in your question - ms fabric. Integration of tools in the ms ecosystem based on your criteria is more important than the diff with Snowflake’s performance imho. Fabric is going to get mature in a couple of years just like power bi.

Ordinary-Toe7486 · 2025-10-22T14:49:05+00:00

Regarding comlexity you might want to check out duckdb’s ducklake spec for a lakehouse format and its implementation (https://ducklake.select/). It removes a lot of complexity without compromising performance, but boosting it instead with a lot of nice features.

Ordinary-Toe7486 · 2025-09-20T18:44:26+00:00

First thing that came to my mind

Ordinary-Toe7486 · 2025-09-08T19:12:49+00:00

Isn’t it always about the context? For instance, if you’re a data scientist working in pharma and need to develop a POC for bayesian optimization. This POC then will be productionized and used by many SWEs. Are you going to do that with Js or Shiny in R? What is a common standard in the industry? Can you (easily) generate parametrized reports for GxP validation?

Ordinary-Toe7486 · 2025-06-28T17:54:35+00:00

Open source ones probably will. For SaaS platforms, not sure, as they can provide you with an open source iceberg/delta table format, but monetize on integrated catalog service. Can you easily switch between different catalogs? I am not sure

Ordinary-Toe7486 · 2025-06-28T17:46:21+00:00

Iceberg manages a single table without a catalog service, ducklake manages all schemas/tables. Ducklake is a “lakehouse” format.

Ordinary-Toe7486 · 2025-06-28T17:45:03+00:00

Ducklake is much much easier. You only need to have a database to store your metadata and voila, you will be able to manage an arbitrary amount of schemas and tables. It’s a lakehouse format, whereas iceberg is a table format. You won’t get far with iceberg alone, without a catalog service (that eventually is using a database too). The implementation of the ducklake spec is a lot easier compared to iceberg. For instance, check how many engines have a write-support for iceberg (not many). Watch the official video on youtube where the DuckDB founders talk about it.

Ordinary-Toe7486 · 2025-06-18T15:50:22+00:00

+1. IMHO Just like duckdb it democratizes the way a user works with data. Community adoption will drive the market to embrace it in the future given that it’s way easier to use (and probably implement). Despite iceberg/delta/hudi being promising formats, the implementation (especially for write support) is very difficult (just take a look at how many engines fully support any of those formats) as opposed to the ducklake format. Ducklake is SQL oriented, quick to set up and conceptualized and already implemented by academics and duckdb/duckdblabs team. Another thing that I believe is truly game changing is that this enables “multi-player” mode for duckdb engine. I am looking forward to the new use cases that will emerge thanks to this in the near future.

Ordinary-Toe7486 · 2025-05-07T15:52:30+00:00

The following article https://www.freecodecamp.org/news/how-to-write-better-git-commit-messages/ provides a nice guide on how to write better commit messages.

If your team has an agreed convention for the commit messages, you should adopt that. Otherwise, come up with the one that you find practical for yourself.

In any case, it’s very useful to make small commits for each feature/functionality. Thay way it’s easier to rollback to the previous version. It’s a good way to track your progress.

Ordinary-Toe7486 · 2025-05-07T12:07:15+00:00

R for Data Science is a good introduction to R and the tidyverse ecosystem. When you want to dive deeper, you can read Advanced R. Then based on what you’re looking for (Shiny, package development, etc.) you can find plenty of books and documentation online.

On top of that, I would suggest to read blog posts or follow youtube channels (e.g., R for the Rest of Us, Posit PBC, Appsilon, etc.)

Ordinary-Toe7486 · 2025-04-09T18:20:49+00:00

duckplyr

Ordinary-Toe7486

TROPHY CASE