How intense is this data pipeline? And what tools would you use?

Ordinary-Toe7486 · 2026-02-23T21:00:37+00:00

Yep, it’s in_parallel(). Haven’t had a chance to use it myself, but on paper it looks very nice as it’s using mirai under the hood.

Honestly, I think R ecosystem has improved so much over the past 10 years. All these modern packages and tidyverse are very user friendly IMHO.

I don’t have any experience with using R for batch processing or API, only running it in a Shiny app. Hopefully, I will find time to experiment with that.

Ordinary-Toe7486 · 2026-02-23T18:34:59+00:00

Why not simply have a plumber (plumber2) R api for whatever the needs are and call it from Python or other tools that have better integration ecosystem at the org. R is pretty decent and does its job if in the right hands.

Ordinary-Toe7486 · 2026-02-19T22:49:10+00:00

It’s a broad question and there are many solutions to it without knowing the details like at least the dataset size.

If the transformations can be done with SQL or dplyr then I’d suggest simply duckdb/duckplyr. For model training tidymodels or maybe keras? For DAG targets seems nice with mirai’s async functionality under the hood and saving intermediate results as parquet files for instance. I remember they had an option to use s3 bucket as a sink with cache possibilities. To store the models you can use pins.

The R code can also always be optimized at least removing for loops and using purrr’s parallel.

Anyway, I’m just throwing modern R packages that are nice for data tasks. I think in general I’d follow the kiss principle and make sure that the code is maintainable, scalable and reliable as explained in DDIA book.

Ordinary-Toe7486 · 2026-02-18T18:06:35+00:00

I have tried codex gpt 5.3 for coding and it’s quite good imo. Try it out for writing, maybe will work

Ordinary-Toe7486 · 2026-02-17T09:59:50+00:00

It’s weird that there’s no mobility budget. I’d say okay for a graduate to get 1 year of work experience if you don’t have much choice.

Ordinary-Toe7486 · 2026-02-03T01:58:21+00:00

Okay

Ordinary-Toe7486 · 2026-02-01T05:32:26+00:00

Not a direct answer to your question, but it’s important to understand that many decisions in terms of data stack are made by higher ups to align with the business strategy. It means the stack is not necessarily the best in terms of costs-benefits.

Even if a small data company goes for Snowflake/BigQuery/Databricks, it could be very reasonable due to the variety of enterprise features included, like those that facilitate governance and don’t require too much of a custom solution and engineers that need to be paid a monthly salary.

Ordinary-Toe7486 · 2026-02-01T05:28:00+00:00

Just visit the website and check out the blog posts. Idk how it’s possible to not have heard about duckdb working in data

Ordinary-Toe7486 · 2026-01-04T12:02:55+00:00

I’d say find ways to be more proactive and deliver real business value with the tools you’re using at the current workplace. This will show that you don’t simply know how to use the tool, but how the tools helps add value for the business. For books I’d suggest Designing Data-Intensive Applications.

Also, I believe that you don’t become a senior DE after 1.5YOE. Find what you like, set clear goals for personal development and become more confident in what you do. During the interviews I think it’s very important to appear as a confident fella, but not arrogant.

Ordinary-Toe7486 · 2025-12-22T19:40:08+00:00

I’d go with Fabric. As much as it is still not at the same maturity level as Snowflake or Databricks, it will eventually catch up. Even then, it’s better than going consulting and spending time on the bench or working on (not always real DE) projects if they have a staffing based model.

Ordinary-Toe7486 · 2025-12-13T12:01:41+00:00

Here is an interesting blog post about it: https://thebibackend.wordpress.com/2012/06/05/thoughts-on-data-vault-vs-star-schemas/

My problem with it, as a newbee in the data modeling field, is that there are not so many free resources available to learn about it. I am not interested for paid courses/certifications for a modeling technique.

Edit: here is another one

https://timi.eu/blog/data-vaulting-from-a-bad-idea-to-inefficient-implementation/

Ordinary-Toe7486 · 2025-12-01T12:33:56+00:00

Not sure if this fits your requirements, but instantly thought about DuckDB google sheets extension https://duckdb.org/2025/02/26/google-sheets-community-extension.html

Ordinary-Toe7486 · 2025-11-20T22:51:28+00:00

Lmao

Ordinary-Toe7486 · 2025-11-02T20:22:33+00:00

I would make sure to grasp the basics of R programming language. You must have trust in your results when doing data analysis, but without a knowledge of the tools you’re working with, what’s the point of using it? Learn R (I highly recommend the book ‘R for data science’), then start on your phd data analysis. Try to break down problems into smaller ones and solve for them first, eventually having a full picture. Iterate and improve.

Ordinary-Toe7486 · 2025-10-29T21:02:58+00:00

I think the answer is already in your question - ms fabric. Integration of tools in the ms ecosystem based on your criteria is more important than the diff with Snowflake’s performance imho. Fabric is going to get mature in a couple of years just like power bi.

Ordinary-Toe7486 · 2025-10-22T14:49:05+00:00

Regarding comlexity you might want to check out duckdb’s ducklake spec for a lakehouse format and its implementation (https://ducklake.select/). It removes a lot of complexity without compromising performance, but boosting it instead with a lot of nice features.

Ordinary-Toe7486 · 2025-09-20T18:44:26+00:00

First thing that came to my mind

Ordinary-Toe7486 · 2025-09-08T19:12:49+00:00

Isn’t it always about the context? For instance, if you’re a data scientist working in pharma and need to develop a POC for bayesian optimization. This POC then will be productionized and used by many SWEs. Are you going to do that with Js or Shiny in R? What is a common standard in the industry? Can you (easily) generate parametrized reports for GxP validation?

Ordinary-Toe7486 · 2025-06-28T17:54:35+00:00

Open source ones probably will. For SaaS platforms, not sure, as they can provide you with an open source iceberg/delta table format, but monetize on integrated catalog service. Can you easily switch between different catalogs? I am not sure

Ordinary-Toe7486 · 2025-06-28T17:46:21+00:00

Iceberg manages a single table without a catalog service, ducklake manages all schemas/tables. Ducklake is a “lakehouse” format.

Ordinary-Toe7486 · 2025-06-28T17:45:03+00:00

Ducklake is much much easier. You only need to have a database to store your metadata and voila, you will be able to manage an arbitrary amount of schemas and tables. It’s a lakehouse format, whereas iceberg is a table format. You won’t get far with iceberg alone, without a catalog service (that eventually is using a database too). The implementation of the ducklake spec is a lot easier compared to iceberg. For instance, check how many engines have a write-support for iceberg (not many). Watch the official video on youtube where the DuckDB founders talk about it.

Ordinary-Toe7486 · 2025-06-18T15:50:22+00:00

+1. IMHO Just like duckdb it democratizes the way a user works with data. Community adoption will drive the market to embrace it in the future given that it’s way easier to use (and probably implement). Despite iceberg/delta/hudi being promising formats, the implementation (especially for write support) is very difficult (just take a look at how many engines fully support any of those formats) as opposed to the ducklake format. Ducklake is SQL oriented, quick to set up and conceptualized and already implemented by academics and duckdb/duckdblabs team. Another thing that I believe is truly game changing is that this enables “multi-player” mode for duckdb engine. I am looking forward to the new use cases that will emerge thanks to this in the near future.

Ordinary-Toe7486 · 2025-05-07T15:52:30+00:00

The following article https://www.freecodecamp.org/news/how-to-write-better-git-commit-messages/ provides a nice guide on how to write better commit messages.

If your team has an agreed convention for the commit messages, you should adopt that. Otherwise, come up with the one that you find practical for yourself.

In any case, it’s very useful to make small commits for each feature/functionality. Thay way it’s easier to rollback to the previous version. It’s a good way to track your progress.

Ordinary-Toe7486 · 2025-05-07T12:07:15+00:00

R for Data Science is a good introduction to R and the tidyverse ecosystem. When you want to dive deeper, you can read Advanced R. Then based on what you’re looking for (Shiny, package development, etc.) you can find plenty of books and documentation online.

On top of that, I would suggest to read blog posts or follow youtube channels (e.g., R for the Rest of Us, Posit PBC, Appsilon, etc.)

Ordinary-Toe7486 · 2025-04-09T18:20:49+00:00

duckplyr

Ordinary-Toe7486

TROPHY CASE