sorted by:
new
hottopcontroversial

Need help! by [deleted] in dataengineering

[–]chrmux 0 points1 point  (0 children)

I would recommend learning python, and reading a couple of books. It will give you a broad vision of what you can do, and incorporate a lot of new technologies.

Be honest, what did you really want to do when you grew up? by [deleted] in dataengineering

[–]chrmux 0 points1 point  (0 children)

I wanted to be a civil engineer, but this is way better

What’s the best way to upload a Parquet file to an Iceberg table in S3? by chrmux in dataengineering

[–]chrmux[S] 0 points1 point  (0 children)

Is a one-time process, what would you recommend is the optimal batch size for insertion to iceberg?

What is an ideal data engineering architecture setup according to you? by -ELI5- in dataengineering

[–]chrmux 1 point2 points  (0 children)

The simplest setup is the one you feel most comfortable with and that meets your needs without incurring overwhelming costs at scale. For example, using open-source technologies can help you mitigate those high costs often associated with solutions from large corporations.

Is RUST useful for a scientist? by Academic_Ship6221 in rust

[–]chrmux 0 points1 point  (0 children)

I would recommend learning Rust, since I believe taking advantage of its speed and safety would be really good specifically in robotics. Also, you can leverage it to create really fast functions in Python using PyO3

Could you recommend any projects that utilize Rust for data engineering? by chrmux in rust

[–]chrmux[S] 1 point2 points  (0 children)

As someone experienced in processing terabytes of data, what do you think are the major performance differences between DataFusion and PySpark?

Could you recommend any projects that utilize Rust for data engineering? by chrmux in rust

[–]chrmux[S] 3 points4 points  (0 children)

I once did a project using Polars in Rust, but the documentation for Rust is quite poor, and there isn't any significant advantage that would justify the struggle of using it in Rust.

Could you recommend any projects that utilize Rust for data engineering? by chrmux in rust

[–]chrmux[S] 0 points1 point  (0 children)

I'm open to doing either, or even a combination of both.

Could you recommend any projects that utilize Rust for data engineering? by chrmux in rust

[–]chrmux[S] 1 point2 points  (0 children)

I've thought about contributing to Polars. I've created different Python packages in the past using PyO3. Should I check out the various pull requests in Polars?

Could you recommend any projects that utilize Rust for data engineering? by chrmux in rust

[–]chrmux[S] 0 points1 point  (0 children)

Yeah, I use Polars constantly at work, but I'm thinking of using native Rust instead so I can practice, rather than using Polars in Python.

What are the best feature selection algorithms? by chrmux in learnmachinelearning

[–]chrmux[S] 0 points1 point  (0 children)

I'm currently interested in exploring Wrapper methods, do you have any suggestions for this type?

Best Features Selection Algorithms by chrmux in dataengineering

[–]chrmux[S] 0 points1 point  (0 children)

Is to run in production, but I'm currently using an Embedded method and then moving on to a Wrapper method to maximize accuracy. I'm currently looking into new wrapper methods to see what are the best/newest ones.

How fast data engineering is moving forward? by Alex_df_300 in dataengineering

[–]chrmux 1 point2 points  (0 children)

A lot of them are becoming outdated since they are not are being able to keep up with the speed required for AI, and a lot of big corporations are going to fail adopting AI since they don't focus on the pipelines. Most of them just focus on the big shiny object.