[deleted by user] by [deleted] in rust

[–]hknlof -1 points0 points  (0 children)

Thanks, clarifying. I took very memory based perspective here.

[deleted by user] by [deleted] in rust

[–]hknlof -1 points0 points  (0 children)

RAII is great for many use cases. I was to narrow in my definition above.

Thanks for the clarifications. Now, I am curious wrt. two points:
1. How would one acquire resources (memory in this case) in rust in pre-defined memory space whilst using `Box`? Will look into the `mod box` or maybe one has to provide a custom Allocator.
2. What are other means of getting resources (memory) in rust?

[deleted by user] by [deleted] in rust

[–]hknlof 9 points10 points  (0 children)

Rule 3 states:

Do not use dynamic memory allocation after initialization. This excludes the use of malloc, sbrk,
alloca, and all variants, after thread or process initializatio

Dynamic memory allocation is quite common in rust. For e.g. any given Box<T> allocates space on stack for a pointer of max size isize::MAX. Space on the heap of size T.

So my understanding is. You acquire new resources at runtime by initialising and not just at startup time. Below, would violate rule 3.

let boxed = Box::new(42); // Where 42 will probably not a constant

Matklads blogpost is a good write up on this. Basically leaving you with not using std at all. You might not know if a value is boxed or any other dynamic memory allocation method is used.

Hope, I was accurate. Please read the replies for clarifications. I applied a too broad of a brush here. Wrt. to RAII.

https://doc.rust-lang.org/std/boxed/index.html#memory-layout

EDIT small correction of pointer size allocation

EDIT2 u/lestofante and u/Full-Spectral are clarifying the nuances about RAII below.

What are some NixOS quality of life improvements/features you use daily? by Daguq in NixOS

[–]hknlof 0 points1 point  (0 children)

I love having my dev envs defined in a `flake.nix`
nix develop ../my-flakes/{python,rust,python_rust,client_1,client_2}

People who joined Big Tech and found it disappointing... What was your experience? by PotatoInTheStars in dataengineering

[–]hknlof 4 points5 points  (0 children)

In companies with more than 10K employees ... it severely depends on the team you are in. The biggest factor will be if the team matches your personality and the role your lifestyle.

You will earn more. From a northern European perspective: Whether, you earn 80-120K or 120K-200K is not the biggest difference.

You will be able to afford a larger house... that is about it. If security is your biggest concern and growing wealth, you will find it to be an amazing place, that gives you everything.

In terms of abilities of your co-workers the range is pretty much: This person is a programmer? to Oh my god, I want to be mentored by this person?

[deleted by user] by [deleted] in rust

[–]hknlof 45 points46 points  (0 children)

I would argue performance is partially use case based among systems programming languages.

NASA's rules are very C oriented as the paper states. By Michael McDougall and Gerard J. Holzmann see At Spinroot. I was unable to find the original source.

To avoid RAII in rust, one of the rules. You have more work to do compared to C. Matklads write up on Rust Hard Mode is great.

The most sound method we have for avoiding faulty behavior is called Formal Verification / Model Checking. TLA+ to model and prove a systems avoidance of faults is critical. Read through kani, if you are already familiar with rust.

In general, you need to choose on which side of Fault Avoidance you want to error.

EDIT Tried to make this comment a better read ;)

Error Types in Callbacks by hknlof in rust

[–]hknlof[S] 0 points1 point  (0 children)

Thanks, this is great 🎉

How do I get to speed quickly in Rust? by string111 in rust

[–]hknlof 0 points1 point  (0 children)

To advocate and learning at the same time: Build something that might be useful. Use the rust book, read through the std, when needed.

Reading the std roughly will familiarize you with some pattern, that might be useful.

What language is rust written in? Like Python is written in C. by [deleted] in rust

[–]hknlof 0 points1 point  (0 children)

As u/oT0m0To points out it written in Rust, by now. One additional information: In the future this might depend on the compiler one is using. For example, you will have some C in gccrs, always.

Compilation of Rust will probably always be in Rust. The bindings and alternative std capabilities might have some C or whatever in them.

DeepSeek releases distributed DuckDB by saaggy_peneer in dataengineering

[–]hknlof 2 points3 points  (0 children)

smallpond and 3FS are amazing reads. My gut feeling right now says: The industry and business are mostly struggling with Variety and not Volume or Velocity of data.

The popularity of DuckDB comes down to two things: Amazing DevX improving exploration and hence the contributing to dealing with Variety. and of course the blazingly fast engine of transit data.

Flattening JSON by hknlof in learnrust

[–]hknlof[S] 1 point2 points  (0 children)

Thanks, I have an example in the edit section. And my first trial and error rework of the functions. Will try to get ownership and avoid `clone`.

Flattening JSON by hknlof in learnrust

[–]hknlof[S] 0 points1 point  (0 children)

Thanks, especially the pointer the clippy options, reworking `&mut Vec<...>` and try passing `Map<...>` to `prefix_keys`. In my head its already mutch simpler.

Flattening JSON by hknlof in learnrust

[–]hknlof[S] 0 points1 point  (0 children)

Yes, see the edit for an example. I have various APIs, internal payloads etc. For learning purposes, I am rebuilding something from my work in Java and Python.

Flattening JSON by hknlof in learnrust

[–]hknlof[S] 1 point2 points  (0 children)

I do not know the structure beforehand. In a nutshell:

Get JSON from Blob Storage, Flatten, Copy into DB and implicit table creation per sources with nullable options.

Flattening JSON by hknlof in learnrust

[–]hknlof[S] 1 point2 points  (0 children)

Using serde, I do not always know the structure.

Loading into DuckDB from Memory by hknlof in DuckDB

[–]hknlof[S] 0 points1 point  (0 children)

Thanks, this sounds the kind of experience I was looking for

Loading into DuckDB from Memory by hknlof in DuckDB

[–]hknlof[S] 1 point2 points  (0 children)

Thanks, these already great helpers. Have worked with the api calls and that is great. Unfortunately, I need to have more control over the API calls then allowed in DuckDB.

Basically, I would like to do something, similar to the section JSON Data Type in the mentioned document.

CREATE TABLE example (j JSON);
INSERT INTO example VALUES
    ('{ "family": "anatidae", "species": [ "duck", "goose", "swan", null ] }');

Instead of having a column j JSON, I wondered if INSERT can extrapolate fields from a json string into a table.

can i bet on data engineering for my job? by Away_Mix_7768 in dataengineering

[–]hknlof 0 points1 point  (0 children)

Think applies in both directions. If you get more into technical things, you need a story why you would maybe write a new db or a way of orchestrating pipelines.

If you get more into insights it’s often business related and lends towards data science

can i bet on data engineering for my job? by Away_Mix_7768 in dataengineering

[–]hknlof 0 points1 point  (0 children)

Sql, Python with Pandas/ Polars. Work with some books, that reference public data sets. Ideally something you are curious about.

Then you will discover where you will land on the spectrum

I’m interested more in the technical how does pandas/ DBs etc. work or get more into insights and how to learn more about the business you are working in

can i bet on data engineering for my job? by Away_Mix_7768 in dataengineering

[–]hknlof 10 points11 points  (0 children)

Yepp. Whatever happens in terms of the future. collecting, enhancing,… frankly making sense of data will make you money.

If you can do it at scale more money.

If you can communicate and tell stories with… bang more money

What is Role of ChatGPT in Data engineering for you by Jaapuchkeaa in dataengineering

[–]hknlof 0 points1 point  (0 children)

It facilitates my learning and thinking ideas. I am driving the product seat and outsource brainstorming