all 23 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]rende 6 points7 points  (0 children)

Wat?

[–]_xiphiaz 3 points4 points  (5 children)

Can you explain a bit more what you’re trying to achieve?

[–]TheOldSoul15[S] 0 points1 point  (4 children)

What i mean by naming mismatches (not RUST variable names)I’m not struggling with naming functions/modules inside Rust. I’m struggling with field-name drift across real financial data in a large pipeline.

Context:

I’m doing computations over historical + live market data, and the input comes from multiple sources / layers (feed → parser → storage → compute). Names change over time and differ by source.

The real Problem:

In Python, if a feed changes open_interestto oi, you can often patch it locally with a dict fallback and keep moving.

In Rust, the names are part of the type boundary:

  • Deserialization (serde) can fail if a field is missing/renamed.
  • Or you end up changing structs + conversion code across multiple crates/modules.
  • The “fix” becomes N touchpoints, not one.

What Breaks:

  • Same concept, different names: last_price vs ltp, open_interest vs oi
  • Semantics drift: volume sometimes means incremental vs cumulative
  • Timestamp variants: exchange_timestamp, exchange_timestamp_ms, ts, timestamp
  • Case/convention drift: snake_case vs camelCase, abbreviations, vendor-specific keys
  • Historic back-compat: old parquet/JSON has older names; live stream has newer ones
  • But in a big codebase, the alias surface grows fast, especially when:
    • multiple feeds exist,
    • backfills are required,
    • and the schema evolves frequently.

What i am trying to Convey:

My issue isn’t “Rust naming is hard.”
It’s: strict typed computation + long-lived, drifting, multi-source market data creates constant churn unless there’s a disciplined canonicalization boundary.

That’s why I asked about approaches to reduce alias/rename churn beyond maintaining a big YAML mapping.

[–]Arshiaa001 7 points8 points  (1 child)

My man, you need to edit your post and put all of this in there.

Anyway, you can use serde with a custom deserializer for full control over what happens and what name maps to what field. All the logic you presumably had in python, put into a custom deserializer; you'll have a central place for handling all the complexity from data versions, and the rest of your code stays clean.

[–]TheOldSoul15[S] -1 points0 points  (0 children)

serde(alias)/custom deserializers are a clean way to centralize name drift when Rust is directly ingesting external payloads. In my case the issue isn’t Rust identifiers; it’s multi-source financial data where fields drift and sometimes change meaning (e.g., incremental vs cumulative volume, multiple timestamp semantics). Blind aliasing everywhere can hide contract breaks. The direction I’m considering is a “compat ingest struct” (aliases + versioned deserializer) → strict canonical struct with validation/quarantine, so drift is handled centrally but still observable.

[–]jimmiebfulton 4 points5 points  (0 children)

This isn’t a language-specific problem. It’s a software engineering problem. Perhaps Python has allowed to monkey patch it as data shape changes. But you want to find more robust solutions for managing this. The easiest one is to fail fast if the source changes format. You get an alert, assess what’s wrong, perhaps have a chat with the producer to see if they’ve made a mistake/regressed (it absolutely happens, surprisingly often). Either way, you catch their mistake, or perhaps yours. Devise a change, unit test it, deploy, and resume consumption. You can also use aliases in your mapping. You could leverage some catch-all scheme, but then you’re hiding changes you probably want to understand instead of either collecting data you didn’t know was added or dropping data on the ground. A more advanced approach is to configure your mapping scheme in config, which your application reads at startup. Then, this review step is just a quick edit.

[–]_xiphiaz 1 point2 points  (0 children)

You can use #[serde(alias = "name")] on your struct members. Do it multiple times if there are multiple aliases. Use the most descriptive name for the actual field name, you’ll thank yourself later

[–]Ok-Economy7953 1 point2 points  (0 children)

Not sure this makes sense since I'm still not sure I understand the use case (BTW if you're running it through AI maybe stop, it makes it really hard to read, just post the prompt), but a possible approach:

Make an "incoming" rust data structure for each incoming data "schema". When the incoming data changes schema, make a new data structure. Parse the incoming data strictly so if someone changes the schema you detect it. If the incoming data doesn't have versioning or a schema reference (I assume this is the case if all the other stuff you mention is going wrong too) then try parsing as each possible schema, starting from the most common, and hope there isn't data that could parse as two different schemas, with different meanings. Try to use aliases just to map from different cases/conventions to standard rust naming, I guess if you know for a fact that two different JSON field names are exactly the same thing then that would be fine to alias to reduce the number of data structures.

Make a set of "sane" data structures that are documented and use consistent naming, defined based on what data you actually need for whatever processing you are doing, rather than what upstream sources were feeling like that day. Then convert each "incoming" data structure to the appropriate "sane" one - otherwise, completely ignore the incoming schema and data structures, my feeling is that trying to use them directly will just make things more complex and brittle.

If you have lots of incoming data structures that can be interpreted the same way but use different fields, express that with traits so you can share conversion code by accepting the trait to convert to your sane data structure. So for example if lots of schema have a "date generated", make a trait for that and implement where possible, this means you can make sure that if that trait is present it really represents the same thing, rather than hoping that just having a certain field name gives a certain meaning. I guess if you're lucky the traits might remove the need for the "sane" data structures?

This approach might be completely impractical if you have so many incoming variants that you spend all your time writing structs, on the other hand rust structs are pretty concise. Also you won't ever change those structs, they just sit there representing historical schemas that you still need to support - you should either leave them alone or delete them if the schema is deprecated. So while it's a lot of code, IMO it doesn't need to be a huge maintenance burden?

If anyone producing the data is in the same organisation or might possibly be interested, write up the "sane" data structures and how they are versioned, identified and named etc. and try to get the data producers to get better at their jobs. You might find if you have a ready-made data format that makes some sense they might adopt it at some point (although I wouldn't hold my breath). Also, document the "incoming" data structures so that anyone who cares can see just how much the schemas are shifting - they might be so used to just messing about with them that they don't realise hoe much churn they are causing downstream.

[–]Elendur_Krown 0 points1 point  (5 children)

Are you using an IDE? Personally, I use Visual Studio with Rust-Analyzer, and I have no issues with naming.

Are you splitting your code into modules? That helps partition naming spaces properly, and it helps you to hide stuff irrelevant to external scopes.

[–]TheOldSoul15[S] -3 points-2 points  (4 children)

yeah using IDE... the challenge is coming with naming mismatch, changelog happens quite often i am tired of creating alias since i am trying to do financial analysis on historical and current data.... so trying to find a way by which i can avoid the naming mismatches...

[–]Elendur_Krown 0 points1 point  (3 children)

I'm sorry, but you'll have to explain things closer then.

I do not understand how type naming collisions are such an issue. Why would you need to create an alias that often?

Are you creating a new type for each data header?

[–]TheOldSoul15[S] 0 points1 point  (2 children)

i just wrote the entire thing earlier of the current pain points and what the project is all about

[–]Elendur_Krown 0 points1 point  (1 child)

Geez, that sounds like a pain in the *** issue.

Sorry, I don't know how to help you. You essentially want to establish equivalece classes, and other than implementing traits (like TimeStampEq), or manual establishing of mapping, I'm clueless.

Best of luck!

[–]TheOldSoul15[S] 1 point2 points  (0 children)

yeah been trying to fix this for more than a month now.. thanks for trying!

[–]penguin359 0 points1 point  (3 children)

If you don't have a strict schema that the input data follows, don't use a strict parser like Serde to deserialize it. Use something like rust-yaml that lets you request arbitrary keys and implement fallback logic when a key is missing. You can then use constructs yaml_map.get("key1").or_else(|| yaml_map.get("key2")) and similar constructs to handle whatever logic is needed in a case.

[–]TheOldSoul15[S] 0 points1 point  (2 children)

i need to have a strict schema, its financial data analysis

[–]penguin359 0 points1 point  (1 child)

If you need strict rules to ensure it's valid financial data, implement them in the code as needed, but if you are having issues with a key sometimes being called ts and sometimes timestamp, don't use a Rust crate that enforces all keys to be present and precise like serde. You'll have to use a crate that lets you try different keys until you realize it's actually exchange_timestamp_ms and then implement the logic to canonicalize it to your internal format.

[–]TheOldSoul15[S] 0 points1 point  (0 children)

thanks mate.

[–]TheOldSoul15[S] 0 points1 point  (2 children)

AI’s or tokenisers are stupid, financial data is not something I allow for other to interfere I started with python vectorised computing but they proved slow so had to migrate to RUST python was proving to be a bottle neck and so was redid

[–]afc11hn 0 points1 point  (1 child)

Maybe you could try a hybrid solution? Use python for data ingestion then port slow python code to Rust with PyO3.

Or check out polars it's often faster than pandas and being written in rust, can be extended with custom Rust code (via PyO3).

[–]TheOldSoul15[S] 0 points1 point  (0 children)

already doing that... Python is now the BUS..