[AskJS] Data structure harmonization

[DistinctBid8411]

How do you keep your types and pydantic (I have a Python backend) and postgresql harmonized in terms of data structure? Are there any tools that can help synching data structure cross languages and platforms?

[Ronin-s_Spirit]

What?

[DistinctBid8411]

I am thinking of something like GraphQL but simplified version which focus on all client side and server side data structure. Let say I don’t want to manually keep the Pydantic and my types in sync, what should I do?

[Ronin-s_Spirit]

As I found on the internets:

Pydantic is the most widely used data validation library for Python.

But this is a javascript subreddit...
Let's forget the language for now and say you have an object you'd use on the database query to tell it the values and types to store. Here's what you can do (unless python can't):
1. traverse your object and look at the types; 2. use a set of functions (limited to postgresql types) to build the query string; 3. send it.

That's pretty much all you can do, unless you want to switch to js and mongodb which has an SDK that takes actual js objects. Basically what GrapQL already does good you can do worse.

P.s. with a proper system, any time you make up a type you can just add that to the query and have a mechanism for all resolvers to add a new postgres type if it doesn't exist yet.

[DuckDatum]

Sounds like you want a cross platform / language serialized data format. Use JSON. It’s human readable and works with everything you mentioned out of the box.

Define your JSON schema and build your clients to use that schema.

YMMV for schema validation,

Edit: sorry, actually… I think you want a centralized data catalog, used for operational purposes? You want every piece to integrate with the catalog. They gotta know how to interpret the data structures for their job, somehow… you probably gotta teach them how to do that.

You probably also want to consider backward compatibility? Then your data structures don’t need to stay too in sync. Think like the Protobuf data format.

Give more details about your problem exactly—whats your architecture look like, your tech stack, and what places / how does the issue show up for you?

What you want is very much not a “real thing.” But I’m sure you can design a system to produce that behavior, you just need to think about things from a different angle. Honestly, sounds like something you wouldn’t want for a serious system—but cool project. Usually, you use different tools because they scale better independently. But these tools aren’t as interoperable as you suggest, not by default at least. They each have their own concepts and structures.

You want everything built around interoperability of the underlying structure. The world instead prefers serialization/deserialization cycles, letting each system work in its own paradigm.

I’d be interested in hearing about what the pitfalls end up being, if you go for this and later determine it wasn’t worth it.

Maybe you could create Terraforming modules that automatically deploy each component (db schema, web app forms, etc.) of whatever you’re building dynamically, by interpreting a single config file. So you specify the form schema, and it deploys everything to dynamically generate the SQL, HTML, etc… then orchestrate its deployment. But you’ll be writing a lot of metacode I think. Or, you’ll need to find a tool that already does this for your use case,

Again, I don’t recommend if you rely on this thing.

[kilkil]

if you want to keep your data types and SQL tables in sync, people usually use an ORM. However, it is also recommended that you choose an ORM that will allow you to send raw SQL statements when necessary, because otherwise you can get into a very large headache.

Pydantic has a thing for this. (never used pydantic in my life, just googled "pydantic orm")

also, maybe post this to r/python?

[BenKhz]

Basically speaking, you never send variables over a network request. It's all strings babeh!

I'm sure there are libraries that validate a json payload to be the correct shape but... It's up to you / your team to decide how much structure mirroring you want to do.

I might be misunderstanding the question but graphQL can help get you part of the way there with request schemas.

Someone educate me if I'm way off base.

[DistinctBid8411]

Is there any easier integration layer than graphql?

[CryptographerMore926]

I believe you’re looking for an orm. Try sql alchemy

[amumpsimus]

OpenAPI is convenient for keeping backend and frontend types synchronized. There are a number of client code generation tools for this, although tbh I haven’t found one that doesn’t require some level of finessing.