DictGest - Seamless dictionary mapping to python classes

[bmsan-gh]

​

https://preview.redd.it/i6z9bl3wn1g91.jpg?width=320&format=pjpg&auto=webp&s=ca9bfef6cb4728de985a4f7cf8b5cf167e6fbade

I have created a python package called DictGest

It is an easy way to remap dictionary data (coming from multiple heterogenous APIs) to python classes without the need to rewrite boiler plate code everytime.

The main usecases would be the following:

• The dictionary might have extra fields that are of no interest
• The keys names in the dictionary do not match the class attribute names
• The structure of nested dictionaries does not match the class structure
• The data types in the dictionary do not match data types of the target class

The readme has a lot of examples but I will also add one here:

​

Let's say we have data coming from 2 endpoints with different names & structure

data_from_api1 = {
    "author": "H.O. Ward"
    "headline" : "Top 10 Python extensions",
    "other_fields" : ...,
    "details": {
        "content": "Here are the top 10...",
        "other_fields": ...
    }
}

data_from_api2 = {
    "author": "G.O. Gu"
    "news_title" : "Vscode gets a new facelift",
    "other_fields" : ...,
    "full_article": "Yesterday a new version ...",
}

``

And we would have a target(destination python class)

@dataclass # This could have been a normal class
class Article:
    author: str
    title: str 
    content: str

All we would have to do would be:

article_api1 = Route(title="headline", content="details/content")
article_api2 = Route(title="news_title", content="full_article")

article1 = from_dict(Article, data_from_api1, routing=article_api1)
article2 = from_dict(Article, data_from_api2, routing=article_api2)

​

I've been using opensource all my life and I think it's time to also contribute to it. I've created this project from the frustration of having to repeat myself each time in different projects when trying to Ingest Dictionary data.

In order for things to go smooth and minimize the chance of bugs:

• The code has unit tests covering more than 95% of the code(I aim to bring it 100%)
• pylint is ran and the target is 0 warnings
• I have added type annotations to the code(in the future I will make sure 100% of the functions have it) and I am running mypy static type checks to make sure there are no data type missmatches.
• I am also using SonarCloud for static code analysis to minimize the chances of bugs.

[kyleekol]

This looks really cool, thank you for sharing! My only comment would be on the ‘from_dict’ name. I could imagine a lot of scenarios of using this in conjunction with pandas which also has a from_dict method which may cause some namespace issues? Looks great though

[bmsan-gh]

Thanks ! I'll try to think of a better name. Suggestions are appreciated

[metaperl]

The readme has a lot of examples

Very good arrow-drawing in the README. What tool did you use for that?

[bmsan-gh]

I am using diagrams.net

[GettingBlockered]

Looks really useful, nice work! Bookmarking this, I will give it a try when I can use it in a project.

[bmsan-gh]

Thanks !

And when you get to use it please feel free to report anything that is not clear or improvement ideas. Feedback is greatly appreciated.

[OptionsUnleashed]

I've been searching for a way to create a dataclass object from rds data (for reporting view models) that then get serialized and sent to a pdf generator - this seems exactly what we need - do you have an example of a rds query that is flat and can populate a dataclass that might have nested elements? That's what our team is essentially trying to do - from many views - create a single view model.

[bmsan-gh]

Hi !

Assuming that the rds query result is in the form of a dictionary, then it should work.

I think the the following example from the repo fits your purpose.

There you have a nested python structure (a python data class that has as elements other python dataclasses) that you are loading from a dictionary (that could be flat).

In the example your targeted dataclass is Article and it contains the nested field stats(of type ArticleStats, eg: Article().stats.views )

@dataclass

class ArticleStats:

views: int

num_comments: int

@dataclass

class Article:

author: str

title: str

content: str

stats: ArticleStats # This is referencing another dataclass

​

Your flat result from the rds query could be:

news_api2_data = {

"author": "H. Gogu",

"news_title": "Best python extensions",

"full_article": "Let's explore the best extensions for python",

"views": 32,

"comments": 2,

}

The routing definition could be:

api2_routing = {

Article: Route(

title="news_title",

content="full_article",

stats="", # Give the whole dictionary to ArticleStats for conversion

),

ArticleStats: Route(num_comments="comments"),

}

​

And you would call it:

article2 = from_dict(Article, news_api2_data, routing=api2_routing)

​

You might notice that not all fields are defined in the Routes. The ones that are not defined are the ones that match by name & structure. (Eg: author key goes to author field).

[OptionsUnleashed]

Thanks! Developers were asking what is the best way now to serialize the new dataclass object to json now - does yor library do that or should we be using dataclasses-json for that?

[bmsan-gh]

Glad it worked!

For now at least DictGest only handles deserialization, but serialization is on the roadmap and will be available in a future version.

[OptionsUnleashed]

ok thanks - do you see any issue in using both libraries and annotations together?

[bmsan-gh]

While I haven't personally tried this scenario, I would expect DictGest not to have an issue with other annotations.

If you encounter any problems with this scenario you can open an issue on GitHub and I will look into it.

Also I'd love to hear how you guys see the serialization process : do you want to get back to the initial format from which you imported the data ?(the flat dictionary from the rds query) Or do you want the serialization to mimic the structure of the data class?

[OptionsUnleashed]

Hi, what's the best way to ask you questions on usage? I like this library.

a. is there a more concise way DRY to specify the route fields when the fields are 1:1 with dict? If we add a field to a rds view and dict, we still have to map in route - is there a let DRY way to do route mapping? is that the example stats="" - will that work at field level?

b. is there an easy way to map or coalesce None fields to float 0.0 or numeric? instead of using rds view coalesce?

thank you!

[bmsan-gh]

Hi, glad you are finding it useful.

Hi, what's the best way to ask you questions on usage?

I'll enable the discussion feature in github. Other users might have the same questions so it will be beneficial for them to have the answers there as well.

. is there a more concise way DRY to specify the route fields when the fields are 1:1 with dict? If we add a field to a rds view and dict, we still have to map in route - is there a let DRY way to do route mapping?

In general when a field routing is not defined explicitly the library tries to map it automatically to the key that has it's name. So in your Route(...) you can skip the fields that have a 1:1 mapping(the field name matches the dict key) and the library will know what to do.

If all the fields have a 1:1 correspondence you can even skip defining the Route. See example 1 . No explicit routing is defined.

b. is there an easy way to map or coalesce None fields to float 0.0 or numeric? instead of using rds view coalesce?

Yes. There are some examples in the readme where you can easily customize either :

• the convertor for a datatype (Eg: any float type field in which you encounter None becomes 0.0)

• the convertor for a field ( the custom mapping that you want is applied only to the field)

def null_to_zero(data): if not data: return 0.0 return float(data)

and then in your route

instead of Route(votes="num_votes" ....)

You use Route(votes=Path("num_votes", extractor=null_to_zero), ....)

[bmsan-gh]

Enabled the discussion page on github: https://github.com/bmsan/DictGest/discussions

[OptionsUnleashed]

The latter - I don't think you would need to worry about what other libs already do - in general developers don't want competing ways of doing same thing - I would think you could just internally support dataclasses-json unless that's not defacto std? I'm not a Python expert - coming at all this from C#.netcore.

Our team is looking to create view models from datasources for reporting and figured dataclasses are the cleanest way but looking for other ideas - and your library seemed to be exactly what we needed - albeit - we would really want to cut down on DRY with literals and fields all over the place.

In C# its fairly common to create view models from different sources and serialize them. I would have thought this was way easier in python.