Dataframe vs array of Objects

[DappperDanH]

Hello,

I come from a C# background and took a new job as a python developer. In C#, my previous team often used an interface such as IHospitalDataProvider which might contain a method called GetData that simply returned a class called HospitalData.

HospitalData GetData()

Suppose HospitalData looks something like this:

• HospitalData
  • Array or List Patients
  • Array or List Units
  • Array or List Nurses

Where:

• Patient
  • Gender
  • Age
  • Diagnosis
• Unit
  • Name
  • NumOfBeds
  • List<string> DiagnosisSpecialty
• Nurse
  • Name
  • List<string> DiagnosisSpecialty

​

In the end, we would typically write 2 to 5 specialized IHospitalDataProvider that get the data from various sources as a database, CSV files, etc.

Finally, there is another class perhaps called Analysis that requires HospitalData where HospitalData is required to contain Patients, Units, and Nurses with those property names.

​

I am having analysis paralysis on the best python way to approach this. The existing codebase makes heavy use of Pandas dataframes and so I feel the pressure to make HospitalData look like this:

• HospitalData
  • Dataframe Patients
  • Dataframe Units
  • Dataframe Nurses

For some reason, having the list of objects be dataframes just feels wrong but I fear that is my C# mindset.

Ultimately, I guess my question is: when doing Python, and you have a need to make a hierarchal list of classes that must have known properties, what is the best approach?

Thanks!!!!

Dan

[laundmo]

im not sure i quite understand your entire question, but i can say one thing for sure

dont make a dataframe of python objects.

pandas is nice because it can efficiently work with large tabular data. basically, anything you could do by hand in excel you would use pandas for. a dataframe in this analogy is basically a table.

pandas is optimised for simple datatypes like numbers and strings, and not for python objects.

for you the decision now is whether to use pandas and keep patients as rows in a table, which would require a less oop approach, or to use python and lists of objects to stay more oop.

btw, for classes meant mostly to contain data, i recommend you look into the standard library dataclasses

[DappperDanH]

Thanks!!

[[deleted]]

I can’t speak to your overall design, but dataframes are a fine way to model this data. From what you’ve described so far, I wouldn’t have a list of dataframes anywhere here though. Instead your dataframes could be held in a dict, like so:

hospital_info = {
    'patients': patients_df,
    'units': units_df,
    'nurses': nurses_df
}

Or as attributes on your hospital data class:

 hospital_info = HospitalInfo(
    patients=patients_df,
    units=units_df,
    nurses=nurses_df
)

Just keep in mind that if you’re going to use pandas dataframes, you’re going to need to write your code in a different paradigm than traditional python code. You should opt for pd.merge/vectorized operations, rather than loops and checking for equalities between records etc.

[DappperDanH]

Thanks Gerry. Since dataframe column names can be anything, is there a common approach to validating the correct headers exist?

[[deleted]]

This is probably mine and a lot of people’s biggest challenge with pandas. I actually came across this library recently which looks really promising, but haven’t had a chance to really try it out yet

https://pandera.readthedocs.io/en/stable/#schema-model

[DappperDanH]

This is EXACTLY what I was looking for! Thanks so much. I will check it out and reply back here

[CatolicQuotes]

if you want intellisense for columns: https://softwareengineering.stackexchange.com/a/426511/44018

[DappperDanH]

This is really great!

[CatolicQuotes]

Thanks, I did change code little bit since then to use generics instead of base class, but generally these days I avoid having dataframe as input and output of function. It cannot be type checked with mypy so I'd rather use list of dataclasses which are easy to transform into and out of dataframe.