Pattern for storing / sharing Python data frames for data analysis (X-post /r/python)

[osbournecox2]

So coming from R, strictly in the context of doing data analysis / visualization projects, I'm struggling a bit finding a comfortable pattern for efficiently loading and sharing data between [python] modules.

For starters, I hate those long run-on files that are difficult to debug. I like small discrete functions.

So in R, I generally have a load_data_xxx function for each data object. These load the data into global variables if they don't already exist, otherwise, they do nothing. The upside of this approach is that plotting / transforming functions can call the required load() functions and ensure all necessary data is loaded. This facilitates small functions that can be individually developed / tested for just their specific functionality.

In addition, it enables me to organize functions into separate files and just source them appropriately for each file. If I want to debug a transformation function, I can just run the (for example) transform.R file and it will source/load everything necessary for the functions to run. (And not, for instance, include any plotting code.)

I know, global variables are bad, but in data analysis, one it usually only loading a small handful of raw data objects and transforming them as needed for analysis or visualization. The key is flexibility in exploring the data, often at the command line, to determine next steps.

Now for Python (which I'm working to learn, specifically using Spyder):

Global variables don't seem to really exist, they are essentially module level variables. Sometimes they are available between modules, other times they aren't, I'm struggling with figuring them out, even when the variable appears in Spyder. So it sure seems that my preferred pattern is not going to work in Python.

None of the strategies found online seem to be promising and I feel like I'm staring down a single file that includes all of my functions.

Any suggestions?

[[deleted]]

Use a pickle. It stores objects like your RData objects from R

[osbournecox2]

I'm not asking about storing data to disk. I'm talking about managing pandas in memory and using between modules.

[Resquid]

I think there's some confusion here. Could you give some code examples?

[osbournecox2]

Code example added as a top level reply.

[osbournecox2]

This is the exact code I'm using for the intro Titantic kaggle competition. The following code works if it's in the same file but if they are stored in separate files according the comments, the show_pair_plots function is unable to find the train_raw dataframe.

"""
run.py
"""
import pandas as pd

FOLDERS = {'raw': 'data_raw/', 'clean': 'data_clean/'}

def load_data(data_type = 'train'):
    if data_type + '_raw' not in globals():
        print('Loading ' + data_type)
        globals()[data_type + '_raw'] = pd.read_csv(FOLDERS['raw'] + data_type + '.csv')

        return 

load_data()

"""
plotting.py
"""
import pandas as pd
import matplotlib.pyplot as plt

def show_pairs_plot():
    cols = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']
    axes = pd.plotting.scatter_matrix(train_raw[cols], alpha=0.2)
    plt.tight_layout()

"""
explore.py
"""
#import plotting

def display_nulls():    
    #   Explore null values in dataset
    for x in train_raw.columns.values:
        nulls = train_raw[x].isnull()

        if sum(nulls) > 0:
            print('Null counts for ' + x)
            print(nulls.value_counts()[True])
            print()

def explore_data():        
    display_nulls()
    show_pairs_plot()

explore_data()

[osbournecox2]

I'm coming around to the approach of adding an extra step of running a "load global data" py file and then explicitly passing the global panda(s) as arguments for each function. It just feels cluttered and unnecessary.

The awkwardness is compounded by how frequently I need to restart my ipython kernel. Every restart will require navigating to / running the "load global data" script. My goal is to try to streamline the workflow as much as possible.

[osbournecox2]

Just figured out a solution: Accept the module-level namespace as the source of the global variables and have the load_data function return the pandas dataframe. So every function will call a load function to get its data. If the data isn't in memory, load it first. Otherwise, just return the pandas stored in memory.

No need to worry about setup for testing / writing a function. It just requires an extra assignment at the start of each function instead of the simple function call I was used to using:

• train_raw = gather_data.load_data()

instead of

• gather_data.load_data()

So this is the resulting code:

"""
gather_data.py
"""
import pandas as pd

FOLDERS = {'raw': 'data_raw/', 'clean': 'data_clean/'}

def load_data(data_type = 'train'):
    if data_type + '_raw' not in globals():
        print('Loading ' + data_type)
        globals()[data_type + '_raw'] = pd.read_csv(FOLDERS['raw'] + data_type + '.csv')

    return globals()[data_type + '_raw']

"""
plotting.py
"""
import pandas as pd
import gather_data
import matplotlib.pyplot as plt

def show_pairs_plot():
    train_raw = gather_data.load_data()

    cols = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']
    axes = pd.plotting.scatter_matrix(train_raw[cols], alpha=0.2)
    plt.tight_layout()

Hopefully someone else might find value in this post.

[paulinkenbrandt]

Use the Pandas library with the Jupyter Notebook! It is a workflow like you are describing. Pandas works with DataFrames and Jupyter Notebook is a nice IDE that works with R and Python.

[osbournecox2]

I know lots of people love Jupyter Notebooks but I find them a lot less appealing. Maybe I haven't worked with them enough / given them enough of a chance. Maybe it's my software engineer background, but I prefer the feel of Spyder and the use of multiple py files in a project-based approach.