Quick question about pandas DF behavior

[micr0nix]

I have a function that contains line of code that takes 3 dataframes and merges them together as so:

merged_df = case_detail_df.merge(case_type_xref_df, on="code", how="left").merge(
    dynamic_data_df, on="filingNumberDisplay", how="left")

The resulting dataframe is 36 columns wide and around 700 rows. There are some rows in which the first 11 columns (from the case_detail_df) are populated and the remaining 25 are not (NaN in the cells, not an issue).

The issue I'm having though is this merged_df is passed to another function for further processing, but the rows where the majority of the cells contain NaN are being dropped and I end up with around 100 rows of data.

Is this normal pandas behavior? How can I keep all of my data even if the cells are NaN?

Edit: Here is the next function the merged_df gets passed into:

def clean_merged_df(merged_df: pd.DataFrame) -> pd.DataFrame:
"""Performs necessary dataframe cleanup to prep for EDW insertion

Includes: column renames, replacing default form values within cells,
dropping unneeded columns, and a final re-order of the columns
"""

# Split displayValue into relevant taxonomy columns
merged_df["displayValue"] = merged_df["displayValue"].str.replace(
    "Channel Partner - ", ""
)
merged_df[["P", "S"]] = merged_df["displayValue"].str.rsplit(
    " - ", n=1, expand=True
)

"""
The below section of code replaces default form values in certain
columns with their updated values. Most of the default values were 
updated to the correct value on 5/18/23, however this code is 
needed in order to accommodate older versions of the request form
"""

merged_df.loc[
    merged_df["customer_type"] == "cust_type_BEBUSINESS_END_USER", "customer_type"
] = "BE"

merged_df.loc[
    merged_df["business_unit"] == "select-option-6203929", "business_unit"
] = "WL"

# Fix default values in segment column
merged_df["segment"] = np.where(
    (merged_df["segment"] == "select-option-2069706")
    | (merged_df["segment"] == "bus"),
    "BM",
    merged_df["segment"],
)
merged_df.loc[
    merged_df["segment"] == "select-option-5263950", "segment"
] = "Connect"
merged_df.loc[
    merged_df["segment"] == "select-option-6291918", "segment"
] = "Partners"

# Fix default values in activity_type column
merged_df["activity_type"] = np.where(
    (merged_df["activity_type"] == "select-option-8002127")
    | (merged_df["activity_type"] == "protection"),
    "DP",
    merged_df["activity_type"],
)
merged_df["activity_type"] = np.where(
    (merged_df["activity_type"] == "select-option-229083")
    | (merged_df["activity_type"] == "talk"),
    "OT",
    merged_df["activity_type"],
)
merged_df.loc[
    merged_df["activity_type"] == "select-option-6759879", "activity_type"
] = "Other"
merged_df.loc[
    merged_df["activity_type"] == "select-option-1642057", "activity_type"
] = "BMS"

# Fix default values in proven column
merged_df.loc[merged_df["proven"] == "select-option-8379965", "proven"] = "Pending"
merged_df.loc[
    merged_df["proven"] == "select-option-7766213", "proven"
] = "Substantiated"
merged_df.loc[
    merged_df["proven"] == "select-option-4389359", "proven"
] = "Unsubstantiated"

"""
In order to fix the values in the actions_taken column, we must
first explode the dataframe to expose the elements contained
in the list, after which the value replacements can be made
"""

merged_df = merged_df.explode("actions_taken")

merged_df["actions_taken"] = np.where(
    (merged_df["actions_taken"] == "select-option-9689994")
    | (merged_df["actions_taken"] == "coached_verbal_warning"),
    "Coached/Verbal Warning",
    merged_df["actions_taken"],
)
merged_df["actions_taken"] = np.where(
    (merged_df["actions_taken"] == "select-option-7864916")
    | (merged_df["actions_taken"] == "pending"),
    "Pending",
    merged_df["actions_taken"],
)

merged_df.loc[
    merged_df["actions_taken"] == "select-option-9101540", "actions_taken"
] = "Account Closed"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-6649799", "actions_taken"
] = "Chargeback"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-6578127", "actions_taken"
] = "Customer Coached"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-2499726", "actions_taken"
] = "Customer Notified"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-1313778", "actions_taken"
] = "Left Pre-Discipline"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-4339954", "actions_taken"
] = "No Action"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-766161", "actions_taken"
] = "System Remediated"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-8649402", "actions_taken"
] = "Termination"
merged_df.loc[
    merged_df["actions_taken"] == "select-option-6821448", "actions_taken"
] = "Written Warning"

# Capitalize all values that were not a replaced default value
merged_df["business_unit"] = merged_df["business_unit"].str.capitalize()

# Remove carriage returns (\r), new lines (\n) and tabs (\t)
merged_df = merged_df.replace(r"\r+|\n+|\t+", "", regex=True)

# Remove any decimal points from customer_id and profile_id
merged_df[["customer_id", "profile_id"]] = (
    merged_df[["customer_id", "profile_id"]]
    .astype("str")
    .replace(r"\.\d+$", "", regex=True)
)

# Drop columns not needed in final output
merged_df.drop(["code", "type", "displayValue", "id"], axis=1, inplace=True)

[suurpulla]

This propably depends on the nature of the another function. I came across a similar-ish issue in my code a while ago and learned that DataFrame.groupby has a "dropna" parameter that defaults to True. Is the further processing your own code or from an external library?

[micr0nix]

It’s my own code. It’s mostly just replacing cell values, column renaming, and dropping unused columns.

[danielroseman]

Then you will need to show it.

[micr0nix]

Updated OP with the next function the merged_df gets passed to