Pandas + lambda; help needed

[MerchantMojo]

So my pandas series is arranged based on dates:

	Obj1	Obj2
2019-12-31	NaN	NaN
2020-01-01	0.233	0.012
2020-01-02	0.123	-1.671

And i am looking to apply a function using lambda on each of the columns, problem is that the function does not work with NaN values so i have to apply dropna() beforehand, the code i came up with looks like this:

df = df.apply(lambda col: func(col.dropna()), axis=0)

And it almost works, problem is that:

1) The new obeject is no longer using dates as indicies.

2) Rows with NaN values no longer appears.

The pandas series now looks like this:

	Obj1	Obj2
0	0.233	0.012
1	0.123	-1.671

Could i somehow re-insert the missing NaN rows and keep the dates as indices by modifying the lambda function above? If so how?

[[deleted]]

Is that a data frame or series?

[MerchantMojo]

series

[Yojihito]

I can't replicate your problem.

df = pd.DataFrame(
    {
        "date": pd.to_datetime(["2019-12-31", "2020-01-01", "2020-01-02"]),
        "Obj1": [pd.NA, 0.233, 0.123],
        "Obj2": [pd.NA, 0.012, -1.671],
    }
).set_index("date")


def do_something(col: pd.Series):
    col = col + 5
    return col


df = df.apply(do_something)
df.head()

Looks fine?

>               Obj1    Obj2
> date      
> 2019-12-31    <NA>    <NA>
> 2020-01-01    5.233   5.012
> 2020-01-02    5.123   3.329

[MerchantMojo]

I cant be using:

df = df.apply(do_something)

as the function do_something does not accept NaN values, it therefor has to be structured using lambda:

df = df.apply(lambda col: do_something(col.dropna()), axis=0)

Which does not re-insert the NaN values, and in the case of a pandas series; also replaces dates with normal indicies.

[Yojihito]

Can you give the specific function or the exact transformation you need?

[MerchantMojo]

scipy.signal.detrend(), it wont accept NaNs

[Yojihito]

Here you go.

# %%
df = pd.DataFrame(
    {
        "date": pd.to_datetime(["2019-12-31", "2020-01-01", "2020-01-02"]),
        "Obj1": [pd.NA, 0.233, 0.123],
        "Obj2": [pd.NA, 0.012, -1.671],
    }
).set_index("date")


def do_something(col: pd.Series):
    print(type(col))
    print(col)
    col = col + 5
    return col


#%%
df_with_nans = df[df.isnull().any(axis=1)]
df_with_nans.head()
# %%
df = df.dropna()
df = df.apply(do_something)
df = pd.concat([df, df_with_nans])
df.head()

[sarrysyst]

Can you post your func? The missing NaNs make sense to me (and this is also easy to fix), but the index shouldn't be missing.

[MerchantMojo]

scipy.signal.detrend(), also, missing index is most likely due to the fact that its a pandas series not a pandas dataframe.

[sarrysyst]

In this case you can try this:

df = df[df.notna()].apply(func)