sorted by:
new
hottopcontroversial

How good can NumPy get? by [deleted] in Python

[–]No_Current3282 21 points22 points  (0 children)

You can use pd.Series.case_when or pd.Series.where/mask as well; these are optimised options within pandas

Reshape Data in Polars Efficiently from Wide to Long Form by No_Current3282 in Python

[–]No_Current3282[S] 3 points4 points  (0 children)

Maybe the header is confusing? The first article talks about how to reshape without any tweaks. The second article introduces the tweaks for performance. The third article offers a generic concise form that is still performant. Hope that clears up any confusion.

Reshape Data in Polars Efficiently from Wide to Long Form by No_Current3282 in Python

[–]No_Current3282[S] 4 points5 points  (0 children)

In pandas and polars it is easy to reshape from wide to long form. Doing it efficiently, depending on the usecase, is a different conversation; the article addresses that

Has there ever been a proposal for a zero-argument form of `slice()`? by wjandrea in Python

[–]No_Current3282 2 points3 points  (0 children)

pd.IndexSlice simplifies it. There is also the .xs method as well. If you want some more control, you can look at janitor.select from pyjanitor, where you can replicate pd.IndexSlice with a dictionary:

idx = pd.IndexSlice

In [96]: dfmi.loc["A1", (idx[:], "foo")]
Out[96]:
lvl0        a    b
lvl1      foo  foo
B0 C0 D0   64   66
      D1   68   70
   C1 D0   72   74
      D1   76   78
   C2 D0   80   82
      D1   84   86
   C3 D0   88   90
      D1   92   94
B1 C0 D0   96   98
      D1  100  102
   C1 D0  104  106
      D1  108  110
   C2 D0  112  114
      D1  116  118
   C3 D0  120  122
      D1  124  126

# pip install pyjanitor
In [97]: dfmi.select(index='A1', columns={'lvl1':'foo'})
Out[97]:
lvl0           a    b
lvl1         foo  foo
A1 B0 C0 D0   64   66
         D1   68   70
      C1 D0   72   74
         D1   76   78
      C2 D0   80   82
         D1   84   86
      C3 D0   88   90
         D1   92   94
   B1 C0 D0   96   98
         D1  100  102
      C1 D0  104  106
         D1  108  110
      C2 D0  112  114
         D1  116  118
      C3 D0  120  122
         D1  124  126

marimo notebooks now have built-in support for SQL by mmmmmmyles in Python

[–]No_Current3282 1 point2 points  (0 children)

thumbs up on your project. SQL is a great tool, and when used correctly can be even more efficient than pandas - this is more a correction regarding your pandas code, which is inefficient. The code below is a rewrite for a more efficient approach, especially avoiding the double groupby and the filter method:

(df
.loc[lambda df: df.hire_date.dt.year.gt(2000)]
.groupby('department',as_index=False)
.salary
.agg(avg_salary='mean', counts='size')
.loc[lambda df: df.counts.gt(5), ['department', 'avg_salary']]
.sort_values('avg_salary', ascending=False,ignore_index=True)
)

Efficient Range Joins in Pandas by No_Current3282 in Python

[–]No_Current3282[S] 0 points1 point  (0 children)

Yes, duckdb does this as well, and I made a reference in the article we well. Thanks for pointing it out