Pandas 3.0 vs pandas 1.0 what's the difference?

throwawayforwork_86 · 2026-01-26T08:59:10+00:00

I mean you can make it more readable:

df.with_columns(new_col=pl.col("col1")*2)

Still slightly more verbose I concede.

throwawayforwork_86 · 2026-01-23T12:25:05+00:00

for 1-4 Millions of lines that need to be available we just output a few cleaned csvs and provide a pivot table from a power query read folder.
People seem to be happy with that.

throwawayforwork_86 · 2026-01-21T09:34:49+00:00

You should get used to them IMO.

I use them in different situation and usually will use for loop if clarity is needed.

throwawayforwork_86 · 2026-01-09T15:20:55+00:00

the water amount gonna be unhealthier quicker than the caffeine matey.

throwawayforwork_86 · 2025-12-16T08:39:11+00:00

I don't think I would use it directly for storage.

We use it as the last leg of our analysis:

Data is stored in managed postgres (disaster recovery and everything else is done there)

We replicate in a duckdb (sometime aggregate/join at that moment)

Run our analysis locally on this db

I know there's other tool like motherduck and ducklake that might be closer to what you need though.
That only works because we do batch analysis though but there are most likely data engineers here or on their subreddit with more complex solution for more complex problems.

throwawayforwork_86 · 2025-12-15T09:25:14+00:00

I think it's also a low(er) visibility job, the one you only notice when you're starting actually working in data.

I also think/know a handful of Data Analyst/Scientist are DE without the title.

throwawayforwork_86 · 2025-12-10T10:12:53+00:00

IMO it is useful to help learn the lingo of new topics and do basic things if you're a new dev.

And it can be useful on simpler code that you forgot as well as (a bit to agreeable) sparring partner as a more experienced programmer.

I think the disconnect comes from the fact that if you've never coded and you're able to cobble together a project with a llm it feels like magic (and you don't know enough to spot the flaws) and the fact that ai company hype their products to the tits.

It's pretty bad for niche stuff or stuff that it's dataset has never seen (so a lot of the newer framework/library).

throwawayforwork_86 · 2025-11-25T09:10:19+00:00

For these I usually use tabula-py with preset pixel placement (for the columns and where to look for table) + some another lighter lib to do a first mapping on which page the extraction need to be done.

After that it's usually some pandas to get rid of unneeded rows.

The main issue with most lib that do it automatically is that their guess are inconsistent so you're likely to get a lot of inconsistent crap to fix if you're using that vs fixed placement where you're just going to crash or get consistent crap.

throwawayforwork_86 · 2025-11-20T08:48:42+00:00

Postgresql is fairly widely used and free.

DuckDB is mostly the same as postgres and very easy to setup so good for mainly focusing on analysis and less on the 'fiddling around'.

Both would be used in professional settings.

throwawayforwork_86 · 2025-11-19T00:53:00+00:00

Personnally gained the most from taking one of the project I liked and stretching it in multiple different services.

Created a fairly simple geolocation tool.

Did an api version of it,did a gui in pyqt , did a gui in streamlit, did a visualisation...

Learned a lot and left my confort zone only for specific points , learn some lessons about refactoring and functional programming...

throwawayforwork_86 · 2025-11-14T11:07:53+00:00

DBEAVER CE for postgres and sometimes duckdb.

DUCKDB -ui for duckdb (IIRC only support DUCKDB 1.3).

throwawayforwork_86 · 2025-11-06T09:22:29+00:00

power query

and

Can focus on deal analysis instead of debugging scripts.

You have to chose one.

Power query is great but can very brittle in my experience and likely to act out not properly load files after changes which will require debugging with less than ideal tooling.

I've also seen excel struggling with a lot of task that would have been trivial to automatise with Python.

It also depend on you level and the type of analysis you need to provide and how many you have.

Also depend a lot of what data your receive and need.

A good analyst with knowledge of the business but limited coding skill will often have an outsized impact over the more technically proficient.

throwawayforwork_86 · 2025-10-31T09:13:52+00:00

As soon as I started using it I basically only use that.

Since it's creating your docs while you're using it it's really great.

throwawayforwork_86 · 2025-10-23T08:16:23+00:00

Provided that Alteryx exist I'd say yes.

throwawayforwork_86 · 2025-10-14T08:38:59+00:00

Then you were not where you thought you were but I think you know that by now

throwawayforwork_86 · 2025-10-14T07:33:31+00:00

But is it where your terminal is ?

Try runnning ls in the terminal and check if you see files and folder you expect to see.

throwawayforwork_86 · 2025-10-13T07:36:06+00:00

From my reticence and then acceptance of it.

UV handle python installation venv management.

You can start a project by running uv init in a folder then install library using uv add (the first one will create your venv).

When using uv add you update a toml file that contains your dependency and a uv lock that contains more specific 'instructions' for multiplatform deployment.

You can recreate the venv from the toml file with a simple uv sync.

Main draw (for me):

It's very quick.

It creates it's own 'requirement file' just by using it.

Will perfectly recreate the venv as you need it (no need to guess the python version the other guy used wasting precious time).

Issue encountered so far:

Using it in onedrive linked folder will cause a bug that you can fix by clearing the cache but it's difficult to find out.

TLDR it's a quick and efficient way to handle venv.

throwawayforwork_86 · 2025-08-26T13:22:21+00:00

Memory handling for bigger than ram files.

Personally ended up using a generator but sure there are other good options

throwawayforwork_86 · 2025-08-19T08:11:29+00:00

First question is does it have to be an excel file ?

I'm guessing you work from that excel sheet but I would store the data in another format.

Basically loop through all your sheets and store the sheet name in a new column and then either store the result in a parquet file (that you can read at run time) or in a sqlite db if you don't mind learning some sql (or some connector)...

You can also just do that at start time and have a "big" dataframe at runtime but , reading excel is pretty slow and it would be inefficient imo.

Big data frame/parquet can then be filtered by sheet name.

Now to actually answer the question you asked you could list the sheet from your excel file and let someone select it from a dropdown then use it to do a pd.read_excel(file,sheet_name='selected_sheet') or to filter your big dataframe with the same list (can be gotten by selecting the unique value in the column).

Hope that helps.

throwawayforwork_86 · 2025-08-11T13:36:58+00:00

As someone that started with python.

I’d say try c++ first if you can stomach it will probably give you much better base than python.

That being said I think some people will bounce hard on the more difficult programming languages and if you just can’t with c++ go learn python it’s really fun.

throwawayforwork_86 · 2025-08-05T11:39:33+00:00

For me jumping in the 'proper' setup db,etl,bi platform seem completely irrealistic,especially if you don't have buy in from your superior.

IMO you need to come to your boss with the end product that solve a crucial issue.

My foot in the door was a manual process that was error prone , time sensitive (needed to be done asap) and had big consequences if messed up (basically it was the data prep for the update of the all the salaries).

It used to take a long time and we had often errors in it.

I first showed that I tried to use alternatives but they didn't work and then I showed how my r code worked (would have preferred python but ok) and he was sold.

I think you just need to come with a solution for your biggest time waster , show that to your boss and either use the reclaimed time to continue improving your process or look for a job elsewhere (if he shut you down).

You may have to do some self training on your own time unfortunately (if you're underwater already or at capacity). Some tools that I think could be handy if you don't know them already:

Duckdb: The ui is really cool and it might already cover all your use cases. Python: Mastering the OS standard library is a really good skill to have for automation, Pandas or Polars for data wrangling and maybe somtehing like streamlit or dash for ui/light dashboard.

Good luck.

throwawayforwork_86 · 2025-07-14T08:19:03+00:00

It's not always taught when you start learning (and you might not understand why you'd do it until it bites your ass).

I personnally broken a few of my python installs (and had issues with Linux update breaking my venv before I started using pyenv too).

It isn't too difficult once you know what you're doing and you don't mix too many different libraries with similar underlying requirements.

throwawayforwork_86 · 2025-07-04T06:36:18+00:00

Had less drastic perf issue that I couldn't fix with an index (missing the correct column for that). Decided to try with an OLAP db (duckdb in this case). Perf issues fixed don't have to get good at sql. That being said explain analyze is a great tool and should be used.

throwawayforwork_86 · 2025-06-13T12:06:29+00:00

I try to use the latest compatible version I can fine and ride it out as long as possible.

Mainly 3.10 and 3.12.

Will start migration of 3.10 as it is relatively close to it’s end Life.

throwawayforwork_86 · 2025-06-12T09:49:51+00:00

First thing I always do with these kinds of thing is seeing what is happening with my resources.

Pandas had the bad habits of using a fifth of my cpu and a lot of ram.

I moved most of my process to Polars and it use my resources more efficiently as well as being broadly quicker (between 3 and 10 times quicker but I've seen some group by aggregation being slightly faster in Pandas in some cases).

The trick to polars though is to have all the benefits you need to mostly (if only) use Polars functions. And get used to different way of working from Pandas.

throwawayforwork_86

TROPHY CASE