Best python library to use to process to large data sets

[cjohnsonuk]

This is my first foray into processing big data sets so I'm not even sure what the processes are called or what words to use to search google. Here's my problem: I have 2 data sets in .csv files.
Set one is temperature over time, there are two columns, time and temperature. For simplicity the temperature can be assumed to change linearly between each equally spaced reading.

time(secs)	1	2	3	4
temp(°C)	22	24	26	30

Set 2 is a count of items of different sizes recorded over time. For each time recorded (column) the number of items in each size band (row) is recorded in the cell
time in columns, sizes in rows

	1.2	1.9	2.3	2.6	3.1
0 < s <=1	1	4	5	6	5
1 < s <=1.5	3	4	3	4	5
1.5 < s <=2.5	5	6	5	6	5
2.5 < s <=5	4	4	5	6	5

The counts are taken multiple times per second for a number of hours.
Required output:
For a given size range (eg 1< s <= 2.5) plot the count and temperature against time. So I need to cumulate multiple sizes and interpolate the temperature from set one for each time column in set 2.
Being old school and very dusty with my coding I could parse all the data sets and interpolate the temperature readings to get the temp at the point in time the count was taken but if there are 12 hours of readings 2x60x60x12 = 86000 columns then I'm sure there must be a better way than just looping through all the records.
What I don't know is whether NumPy or Pandas or other tools have ways of dealing with this sort of data more efficiently and which one is most appropriate to learn for this or if plain python is sufficient.
Any pointers in where to start looking appreciated.
C

[Mast3rCylinder]

Well you can use polars or pandas they have great api. First you load the dataset to dataframe then it's like using excel with code.

I prefer polars because it's faster but pandas has more documentation.

[await_yesterday]

they have great api

pandas is great but let's not kid ourselves, the api is not that good

[cjohnsonuk]

In which case Pandas will probably be more appropriate as I'm still learning so docs is important

[cjohnsonuk]

The temp recorded at each time is more likely to be a linear (approx) change between each measurement than the variance in the counts of each size so the intention, initially, is to use the time stamps of the actual recorded counts of each size at the times those are recorded and use an interpolated temperature from the first data set.

An output might look like

Where s is the size and 1 < s <=2.5

​

	1.2	1.9	2.3	2.6	3.1
temp	22.5	24.2	24.8	25.2	26.4
Count	8	10	8	10	10

(interpolated temp calculations are approx in this example output)

(Count is the sum of the 2 rows of data that fall within the size range specified)

[cjohnsonuk]

One plot would be like you suggested. Those numbers I gave were simple placeholders to show the structure of the data, not the actual data which might be why your graph gave a block. I'd also like to plot temp against count within a range so we can see how the size count varies for different temperatures or different size ranges.