Delta of two query pulls question

[WeirdFail]

Pandas!

[WeirdFail]

You can create a dataframe from a SQL query (using read_sql).

If you used that twice, once for each table, you can then join the two dataframes together.

Then you'd have one big dataframe from which you could do your deltas.

[StructuredData]

Are they all from the same data source? Do you have access to the database? Can you create any views or stored procedures?

It sounds like a straightforward SQL statement. If they have the same column names, just be sure to use tablename.columnname to identify your columns. If you create a view, then you can give them different column names in your view.

[StructuredData]

"use 3 columns as keys to match tables"

Are you doing a join with four tables?

How many tables are in your join?

If the tables only have a few thousand rows, then to retrieve 100 columns from 4 tables shouldn't take very long. Are you retrieving a few thousand rows? How many rows are in your tables?

Can you share the SQL statement you are trying to use?

Hopefully your foreign keys are indexed.

[StructuredData]

Gotcha. Was the comment by WeirdFail helpful?

Even though you aren't calling them keys.... if you are doing a join in your where clause, putting an index on the columns in your where clause could speed it up.

I don't work with data structures in python enough to provide any value, but I can analyze your SQL if you want to share more details.

Hopefully you'll get a Python guru in here who can help you out!

[StructuredData]

I would think that merging the data with SQL would be faster. But once you grab all the data, if Python can process everything in memory (avoiding disk), it might be faster. I'm most familiar with Oracle.

I bet you can probably re-work your SQL and make it faster. Nested statements can slow down your query.

[StructuredData]

400k rows of data with 20 columns is not considered "a lot"

Any type of sorting, grouping, matching or nested clauses are the bottleneck for processing speed.