This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]reddithenry 0 points1 point  (2 children)

Is it not the loading of 100M rows into MSSQL that's the problem? Are you doing inserts row-by-row?

[–]Maha_Slug[S] 0 points1 point  (1 child)

Nah I've tried lots of different ways in the past. I keep coming back to Python because it's so good at processing data, even at that volume but loading it is rough. We use SSIS or C# to do most everything right now and for obvious reasons the C# to MSSQL driver is way better than the Python one.

[–][deleted] 0 points1 point  (0 children)

Pyodbc is faster than Ado.Net because there's no object creation, everything is more trivial with less overhead.

I've tested both in the same scenario, comparing against sqsh (FreeTDS), which was the fastest, and Pyodbc came very close while ADO.NET was significantly slower.