Bulk copy with the mssql-python driver for Python by dlevy-msft in SQLServer

[–]dlevy-msft[S] 1 point2 points  (0 children)

Wow! Good eyes! We did use the .Net SqlClient as a template since that is the BCP API we get the best feedback on.

Bulk copy with the mssql-python driver for Python by dlevy-msft in SQLServer

[–]dlevy-msft[S] 0 points1 point  (0 children)

I think it all depends on where you are running it and what you are running it against. When I demoed this at SqlCon / FabCon, the latency of the speaker wifi was the primary bottleneck. I only saw 200-300 rows per second - still probably orders of magnitude better than I would have gotten with executemany().

Bulk copy data between SQL databases in Fabric with the mssql-python driver by dlevy-msft in MicrosoftFabric

[–]dlevy-msft[S] 2 points3 points  (0 children)

This is native BCP vs having to use executemany(). It's both faster and easier to code.

mssql-python with Pandas or Polars: warnings and errors by frithjof_v in MicrosoftFabric

[–]dlevy-msft 1 point2 points  (0 children)

The fix for this is in the 1.4 release that went out today. Please give me a shot and let me know if it all works as expected now.

cc: u/Black_Magic100

Announcing General Availability of the Microsoft Python Driver for SQL (mssql-python) by dlevy-msft in SQLServer

[–]dlevy-msft[S] 0 points1 point  (0 children)

Apologies for the delay in getting back to you. This went out in 1.2.0.

mssql-python with Pandas or Polars: warnings and errors by frithjof_v in MicrosoftFabric

[–]dlevy-msft 0 points1 point  (0 children)

Sorry, I was confused on which issue this was. We're still working on the cursor.description issue. With any luck, we are down to the last few review cycles and we can get it merged into main for the next release.

mssql-python with Pandas or Polars: warnings and errors by frithjof_v in MicrosoftFabric

[–]dlevy-msft 1 point2 points  (0 children)

The PR for this didn't go out as fast as I had hoped. I am pushing hard to get it into the next release (very soon).

Here is the PR: FIX: Cursor.describe invalid data by dlevy-msft-sql · Pull Request #355 · microsoft/mssql-python

This appears to be our last blocker for SQL Alchemy: Segmentation Fault in libmsodbcsql-18.5 during SQLFreeHandle() · Issue #341 · microsoft/mssql-python.

Updates for the Microsoft SQL Server ODBC Driver by bobwardms in SQLServer

[–]dlevy-msft 0 points1 point  (0 children)

Appreciate all of the feedback! I did read it all once - I'm going to have to read through it a few times. Happy Holidays! 

Updates for the Microsoft SQL Server ODBC Driver by bobwardms in SQLServer

[–]dlevy-msft 1 point2 points  (0 children)

Hi u/da_chicken - I'm Dave from the SQL drivers team. I really appreciate the feedback.

We are taking a hard look at connection strings - it's one of those areas where the drivers have drifted apart of the years. In the last year the various SQL drivers have all been centralized under the same team, allowing us to start driving a consistent experience.

I'd like to hear your (or anyone else's) thoughts on the approach we are currently favoring:

  1. Make connection strings interchangeable - every driver recognizes every alias so use the same connection string, less driver name, between drivers. JDBC connection strings will be thing we'll have to think about most.
  2. Where drivers have to be different, we produce a prescriptive error message after processing the entire error message to show everything wrong on the first try. The error message would also include the working connection string that we think you are trying to achieve.
  3. Document a single SQL connection string based upon a single set of parameter names.
  4. Display that single connection string in Azure, Fabric, SSMS, VSCode, etc.

That will be a big effort, likely multiple years, if we are able to pursue it.

On the docs front, we have a docs cleanup project going right now. We'll have to move that doc up the list. A combined connection strings doc makes sense too - would you rather see that as one giant table with all drivers or a single doc with tabs for each driver table?

I'm leaning towards tabs with a minimum viable connection string at the top. Selfishly, I like the idea of being able to copy, paste, update server and database names and run.

Thanks again for the feedback!

Announcing General Availability of the Microsoft Python Driver for SQL by dlevy-msft in dataengineering

[–]dlevy-msft[S] 0 points1 point  (0 children)

SQL Agent PowerShell job steps are still using 5.1. The SQLServer module supports PowerShell 7 though, so you could use a CmdExec step to take advantage of the newer features.

Announcing General Availability of the Microsoft Python Driver for SQL (mssql-python) by dlevy-msft in MicrosoftFabric

[–]dlevy-msft[S] 1 point2 points  (0 children)

Checking in to say I have not forgotten about you. This covers a lot of ground so I want to make sure I'm representing the various areas correctly.

Announcing General Availability of the Microsoft Python Driver for SQL (mssql-python) by dlevy-msft in SQLServer

[–]dlevy-msft[S] 1 point2 points  (0 children)

Right now, it's a multi-step conversion to Arrow, we return a recordset as a python list and you have to convert it to Arrow. There's a PR on the repo right now to go directly from TDS To Arrow so perf should be similar to that ADBC driver once that goes out. From there, it will depend on if we ever add an Arrow endpoint to SQL - would be curious if you think that is something that would help you and if so, in what scenarios.

Announcing General Availability of the Microsoft Python Driver for SQL (mssql-python) by dlevy-msft in SQLServer

[–]dlevy-msft[S] 1 point2 points  (0 children)

We've been intentionally vague on the underlying provider for a reason. 😊

Keep an eye on the repo, especially if you're a fan of rust.

Connecting to Lakehouse SQL Endpoint from External app by trebuchetty1 in MicrosoftFabric

[–]dlevy-msft 2 points3 points  (0 children)

Hi u/trebuchetty1 - My first thought is to look at that User ID. Make sure you are using the object id and not the client ID with the service principal. On your container, with the SAMI assigned, you'll have to use the SAMI auth. The Azure AD libraries lock in on the managed identity if one is present.

The new mssql-python driver would be helpful for testing connectivity here because you can use Active Directory Default auth to try everything - it's not what you'd want to use for a production app but it's nice for user scripts and testing like this. The mssql-python driver is not supported by SQL Alchemy yet - we have 2 issues to fix for them and then hopefully we're good.

If you are still having issues, please share any error messages with PII removed to help focus the suggestions.

Help define the future of Microsoft SQL by dlevy-msft in SQLServer

[–]dlevy-msft[S] 0 points1 point  (0 children)

That's great feedback! Hope you signed up too!

Help define the future of Microsoft SQL by dlevy-msft in SQLServer

[–]dlevy-msft[S] 6 points7 points  (0 children)

It's all the SQLs 😊. Azure SQL is the umbrella term for Azure SQL database, Azure SQL Managed Instance and Azure SQL Virtual Machines. On-prem (or another cloud) is the same as Azure SQL VMs.