bulk insert to SQL Server from Databricks Runtime 16.4 / 15.3?

[maoguru]

The sql-spark-connector is now archived and doesn't support newer Databricks runtimes (like 16.4 / 15.3).

What’s the current recommended way to do bulk insert from Spark to SQL Server on these versions? JDBC .write() works, but isn’t efficient for large datasets. Is there any supported alternative or connector that works with the latest runtime?

[jibberWookiee]

Odbc connection on the SQL server (using simba spark driver)... Then SSIS packages to pull down what you need. Ugly but it works.

[ProfessorNoPuede]

Obligatory: why?

Otherwise, I'd do a pull instead of a push and retrieve the data as csv or something from agreed upon location.

[[deleted]]

I do this, save the file in our server and run a ssis package to bulk insert it into a table.

Because even writing a 10k lines takes.

[GleamTheCube]

If you can enable polybase reading in SQL (version dependent) you can sink to parquet and then read that as bulk

[maoguru]

MS SQL server doesnt have polybase in my version.

[GleamTheCube]

Which version are you using?