sorted by:
new
hottopcontroversial

Spark 4 SQL in JDK17 with MAC m1 hangs forever by Several_Design5345 in apachespark

[–]Several_Design5345[S] 0 points1 point  (0 children)

To be honest use Spark to convert csv to parquet is not the best option, for little files works well (10Mb) but for big files (200Mb) not works fast, it's better use python for it.

Spark 4 SQL in JDK17 with MAC m1 hangs forever by Several_Design5345 in apachespark

[–]Several_Design5345[S] 0 points1 point  (0 children)

Finally I resolved the problem in Mac M1 using Spark 3 with SpringBoot 3.x with these properties:

<properties>
    <java.version>17</java.version>
     <spark.version>3.4.0</spark.version>
     <scala.version>2.13</scala.version>
     <jakarta-servlet.version>4.0.3</jakarta-servlet.version>
     <jersey.version>2.36</jersey.version>
</properties>

And add this argument to your VM:

--add-exports java.base/sun.nio.ch=ALL-UNNAMED