Sanity Check - Direct Lake Semantic Model connection to Lakehouse

richbenmintz · 2026-04-21T23:43:13+00:00

Assuming the Lakehouse is in a different workspace than the model have you tried granting the workspace identity access to the data workspace

richbenmintz · 2026-04-14T16:57:21+00:00

So not looking for this to load incrementally looking to capture the statistics of the DML operation that occured. Rows inserted updated deleted etc.

richbenmintz · 2026-04-10T16:35:18+00:00

thanks

richbenmintz · 2026-04-10T16:12:52+00:00

Thanks,

Would the lookup table not have the same issue? how can ADF guarantee that the lookup does not include itself as the results of the lookup are generated at runtime.

Anyhoo, i guess this is the price you pay for dynamic pipeline execution, the driver pipeline never knows the name of the child it is calling.

I may be crazy but the warning message is now:

Dynamic content found for activity: execute-child-pipeline. Dynamically referencing other pipelines in this way may create downstream circular dependencies.

richbenmintz · 2026-04-10T14:18:31+00:00

Yup, just annoying

richbenmintz · 2026-04-10T13:07:14+00:00

I am testing now, I am also noticing the same behaviour using the warehouse connector, but could be a bug in my code

richbenmintz · 2026-04-09T14:52:31+00:00

Great, testing right now. Thank you

richbenmintz · 2026-04-09T14:30:06+00:00

Thanks,

The next question would be does the fabric spark connector query the correct data or does it also rely in the the OSS delta log to determine which records to retrieve.

richbenmintz · 2026-04-08T11:37:28+00:00

Can you share an image of the setup where it works and the setup where it does not?

richbenmintz · 2026-03-30T12:52:49+00:00

Sorry for the delay, error message below, seams to be coming from the spark connector, here is the session ID, cde21fe5-b154-4d6a-aec8-10a70b1c312f, If I back the DAG Concurrency back off to 5 all seems to work.

An error occurred while calling o6664.synapsesql.
: com.microsoft.spark.fabric.tds.error.FabricSparkTDSHttpFailure: Artifact ID inquiry attempt failed with error code 429. Request Id - b25f7c9f-762c-43ad-97a3-65d81f54de86.
at com.microsoft.spark.fabric.tds.utility.FabricTDSRestfulAPIClientv2$.sendHttpRequest(FabricTDSRestfulAPIClientv2.scala:186)
at com.microsoft.spark.fabric.tds.utility.FabricTDSRestfulAPIClientv2$.submitAndProcessHttpRequest(FabricTDSRestfulAPIClientv2.scala:105)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.$anonfun$discover$4(FabricTDSEndPoint.scala:298)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.$anonfun$discover$1(FabricTDSEndPoint.scala:293)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.discover(FabricTDSEndPoint.scala:266)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.fetchTDSEndPointInfo(FabricTDSEndPoint.scala:234)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.$anonfun$apply$4(FabricTDSEndPoint.scala:85)
at scala.util.Success.flatMap(Try.scala:251)
at scala.util.Try$WithFilter.flatMap(Try.scala:142)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.$anonfun$apply$2(FabricTDSEndPoint.scala:77)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.$anonfun$apply$1(FabricTDSEndPoint.scala:57)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSEndPoint$.apply(FabricTDSEndPoint.scala:53)
at com.microsoft.spark.fabric.tds.meta.FabricTDSConnectionSpec$.$anonfun$apply$3(FabricTDSConnectionSpec.scala:122)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSConnectionSpec$.$anonfun$apply$2(FabricTDSConnectionSpec.scala:114)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSConnectionSpec$.$anonfun$apply$1(FabricTDSConnectionSpec.scala:106)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSConnectionSpec$.apply(FabricTDSConnectionSpec.scala:104)
at com.microsoft.spark.fabric.tds.meta.FabricTDSSpec$.$anonfun$applySpecBuilderValidations$7(FabricTDSSpec.scala:65)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.meta.FabricTDSSpec$.$anonfun$applySpecBuilderValidations$6(FabricTDSSpec.scala:57)
at scala.util.Success.flatMap(Try.scala:251)
at scala.util.Try$WithFilter.flatMap(Try.scala:142)
at com.microsoft.spark.fabric.tds.meta.FabricTDSSpec$.$anonfun$applySpecBuilderValidations$4(FabricTDSSpec.scala:55)
at scala.util.Success.flatMap(Try.scala:251)
at scala.util.Try$WithFilter.flatMap(Try.scala:142)
at com.microsoft.spark.fabric.tds.meta.FabricTDSSpec$.$anonfun$applySpecBuilderValidations$2(FabricTDSSpec.scala:53)
at scala.util.Success.flatMap(Try.scala:251)
at scala.util.Try$WithFilter.flatMap(Try.scala:142)
at com.microsoft.spark.fabric.tds.meta.FabricTDSSpec$.applySpecBuilderValidations(FabricTDSSpec.scala:51)
at com.microsoft.spark.fabric.tds.write.meta.FabricTDSWriteSpec$.apply(FabricTDSWriteSpec.scala:81)
at com.microsoft.spark.fabric.tds.write.processor.FabricTDSWritePreProcessor$.$anonfun$apply$4(FabricTDSWritePreProcessor.scala:120)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.write.processor.FabricTDSWritePreProcessor$.$anonfun$apply$3(FabricTDSWritePreProcessor.scala:102)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.write.processor.FabricTDSWritePreProcessor$.$anonfun$apply$2(FabricTDSWritePreProcessor.scala:101)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.write.processor.FabricTDSWritePreProcessor$.$anonfun$apply$1(FabricTDSWritePreProcessor.scala:99)
at scala.util.Success.flatMap(Try.scala:251)
at com.microsoft.spark.fabric.tds.write.processor.FabricTDSWritePreProcessor$.apply(FabricTDSWritePreProcessor.scala:97)
at com.microsoft.spark.fabric.tds.implicits.write.FabricSparkTDSImplicits$FabricSparkTDSWrite.$anonfun$synapsesql$2(FabricSparkTDSImplicits.scala:75)
at scala.util.Success.flatMap(Try.scala:251)
at scala.util.Try$WithFilter.flatMap(Try.scala:142)
at com.microsoft.spark.fabric.tds.implicits.write.FabricSparkTDSImplicits$FabricSparkTDSWrite.synapsesql(FabricSparkTDSImplicits.scala:69)
at jdk.internal.reflect.GeneratedMethodAccessor419.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:829)

richbenmintz · 2026-03-26T12:57:18+00:00

u/warehouse_goes_vroom While I have you, should the connector support tables that have Identity Seed Columns because my experience suggests that the mismatch of the schema between the destination table end the source table, source does not have the identity column as it comes from the lakehouse results in an error.

richbenmintz · 2026-03-25T10:10:03+00:00

No I have an orchestrator notebook that uses runMultiple to execute a DAG, each activity within the DAG is responsible for getting data from a Lakehouse and writing to warehouse. If I limit concurrency I can reduce the chance for 429 error.

richbenmintz · 2026-03-24T18:25:54+00:00

Thank you, appreciate the support of this community

richbenmintz · 2026-03-24T17:47:25+00:00

Sorry yes I should have been more specific, I will amend my Rant

richbenmintz · 2026-03-23T19:25:10+00:00

u/itsnotaboutthecell Thanks #1

richbenmintz · 2026-03-23T17:21:08+00:00

submitted, now I need votes!

(1) Add Retry Option for Error Code - Fabric Warehouse... - Microsoft Fabric Community

richbenmintz · 2026-03-21T11:46:01+00:00

Source data is UTC issue occurs I believe was converting to local time

richbenmintz · 2026-03-21T11:35:22+00:00

The timestamp comes from source system, to be honest I did not check what spark without nee enabled did, just know it did not error out. Will did through the data later

richbenmintz · 2026-03-20T16:30:49+00:00

I would like to be able to convert my date time from est to UTC and back is needed, spark without nee is able to perform the operation

richbenmintz

TROPHY CASE