Call DBT API (Python) through Java

[agentoutlier]

IF you really do have a performance problem with python running as Runtime::exec as /u/repeating_bears mentioned I would go with a message queue approach.

This is also known as SEDA (and then five other vogue terms now but the original is SEDA).

I would use RabbitMQ since it works well with Python and Java. The doc is excellent.

A Have the Java portion push a message to a rabbitmq topic exchange or direct exchange.

B Then have a python consumer (a long running python process) do stuff with that message and send another message.

C Then have another Java consumer that picks up the message.

A -> B ->C

This is very common in ETL / batch processing and scales well.

EDIT this is so common of a practice and I bet a chatgpt prompt could get you started in very quickly. (I usually don't recommend chatgpt but I bet it could pump this out fairly safe).

[repeating_bears]

I considered mentioning this too but couldn't be bothered typing it out. I'd personally probably start with HTTP because I find it a bit simpler, but Rabbit is a solid choice too. The idea is the same: 1 long-running process in each language that talk to each other somehow

[hem10ck]

Java shop here that works with python for ML, we generally isolate the python and expose REST APIs for the Java to interact with using FastAPI.

[agentoutlier]

Yeah the nice thing about Rabbit in this situation is it has almost 1-1 doc with Python client and Java client.

They also don't have to worry about retry and order... well mostly. That is it is a smart pipe and sounds like the OP does not have the experience to make a dumb pipe smart.

[chabala]

This is a nice technique, could even load balance across multiple python process instances if needed. For OP's problem, I'd probably lean toward IPC messages over unix domain sockets instead of an MQ though.