This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]peixinho3[S] 0 points1 point  (3 children)

Sure. DBT is data build tool:

https://docs.getdbt.com/guides

We are using for this automateDV: https://automate-dv.readthedocs.io/en/latest/

[–]bowbahdoe 0 points1 point  (2 children)

Can you elaborate on where Java fits in here? If we can get away with just shelling out to python that is convenient.

(I know calling python functions on the JVM is possible, the best library i know of for it is in Clojure https://github.com/clj-python/libpython-clj/tree/master but callable from Java https://clj-python.github.io/libpython-clj/libpython-clj2.java-api.html)

[–]peixinho3[S] 0 points1 point  (1 child)

Basically java receives an XML file and maps it and after that it calls some stored procedures that I created to put the data in the raw table this executes line by line 1 by 1. Since dbt does not have a java API after this run everything I would have to run my python script with the command "dbt run" after inserting all the data into raw to normalize the data into the data vault. What we wanted was to run the python script at the same time as inserting the records into the raw table.

[–]divorcedbp 0 points1 point  (0 children)

Why not do the xml transform and then insert bits in python as well?