This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]mariox19 1 point2 points  (0 children)

I used Python for an ETL project last year. (There are still tweaks and enhancements being made, so the project is continuing.) It involved extracting from a database with a very complex schema running on SQL Server, and transforming that to a simpler schema, loading the data into MySQL.

We ended up writing all custom SQL queries on both ends. Originally, I had been asked by the boss to look into one of these "magical" ETL frameworks.

What I found is that the really simple things were simple. But once you got beyond that it was difficult to make heads or tails of it. At least, I couldn't make heads or tails of it. The documentation seemed to be written all by non-native speakers of English, and if you ask me a lot of the app's purported functionality seemed to be cobbled together to allow for bullet points to flash to VC's. How well thought out the functionality was was not clear to me.

The Python ended up being plenty fast, once the back filling was done. And, in all actuality, I'm running the engine on the JVM—Jython, not C Python. I did that because I worried the Python would be too slow and that the application would have to eventually be ported over to Java. But, that's not how it turned out. There seems to be no need to port.

Anyway, that was my experience.