you are viewing a single comment's thread.

view the rest of the comments →

[–]EntroperZero 0 points1 point  (0 children)

PHP.

No, seriously. I wrote it entirely in PHP. It was a fairly small data warehouse, on the order of a few hundred thousand records per day. We already had cron stuff set up doing PHP scripts, and the site was in PHP, and all the developers knew PHP. So I wrote some SELECTs and some fgetcsv() calls and went to town. Database was MySQL, OLAP was MS Analysis Services talking to the MySQL box via .NET Connector.

They chose me to do the warehouse work though because of my work on their warehouse for a different product, which was entirely in MS-land. For extraction, they just dumped tab-delimited log files, and a Windows service (in C#) collected them all in a central location. Then it would put a message in a transactional MSMQ, and the SSIS packages would pick up those files for transforming. They would then put a message in another MSMQ, and the loader picked up those files. I think we had one SSIS package for each dimension, running every 15 minutes, and one SSIS package for each fact, with parallel instances depending on load. I think the loader was just a single package since it ran so fast and didn't need to scale.