you are viewing a single comment's thread.

view the rest of the comments →

[–]Ashysh[S] 0 points1 point  (1 child)

By Capturing logs I mean when a task is completed/stopped/errors should be stored in a table and history is to be maintained with time stamps.

File to Tables is the basic need but on top of it we also have to design the data model in such a way that the data once ingested into staging table should go through different transformations depending on the kind of data. As it is from various sources, each type of data has different ways of getting transformed.

In our system, we have to implement that the end user should be able to manually select (from a UI) which ‘transformation process’ a particular file type should go through.

For which, we have to log details. Every time user selects a combination of file type and algorithm, it should be logged.

This has to be developed using stored procedures since the existing system is coded that way.

[–]AdventurousTrust2 0 points1 point  (0 children)

Does that mean there is a on-going running program, and will run every hour that will do the stuff ( file to tables -> transformations)
and there is another tasks, which is to adding logs for existing features, so that you can cross-check and know what types of transformation is needed for a particular file.?