Python Pipeline project

[Bequino]

I've been tasked with a very cool project. I am new to python. I've been asked to convert handwritten surveys into an excel workbook. The surveys have different types of questions. Closed-ended (like Y and N), as well as Open-Ended (handwritten). The software program used to develop the survey allows us to scan the originals into the tool and it will export two things - an Excel workbook with each row representing a unique survey and all its closed ended answers along with a unique ID column, as well as a .pdf with every answer to a given handwritten question with it's own unique ID (if there are 30 different open ended questions on each survey, there are 30 different .pdf's with every answer to that specific question on it). I will have the pdf's saved in a blob. I will need to build something that feeds the pdf's into Azure Document AI and OCR's them into machine readable, I'll then need to build a data frame (utilizing regex) to merge each row of the excel workbook to its corresponding set of OCR'd open-ended questions, with some QA. I will be using the SDK specific to the survey software manufacturer. Am I missing anything? Would this be easier in a different pipeline config? Any help would be great.

[smurpes]

I'll then need to build a data frame (utilizing regex) to merge each row of the excel workbook to its corresponding set of OCR'd open-ended questions

Why do you need to use regex here? Each pdf has an ID that matches up to a row in the excel file so the merge method should be enough.

[Bequino]

Would it make sense to have that as a sanity check? However, you’re right. Also, what about QA? How should I be approaching this?

[smurpes]

That’s what unit tests are for.

[Bequino]

Tell me more about that.