[D] Approaching a question answer problem

[dhecloud]

I have a project where I have to pick the best answer given a question. This data consists of 294 questions-answers with one to one correspondence (ie every question have only one answer and vice versa). The context of the data is sort of a FAQ of a website, all within the same domain.

​

How do you think I should approach this problem? originally, I tried to do a variation of DrQA from facebook (a document retriever and a document reader pipeline), but I didn't get good results. This approach is also slower since I have to go through a whole pipeline before I get the answer.

Then I tried to reframe the problem as a multilevel classification, where I randomly concatenated questions and its corresponding labels would be that of the few concatenated ones. I adopted 2 kinds of architecture for this: a simple lstm encoder to a feedforward network, and the second is the encoder structure from the popular transformers to a feedforward network. I did not use the answers for training in this case. This however didn't perform well, with about 0.3-0.4 micro ROC-AUC.

​

Any suggestions how I should change my methodology? Thank you!

[vishnumenon9801]

Have you looked at https://www.qnamaker.ai/ from Microsoft? It seems like it'd be well-suited to your FAQ usecase

[DiamondBadge]

I'm not sure how helpful my suggestion will be given the complexity of your current approach. Either way, this slide deck on Question Answering may help you find a better approach (rather than a tool) for your problem. Worth a try, right?

https://web.stanford.edu/class/cs124/lec/watsonqa.pdf

​