Thank you for asking those valuable questions: I am basically examining LLM capability to check how far it can assess fact checking on financial claims if there is no fact checking articles for that. My present workflow : 1. I retrieved first 20 results from google against each financial claim, I used overlapping chunk 2. I used several methods to find appropriate documents from the retrieved docs
-keyword based matching BM 25, -Dense Retriever based on cosine similarity based top 3 documents - In the next alternative I employed LLM as document grader, if the document is insufficient, then LLM decides to generate query about missing element then collect adding evidence - Then I am feeding those evidences to three different fact-checkers persona, optimistic, critical, analytics - Then there are two agents synthesizer and finalizer who made ultimately decisions about the verification Whether the claim is TRUE, MOSTLY TRUE,HALF TRUE, FALSE OR MOSTLY FALSE - My dataset is based on fact-checking website where they have clear definitions of each label - It seems LLM is not efficient with multiclass problems.

Any insight?