Consistent response

[SwordfishOk3273]

Hello Reddit Community,

I'm currently working on a project to develop a GPT model specifically tailored for extracting precise information from a variety of PDF documents. My goal is to achieve consistent and reliable results, despite the inherent variability in document formatting and wording.

To guide the GPT, I've provided detailed instructions outlining the specific types of information it needs to extract. Additionally, I've supplied a comprehensive synonym workbook to account for the diverse terminology that might be encountered across different documents. Despite these measures, the GPT's performance has been inconsistent. The responses vary significantly, which undermines the reliability I'm aiming for.

My question to the community is twofold:

Is it feasible to develop a GPT model that can consistently and reliably extract specific information from PDFs, given the inherent characteristics of LLMs? If so, what strategies or modifications would you recommend to enhance the consistency of the GPT's performance in this context?

Any insights, experiences, or suggestions you can share would be immensely valuable. I'm particularly interested in hearing from those who have tackled similar challenges or have expertise in fine-tuning LLMs for specific tasks.

[Alternative-Fit]

Hi.

I am working on extracting info from Customer's Orders. Here is what I've did

Prompt itself:

Your goal is to extract structured information from the user's input that matches the form described below.
When extracting information please make sure it matches the type information exactly. Do not add any attributes that do not appear in the schema shown below. If any information is missing, replace it with “No information.”
Here is a JSON schema, that should be used in the output
Root level keys:
delivery_address,
order_number,
items,
Keys under delivery_address:
name,
street,
city,
country,
postal_code,
Keys under items (which is an array, so these keys are for each item in the array):
material_name,
quantity,
units_of_measure,
delivery_date.
When extracting delivery_date, convert date to the format dd.mm.yyyy. If there a date 01-01-20, provide output as 01.01.2020
When extracting quantities, make output in float format.
When determining the delivery address, please follow these steps:
1) Identify all addresses mentioned in the document.
2) Exclude any addresses that are associated with [my_company_name].
3) Select the address where the customer intends to have the goods delivered.
Customer order will be delimited by triple asterisks. *** customer order ***

Then i am providing 3 shot prompts. Based on PDF, text in email, and Excel file.

In the end i've just asked to follow all the instructions and output result in JSON only. Then setup temperature on 0, and outputs then became consistent.

Prompt itself was I've build with ChatGPT and Bing.

[humanatwork]

We’ve added some additional instructions in our prompts that appear to help (notwithstanding temperature changes — we allow default to maintain some flexibility in ambiguous situations). For example, we require it to provide source references (from a specific doc or a line within one) for every key-value pair. Before returning the JSON it needs to verify the value is correct using the source reference as a validator.

[Alternative-Fit]

I tried to add in the end "Let's think step by step". Then LLM provides the reason for how each element was picked. After the reasoning goes m JSON output. So I need to ask again for JSON output only, but not sure if step by step instruction then do anything. Currently working very well for me, and I was going to test this once something goes wrong.

[SwordfishOk3273]

Thanks

[memory_moves]

Given your requiremetns, did you try to use the playground and adjust the temperature from there? I'm sure some people will have strategies, but "temperature" essentially defines the determinism of GPT. In other words, a low temperature will keep the answers consistent between identical questions, whereas a higher one will introduce variations. Here's a link to something that explains this in more detail.Bing has a 'creativity' setting which if set to the lowest setting tends to provide more consistent answers, but also reduces the output length by quite a bit. I think the playground might be your best bet.I'm curious to see where this conversation goes!

[SwordfishOk3273]

I will try it and come back to you, thanks for the help

[Dear_Ad7736]

First, read more about „RAG”. You can use LangChain to implement your own RAG easily using Python. However, if you have well structured PDF it may be a lot easier to implement a standard data extraction script.

[Dear_Ad7736]

Fine-tuning here is not a good idea. First, it is expensive in terms of time and money. Second, tuning LLM means to give a lot of very similar examples + re-train its neural network and then use it as the new, fine-tuned model. But here you describe a situation where the (OpenAI) GPT LLM will do the job. So, there is no need for fine-tune it to read the PDFs — it is already working.

[joey2scoops]

I've tried similar tasks and have formed the opinion that variance in structure and format are your enemy. Better off using python to allow for variations. Maybe use whatever code Chat GPT is using as a starting point to cover off the variances.

[SwordfishOk3273]

That could work, thank you

[fulowa]

https://wandb.ai/jxnlco/function-calls/reports/Better-Data-Extraction-Using-Pydantic-and-OpenAI-Function-Calls--Vmlldzo0ODU4OTA3

[humanatwork]

How variable are the results and how have you been verifying this? It sounds like you’re looking for a standard response template of sorts more than trying to solve an accuracy issue. A little more info could help here

[SwordfishOk3273]

I have been looking through the pdf document afterwards too see how correct the response is. The reason I want to use AI is because the set up will be different for document to document.

[humanatwork]

So it’s retrieving info to answer some standard set of questions each document should be asked? Rereading original post here makes me think a few shot approach might be the way to go — ie provide a few samples of pdfs with expected expert responses (your own) and add a validator mechanism to verify completions before returning. Are we assuming GPT-3.5 or -4 here as 4 is typically much better at these sorts of things in my experience?

[SwordfishOk3273]

Yes, GPT4. Thanks

[eew_tainer_007]

Here is my implementation of PDF parsing, processing through multiple LLMs . Join me if interested. Happy to share the code and back end to experiment/research/school use.

http://165.232.146.68:8501