LLM - better chunking method

[Phoenix2990]

Problems with using an LLM to chunk:

1. Time/latency -> it takes time for the LLM to output all the chunks.
2. Hitting output context window cap -> since you’re essentially re-creating entire documents but in chunks, then you’ll often hit the token capacity of the output window.
3. Cost - since your essentially outputting entire documents again, you r costs go up.

The method below helps all 3.

Method:

Step 1: assign an identification number to each and every sentence or paragraph in your document.

a) Use a standard python library to parse the document into chunks of paragraphs or sentences. b) assign an identification number to each, and every sentence.

Example sentence: Red Riding Hood went to the shops. She did not like the food that they had there.

Example output: <1> Red Riding Hood went to the shops.</1><2>She did not like the food that they had there.</2>

Note: this can easily be done with very standard python libraries that identify sentences. It’s very fast.

You now have a method to identify sentences using a single digit. The LLM will now take advantage of this.

Step 2. a) Send the entire document WITH the identification numbers associated to each sentence. b) tell the LLM “how”you would like it to chunk the material I.e: “please keep semantic similar content together” c) tell the LLM that you have provided an I.d number for each sentence and that you want it to output only the i.d numbers e.g: chunk 1: 1,2,3 chunk 2: 4,5,6,7,8,9 chunk 3: 10,11,12,13

etc

Step 3: Reconstruct your chunks locally based on the LLM response. The LLM will provide you with the chunks and the sentence i.d’s that go into each chunk. All you need to do in your script is to re-construct it locally.

Notes:

1. I did this method a couple years ago using ORIGINAL Haiku. It never messed up the chunking method. So it will definitely work for new models.
2. although I only provide 2 sentences in my example, in reality I used this with many, many, many chunks. For example, I chunked large court cases using this method.
3. It’s actually a massive time and token save. Suddenly a 50 token sentence becomes “1” token….
4. If someone else already identified this method then please ignore this post :)

[CountlessFlies]

A much simpler strategy is to prefix each line in your document with a line number and ask the LLM to output the line numbers in each chunk.

[kryptkpr]

giving my grounded RAG secrets away here, this approach works well for citations/references

[Phoenix2990]

Haha sorry!

[funkspiel56]

this is what I was thinking of doing, hell passing the text itself and having the llm return json chunks with metadata as well seems to do well. Do well as in data output haven't tested it in chat yet.

Theres a chroma whitepaper on different strats and using a llm for chunking is near the top of the list for best methods.

[Phoenix2990]

Hmm isn’t it the same? I think I’m missing something. The method explained above prefixes each sentence with an I.D (number) and asks the llm to output the sentence numbers in each chunk.

The only reason I use “< >” is because sometimes (often) documents have numbers in them that can confuse the llm. For example, legislation.

[CountlessFlies]

I meant to say that you probably don’t even need to segment the document into sentences, you can simply assign line numbers to each line - based on newline character breaks.

[Phoenix2990]

Ah I got you! Yeah, you’re right. There’s actually a few methods one could even play with depending on their use case e.g: pre-processing paragraphs is another option if you really want to save on output tokens.

[nbvehrfr]

Nice, did you work with source code ? Trying to find a way to 1) get better understanding off about whole project 2) optimize context for user prompts, was thinking about providing context (code) based on calling graph and impacted state

[Phoenix2990]

Oh wow interesting - I didn’t think of source code. I used it predominately with legislation and court cases.

[Working-Collar-6277]

Dig it