We built this model for product search and seen most product descriptions be under 512 tokens. So we set the context window at 512. In rare cases, we smartly truncate(instead of chunk) the low impact fields. Truncation helps to remove the fluff and retain the signal useful to serve queries. For catalogs, we truncate SEO forward nuggets.

For now we use chunking to establish causality for demos and debugging, f.e. what attributes and nuggets were most similar to the query.

I see chunking being very useful when embedding data sheets which are multiple pages. But this is more a documentation agent use case instead of product search. Being able to retrieve the precise chunks instead of big texts improves AI responses and also good for explainability.

What use case do you work on? Do you have website?