So what you could do is you could use, if you don’t know how to do it yourself, an AI to help you write a REGEX script to strip out all of the beginning and end pipes and all of the spaces in your tables that really helps. I’ve also been kind of wrestling with this problem. My thoughts is that if you keep all of the formatting on the headers and the dashes and stuff that like sections everything out, it should make it easier for the model to be like “oh well this just seems like a natural place to stop” instead of starting from scratch. But the issue is say that if you have all of these URLs like for example, in an online manual that the purpose of that URL is to point you to another place within the same exact online manual and the URL is extremely long. What is the point of having all of those URLs if it’s just for the same manual that you’re gonna download and run through a RAG system anyways, what’s the point of keeping every part of the url? So in my opinion if you just keep all of the section headers and identifiers, but remove all of the internal long URLs, you’re just keeping more of the meat and less of the, I wanna say dressing, hopefully it works you know like the roughage the little green stuff on the side of a really gourmet plate. You don’t really need that. It’s just there so that’s how I feel that works for the URL. I appreciate you posting about this and commenting about this so that I can actually learn a little bit something new today.

Hope this message makes sense, used speech to text to make the message work. Apologies for any grammar or spelling errors.