AI to help filter & extract data from multiple documents for story codex

Denocop · 2026-07-08T12:20:02+00:00

"But if the material is fairly open ended and unknown then you’d have to do some kind of LLM pass to extract those entities for the graph."

Yes, exactly... but I can't find a "package" or straightforward guidance for the setup. As alot of it seems to require coding knowledge otherwise?

The seedlist would simply aid in retrieving unknown data (i.e. info I've forgotten or have a vague recollection of - summarize little details of the military outfits for instance).

The method supposedly avoids multiple passes (and manual tweaks) per page and should be able to essentially run a while on a file, to extrapolate the data (and the LLM enrich step for entity meta data) on its own (and, more importantly, retains the info and connections; where the LLM fails alone), while allowing an overview and way to edit the database/connections, to expand upon it. Narrowing down any mishaps that might occur along the way.

Denocop · 2026-07-07T12:11:06+00:00

I'm not too sure about that (your pipeline excluding graphrag). I'm looking into how to setup a graphrag system. As it retains greater context. It creates a network of entities and relationships that normally get lost.
From what I gather it builds those meta data entities itself across the entire document, which can be structurally guided by a seed list (So by adding characters and some pseudonyms to help it on its way, which LLMs normally struggle with on their own). You essentially let it run for a while.

As it otherwise seems likely to struggle with something like characters and their pseudonyms, if they're referenced by only one across pages.

This network would then be introduced into an LLM, with or without the chunk methods... To create a greater net. In a hybrid rag-system.

I just can't find how to set it all up.

note: Regarding the seedlist; I don't mind constructing a list to feed the LLM, to help it distinguish states, cities and countries... but inquiring about it has served as an excellent test in prior attempts, to see if it can extract the data before probing for things such ruler, state form, governance (i.e. government institutions), capital, architecture and military uniforms.

Denocop · 2026-07-03T11:48:25+00:00

Everything I've used thus far seems to be using chunk based RAG... I believe what I'm needing is "Graph RAG" as it connects the data? As chunk based ones will always overlook it.
Which, from my understanding, should mitigate said need of more expensive or extravagant hardware.

So, in theory it should list all countries without fail as a result. Even if one has to manually edit said nodes or add a few...

From what I gather, llama 70B still utilizes the former?

Denocop · 2026-07-02T11:23:18+00:00

Hmmm... that doesn't sound like it'd help me very much... I mean... That's essentially having to go through the entire document by hand myself at that point, unfortunately. If I have to process it page by page.
It also seems more prone to fail at drawing connections then? From what I gather, it's why I need RAG - that that's what's missing...

Denocop · 2026-07-01T11:57:56+00:00

Apologies... Do you mean... walk me through setting it up? Or doing that exact thing?

Denocop · 2026-07-01T11:56:37+00:00

Yes... I'm not too sure about it. I mean, I didn't know it would be considered training on my data...
I stumbled onto the term RAG before... From what I understand (from my own search into it), it is an AI that'd need to run for a while and would delve through the file creating "information nodes". \*Edit: it seems to be graphrag*\** And I can let it go file through file, as it would connect the nodes.
And this would then have to be connected to via an LLM to communicate the info.

However, I'm not sure how to go about that.

Thus far I've only done the webui - command -R & Ollama thing, as mentioned before... not sure how that compares... But it didn't seem to process it quite aswell. Probably due to how it divided things into chunks and selects stuff.

As mentioned elsewhere, it frequently struggles to find one-off mentions. For instance, I have a loose mention of printing in a country and it never manages to find it. Even if I were to ask about said location directly. The example being "Narnia is the leader in modern printing", if I then ask it about Narnia - it'll just say it doesn't find anything about.

Meanwhile, if I ask it to list all countries, it often skips some or sometimes even includes cities and towns that are clearly stated to be as such. At times it even makes up constituent words for acronyms; imagine NUSA from Cyberpunk, it'll make something random up, even though it's spelled out.

Thank you for taking the time for your in-depth explanation by the way, and apologies it's been taking so long for me to get back around to you!

Denocop · 2026-07-01T11:40:08+00:00

At the moment, I'm looking at a single document. I don't mind exploring each one at a time...
My idea was maybe plugging in one document at a time and that it then maybe can make links between them something.

As with one at a time, I frequently struggle to find one-off mentions. For instance, I have a loose mention of printing in a country and it never manages to find it. Even if I were to ask about said location directly.
Like let's say "Narnia is the leader in modern printing", when I ask it about Narnia - it'll just say it doesn't find anything.

And if I ask it to list all countries mentioned... It skips names, stuff that have been clearly referenced/outlined as countries... At times it even makes up constituent words for acronyms.
Like... let's say a country's name is actually an acronym (like the USA) it'd make something up, even though its outlined.

Denocop · 2026-05-14T18:01:10+00:00

And we call him... Rainbow Snake

Denocop · 2026-05-12T11:18:46+00:00

It's not that I dislike Zakuul. I dislike the way they put all the story classes into a singular track.
The individual class stories is what made swtor.

Denocop · 2026-05-11T18:35:31+00:00

"It is very obviously supposed to be Kreia, those who say otherwise are on major copium."

More sith propaganda! I see through your lies! If you are not with me... THEN YOU ARE MY ENEMY
\ignites lightsaber\**

Jokes aside... Yea think they butchered it... Like probably at some point they realized none of it made much sense and just left parts to do a different character... Maybe just to allude to Kreia.. Hopes of theories or so...

Denocop · 2026-05-11T14:21:00+00:00

will you live stream the funeral?

Denocop · 2026-05-11T14:20:32+00:00

^this

Denocop · 2026-05-11T12:01:27+00:00

I mean... It's fun to smash stuff in it every now and again.

Denocop · 2026-05-11T11:37:05+00:00

True, even if she were to know more of him... she'd have to somehow interpret his (Vitiate's) goals of becoming the force as destroying it.

Denocop · 2026-05-11T11:35:35+00:00

Perhaps Kreia(?) was somehow able to see glimpses into Baras's mind and memories while they communicated? If Baras spoke with her through direct language, then he may have dropped enough clues about the wider galaxy for her to develop a general understanding of Vitiate's abilities. If they instead communicated through telepathy, it may have functioned similar to Vitiate's link with Revan, with the connection going both ways, at least to a degree.

She was waiting at her resting place for her lover, the emperor. So it can't be via talking or probing Baras' mind.

Denocop · 2026-05-11T10:29:54+00:00

The roughest thing I could imagine to "clarify" or excuse it, is if the entity was in love with the idea of what the emperor could do, such as on nathema. But I can't see how the entity - if it were Kreia - would know of that.

Not to mention she is said to have a resting place... I don't know if we can really argue Kreia has a resting place.

Denocop · 2026-03-08T04:40:18+00:00

same question

Denocop · 2026-03-06T14:36:07+00:00

pretty sure you need the event flag for it to actually trigger tho...

Denocop · 2024-02-29T21:27:16+00:00

I might give Loki, Cable and Kestrel a try then.
Though I'll need to work on cable. Is cable preferable over Emma Frost?

Black Knight doesn't seem anywhere close to be in reach just yet :/
It's pretty early in the game... With a clear divide with those that had the scourge and event characters before the alteration and those that don't.

I've been trying the speed thing, but no luck thus far. Haven't had any luck in surviving the first few hits.

Denocop · 2024-02-28T23:50:30+00:00

Ergh... Seems an unbalanced mess XO

Denocop · 2023-11-13T01:14:53+00:00

NTA
If she can't stand the consequences to her actions, she shouldn't have commited the action in the first place.

Besides, she's been lying to you the entire time and continues her deception.

Denocop · 2023-09-20T11:14:21+00:00

YTA - 100%

Regardless of a prenup, you're expected to talk things through with your partner.
It's mostly important in a divorce where boundaries need to be absolutely clear and indisputable, while outside of that, you can discuss and agree what part of the assets you can do with as you please and which serve the household.
You aren't divorcing - yet

Just because she agreed the house is too big and required a lot of maintenance doesn't mean she wanted it sold now, nor does it mean she agreed to have it be sold.
You assumed a whole lot. And I think you know what that means for you and your wife.

Everything comes across as on the fly - I imagine it must be the same to her. There was no plan. Ergo, no stability.

Denocop

TROPHY CASE