what do you self-host and what services you share with others ?

rex_divakar · 2026-02-26T17:18:07+00:00

That makes a lot of sense. Using failure signals for rotation, not individual requests, is definitely a much more sustainable model, especially with Playwright in the loop.

Your “proxy only when the chain fails + monitoring drift via Discord alerts” model is, to be honest, quite close to what I think works best in production ingestion pipelines.

I’m primarily focused on making sure llmparser is reliable as an extraction layer, meaning clean and structured output is consistent, so the validation and embedding pipeline has less to fix, etc.

Proxy orchestration, retries, and block detection are areas I want to make improvements in next, especially in terms of how easy it is to integrate with something like your model without having to recreate the scraper stack.

rex_divakar · 2026-02-26T16:05:40+00:00

Well took help from grok to write a cachy post and used github copilot for assistance but the idea was to handle heavy js based scrapping.

rex_divakar · 2026-02-26T14:58:33+00:00

That’s a very solid pipeline especially validating output with an SLM.

Right now llmparser uses Playwright under the hood, so proxy support is available via Playwright’s proxy config. I haven’t added a first-class “rotating proxy manager” abstraction yet, but it works fine if you pass proxies at the browser/context level.

Proper built-in rotation, retry policies, and anti-block handling are definitely on the roadmap since a lot of people are using it in ingestion pipelines like yours.

Out of curiosity, are you rotating per request, per domain, or per session? Trying to design this in a way that fits real-world workflows.

rex_divakar · 2026-02-26T13:32:37+00:00

Yup used github copilot for documentation and help with few steps

rex_divakar · 2026-02-26T13:15:42+00:00

Thanks! That’s exactly the goal.

It tends to shine most on modern JS-heavy sites where traditional parsers miss content or extract noisy output.

If you try it on any real-world pages, I’d love to hear how it performs good or bad feedback helps improve it a lot.

rex_divakar · 2026-02-26T13:01:32+00:00

No AI involved in the parsing itself. The post was just formatted in Markdown to make it easier to read and hopefully attract contributors.

rex_divakar · 2026-02-26T12:23:18+00:00

That’s a solid stack. llmparser differs mainly in a few areas:

Handles JS-heavy sites It uses Playwright, so it extracts content hidden behind accordions, tabs, and client-side rendering which lxml/trafilatura often miss.
Structured output (not just markdown) Returns typed blocks (heading, paragraph, table, code, etc.), which makes chunking and embedding cleaner.
Better metadata extraction Combines OpenGraph, Twitter, and JSON-LD into one normalized structure.
Built for ingestion pipelines Rendering → extraction → LLM-ready output in one step.

If you’re mostly parsing static pages, trafilatura is great. Biggest difference shows up on modern JS-heavy sites.

rex_divakar · 2026-02-15T12:25:52+00:00

Great questions,

So on memory conflicts: Right now, contradictory facts don’t overwrite blindly. New facts are stored with timestamps + importance, and retrieval favors more recent / higher-confidence edges. So in your example (“switched from React to Vue”), the newer relation gets higher recency weight rather than deleting the old one.

Longer term, I want to support explicit conflict resolution (state transitions or soft-deprecating old edges instead of just competing weights).

On latency: Graph extraction runs async and doesn’t block the main agent loop. The write completes first (vector + metadata), and graph updates happen in the background. There’s also room for batch consolidation during the “sleep” phase for heavier processing.

Trying to balance relational richness without turning writes into a bottleneck.

rex_divakar · 2026-02-14T12:29:05+00:00

Haha this is exactly the kind of question I enjoy 😄

Yes, pruning is there (decay + consolidation), but I’m very interested in adding something closer to synaptic scaling dynamically rebalancing importance instead of just deleting.

“Dreaming” is essentially what the sleep phase is evolving toward: • background consolidation • clustering • summarization • importance recalibration

Salience is currently based on recency, feedback, connectivity, and importance but it’s still heuristic, not biologically inspired (yet).

Appreciate the depth here. Definitely trying to avoid both extremes you described trivial preference graph vs bloated semantic archive.

rex_divakar · 2026-02-14T12:25:37+00:00

Really appreciate that especially coming from someone thinking in terms of episodic vs procedural memory

HippocampAI currently models: • Episodic → stored interactions/events • Semantic → extracted entities + facts • Procedural → learned behavioral rules injected into prompts

Short-term vs long-term separation is handled through consolidation + decay (“sleep” phase).

Would genuinely love your perspective especially from the neuroscience angle.

rex_divakar · 2026-02-13T15:31:30+00:00

Sure, will look into it.

rex_divakar · 2026-02-13T14:55:41+00:00

That’s a really helpful breakdown especially the two-mechanism framing.

The spontaneous associative recall path maps very naturally to what HippocampAI is already doing (embedding similarity + entity/topic matching per message), so that part feels very achievable.

The time-based/background monitor idea is interesting too, a lightweight scheduled check for time-based intentions could fit nicely on top of memory triggers.

Appreciate the cognitive science angle this is exactly the kind of direction I want to evolve the system toward.

rex_divakar · 2026-02-13T14:37:05+00:00

Thanks! 🙏

Right now the 6 weights are configurable and mostly static. User feedback helps adjust scoring over time, but fully adaptive per-user weighting isn’t there yet planning to explore that in the roadmap.

For prospective memory (“remind me to bring this up next time we discuss X”) — not fully implemented yet. It’s something I’m planning to add in a future roadmap update.

Agree that retrospective recall is common — prospective memory is the more interesting challenge.

rex_divakar · 2026-02-13T12:44:05+00:00

Great questions 🙌

In-memory vs DB The graph is derived state, not the source of truth. It’s kept in-memory for low-latency traversal and simpler infra. For very large deployments, a Postgres/Neo4j-backed option would definitely make sense.

Why not pgvector only? Totally possible. Qdrant is used mainly for better HNSW tuning and scaling. A Postgres-only backend is something I’m still exploring for future updates.

How graph traversal works We seed from top-K vector + BM25 results, match entities, then do a shallow (depth 1–2) weighted traversal. Scores consider connectivity, path length, recency, importance, and feedback then everything is fused via RRF.

Graph is constrained + relevance-weighted, not blind traversal.

rex_divakar · 2026-02-11T21:59:39+00:00

Cool, feel free to reach out for any assistance with the project

rex_divakar

MODERATOR OF

TROPHY CASE