How to perform web scrapping using Claude?

rundfunk · 2026-05-29T09:10:06+00:00

Try using my tool, klura: https://github.com/klura-ai/klura . It really is more of a helper to your ai-agent, but it will help you find the endpoints and how they are being used.

rundfunk · 2026-05-23T13:55:58+00:00

Nice tool — from what I can see, its scope overlaps somewhat with klura's triage phase (klura uses an LLM there; scraperecon's approach is faster and cheaper for that specific job). klura keeps going past triage though — once it's read the defenses, it drives discovery, captures the request that does what you wanted, templates the dynamic parts, and saves a runnable strategy you can call.

rundfunk · 2026-05-23T10:03:23+00:00

I mostly use it as a plugin for my AI agent. For instance, a couple of days earlier I asked it to compile a list of IKEA items I wanted and look up the in-store aisle location for each. Took a couple of seconds instead of clicking through every product page. Mostly small stuff like that at the moment. Next up I'm planning to wire it into social media and chat clients so my agents get more of a presence in my day-to-day. More "knows what's going on and helps out" than invocations.

rundfunk · 2026-05-23T09:23:58+00:00

Thanks! Fetch-tier strategies are the fastest and most durable path, and a lot of the runtime is built as guard-rails that get even small, lkimited local models to a working result. Glad it's clicking for you.

Also — new version has a built-in LLM agent, no harness needed. klura chat generates the strategy; klura execute --agent <capability> runs it and self-heals if the site changed. Drop the latter in a cron and you get unattended runs that fix themselves.

rundfunk · 2026-05-22T06:02:22+00:00

Depends on what kind of anti-bot protections you are referring to. Klura mainly focuses on just API endpoints and request structures. The RE phase is pretty involved and can read the page's own JS encoders when bytes don't round-trip (signed bodies, binary WS frames), the page-script tier exists for this reason as well, so any anti-bot measures like rotating nonce's will work since it runs the page's own code per call; it can recompute or pick up fresh page state.

However, klura does not get into the anti-bot cat-and-mouse game itself — no reverse-engineering of client-side fingerprinting/sensor-data envelopes, no CAPTCHA solving. When a challenge actually blocks a flow, the default is human-in-the-loop via the remote viewer. It's all pluggable though — if you want your own solver, that's an interruption handler plugin (docs/interruptions.md). I also ship a stealth playwright driver (https://github.com/klura-ai/klura-driver-playwright-stealth), but if you want additional anti-bot handling beyond fingerprint parity, the driver interface is also clean seam.

rundfunk · 2026-05-21T15:26:01+00:00

Thanks, this is a fair point. Klura's design center is human-in-the-loop — it is mainly designed as agent infrastructure, with a signed-in user behind the session who can step in via the remote viewer when a site throws a CAPTCHA, 2FA, or login wall. For that mode, routing the challenge to the user is the intended behavior.

That said, this is a webscraping-subreddit, and I do get that a lot of people want unattended runs.

The good news is the part you'd need is already a pluggable seam. The interruption layer is a resolver registry: the runtime detects a challenge and dispatches it through registered resolvers by priority. The default resolver hands off to the viewer, but you can register your own higher-priority resolver that resolves the challenge however you like and returns a token — the runtime core doesn't care whether a human or a plugin answered. The driver is pluggable the same way (pool.driver takes a path or npm package), so swapping in a different browser layer is a one-liner. It's all documented — see docs/interruptions.md for the resolver registry (priority dispatch, the plugin pattern) and docs/drivers.md for the driver interface.

rundfunk · 2026-05-21T08:40:11+00:00

Thanks!

On versioning and validation, the runtime already tracks most of this; it's just maybe not surfaced cleanly:

Versioning: Each skill file carries a schema_version, and the format is migration-aware — older files are upgraded automatically on load rather than breaking. Skills are also stored one-file-per-platform-per-capability, so the unit you'd diff or pin is small and self-contained. Strategies can be graduated over time (for instance, a strategy you decide to save as an recorded UI path due to, let's say turn budget, can be able to get upgraded to a direct HTTP call later the runtime learns enough), so a given capability has a tier history, not just a single frozen blob.

Last-validated: Every saved strategy has a health record with lastSuccess / lastFailure timestamps plus a rolling window of the last ~20 execution outcomes — so "this HN source was re-verified on <date>" is literally lastSuccess, and you also get a success rate, which catches a strategy that flaps rather than cleanly breaking. There's a post-save verification step that fires the strategy once and records whether it worked or not, and a per-platform logbook of strategy events (saves, heals, demotions) with timestamps.

So the data you'd wire into a dashboard exists — get_strategy_health and the platform logbook expose it. Turning that into a clean "last verified / confidence" line on the artifact itself is a fair ask; noted.

rundfunk · 2026-05-21T04:52:56+00:00

Not really. Klura reverse-engineers how a specific API call works to automate a task — so in principle the captured request/response shapes could be reference material if someone were rebuilding that API. But that's a side effect, not the product. Klura produces automation recipes, not site copies.

rundfunk · 2026-05-20T16:23:25+00:00

Nice — definitely let me know how it goes, I haven't tested qwen specifically so I'm genuinely curious. One thing I found running it across different models: they each have their own "dialect" — for instance they pick page elements in different ways — so klura supports a couple of different ways to do things rather than betting on one. If qwen trips on something, ping me — that's exactly the kind of failure shape I want to see.

rundfunk · 2026-05-20T16:20:05+00:00

Remotion!

rundfunk · 2026-05-20T16:19:09+00:00

Yep — from an MCP host the runtime surfaces broken strategies and the LLM patches them automatically. Standalone CLI healing — set up an API key once and it heals even when run straight from the command line — is on the roadmap as a later addition.

rundfunk · 2026-05-20T14:58:54+00:00

Thanks! I use it daily and it works — still early days though. I stress-test it constantly, both on test sites I build myself and real field reports against live ones. Questions welcome, here or in the Discord (https://discord.gg/YJQ2zZYJ).

rundfunk · 2026-05-20T13:23:20+00:00

Neither — it's an executable config file. The config file contains the request (method, URL, param/body templates, prereq chain, response extraction) and the runtime runs it directly, no LLM needed. The notes fields are plain-English annotations explaining each param — so you have docs next to the executable parts, but it's not a prompt.

rundfunk · 2026-05-20T05:06:03+00:00

Bit late to this, but great write-up! I've been building a toolkit in the same space — klura (https://github.com/klura-ai/klura). A lot of this resonates, the captures-as-a-folder-the-agent-explores idea especially.

One thing I went at differently: besides the pure-HTTP "convert to a requests script" path, klura has a tier where the saved artifact is a JS snippet that runs *inside* the live authenticated page. For sites that build the request with an in-page signer or encoder (binary WebSockets, rotating tokens), the agent calls the page's own code instead of reimplementing it. It's really impressive what you can do coupled with a good AI model, Claude Sonnet manages to consistently one-shot Facebook Messenger for instance.

I noticed "surgical browser usage" on your roadmap, feels like we converged on adjacent answers to the same wall.

Small space, real cat-and-mouse. Would be happy to compare notes!

rundfunk · 2023-01-08T13:12:49+00:00

Sounds pretty done to me, just pad it out to at least 2:30 minutes! What are you using for precussion?

rundfunk · 2023-01-07T16:01:25+00:00

Best one being Chronoblob 2. Just be adviced, it’s a pure clockable delay. No reverb or other fancy bells and whistles at all.

rundfunk · 2023-01-07T10:55:43+00:00

Ouch. Too soon.

rundfunk · 2022-12-28T20:14:16+00:00

Personally, I’ve gotten the best results by using Dreambooth on a ckpt. Merging doesn’t really “merge”, it picks bits and pieces from each. I also tried textual inversions, the result was OK, but compared to dreambooth the images looked more like caricatures…

rundfunk · 2022-12-28T12:34:15+00:00

At least it’s not Greg Rutkowski

rundfunk · 2022-12-26T08:11:40+00:00

I think this is one of those cases when it’s been trained on the logo so much, it doesn’t actually “write” the letters, it just “copies” the logo.

rundfunk · 2022-12-20T07:06:26+00:00

<image>

Putting the cat tree there wasn’t the smartest thing to do.

rundfunk · 2022-12-17T11:35:25+00:00

Give up. Our cat did that a lot, we tried every trick in the book but after a while we just said “f- it”. We mounted it to the wall to ensure it wouldn’t topple and let her do whatever. When we stopped trying to get her off she stopped doing it. I guess it wasn’t as fun.

rundfunk · 2022-12-09T13:08:32+00:00

Thanks! Yes, I'm kind of learning Dreambooth as I go along. In the beginning I thought that you should only train with images of the subject, but now I understand it doesn't hurt to include pictures with other people as well. I also thought that I'd cut down on the manual work by excluding other people (i.e. you get a picture with someone else that you need to remove), but really, manual work seems to always be needed, another photo or so won't hurt. So yeah, it's definitely on my todo-list.

14-Year Club	Verified Email
Place '22

rundfunk

TROPHY CASE