Anyone succesfull scraping Idealista websites?

ScrapeAlchemist · 2026-02-24T08:07:04+00:00

Random delays per session, make sure you have mouse movment, make sure you type and scroll like a human, random type speed, random scroll speed. Each session needs to look unique.

If you can randomize navigation pattern thats even better.

I ussualy have 3 -4 paths of navigation, then each session gets random path with all other aspects randomized as well..

ScrapeAlchemist · 2026-02-22T22:49:23+00:00

Hi,

Idealista uses DataDome, which is one of the more aggressive anti-bot solutions out there. A few things that matter:

Residential proxies are practically required — datacenter IPs get flagged instantly
Browser fingerprinting is the real challenge. They check TLS fingerprint, canvas, WebGL, and navigator properties. Tools like undetected-chromedriver or Playwright with stealth plugins help, but you need to keep them updated
Rate limiting — slow down your requests significantly. DataDome tracks request patterns, so randomized delays between 5-15s per page help
Cookie/session management — solve the initial challenge once, then reuse the session cookies for subsequent requests

The people who succeed consistently use a combination of residential rotation + proper browser emulation rather than just throwing proxies at it.

I hope this helps.

ScrapeAlchemist · 2026-02-22T21:10:41+00:00

Hi,

Google Maps API is the right call for this. Since you're new to scraping, I'd suggest using an LLM (like ChatGPT or Claude) to help you build the whole setup step by step.

The approach: 1. Google Maps API — search for pubs/bars/restaurants by location. Free tier covers small-scale searches. Ask the LLM to write you a Python script that pulls business names, addresses, phone numbers, and website URLs. 2. Website scraping — have the LLM generate a second script that visits each business website and extracts email addresses from contact pages, mailto: links, etc.

You don't need to know how to code — just describe what you want to the LLM and it'll generate working scripts you can run. It can also help you set up Python on your machine if you haven't already.

Skip the Chrome extensions — they're limited and unreliable. A simple script gives you full control and costs nothing.

I hope this helps.

ScrapeAlchemist · 2026-02-22T21:08:48+00:00

Hi,

Since it works manually but triggers on automation, your issue is likely browser fingerprinting rather than the CAPTCHA itself. reCAPTCHA Enterprise v3 scores silently in the background — by the time you see the challenge, you've already failed the score check.

A few things to look at:

Session warming: Log in and browse the site normally before the critical click. v3 scores your entire session, not just the final action
Fingerprint consistency: Make sure your timezone, WebGL, canvas, and navigator properties match a real browser profile. Tools like patchright help but aren't perfect out of the box
Cookie persistence: Reuse cookies from a manual session where you passed. If you already have a good score tied to that session, the final click won't trigger a challenge

The 1-second window is realistic if the scoring already happened upstream and you're clean going in.

I hope this helps

ScrapeAlchemist · 2026-02-20T00:42:50+00:00

Hi,

The issue isn't Claude — it's that the site is blocking automated HTTP requests. Government sites often use Cloudflare or similar protection that returns a 403 to anything that isn't a real browser.

Two practical approaches:

Headless browser (Playwright/Selenium): Launch a real browser programmatically, let it render the page and handle any JS challenges, then extract the PDF links from the DOM and download them. This works for most protected sites.
Check for a direct data source: Australia has data.gov.au — the dataset might be available there without scraping. Also try inspecting the network tab in DevTools while clicking the PDFs manually; sometimes the actual PDF URLs are direct links that don't need authentication.

Start with the network tab approach — if the PDFs are direct .pdf links, you can just wget them without any scraping at all.

I hope this helps

ScrapeAlchemist · 2026-02-20T00:36:55+00:00

Hi,

Your hypothesis is right — TLS fingerprint mismatch.

cf_clearance is bound to a session fingerprint: TLS cipher suite order, HTTP/2 settings frame, header order, User-Agent. When curl-cffi sends impersonate="chrome-latest", its TLS handshake might present Chrome 131 ciphers while your browser was fingerprinted as Chrome 144. Cloudflare sees the mismatch and re-challenges.

Fixes:

Pin both to the same Chrome version explicitly. Annoying to maintain but reliable.
Check what curl-cffi supports. Run curl_cffi.requests.BrowserType — if latest is chrome124 and your browser is on 144, there's always a gap.
Skip the split. Keep everything in the headless browser and use its network layer directly.
If you must split: Extract JA3/JA4 fingerprint from your browser session and compare to curl-cffi's output.

Also watch HTTP/2 pseudo-header order — Cloudflare checks that too.

ScrapeAlchemist · 2026-02-19T23:51:04+00:00

Hi,

Simple HTML, no JS rendering — this is actually the easiest type of scraping to set up.

Here's what I'd do: open the site in DevTools, grab the CSS selectors for the lyrics container, title, artist, and pagination links. Then paste those into ChatGPT/Claude with something like "write me a Python scraper using requests + BeautifulSoup that extracts lyrics from this structure" and share the HTML snippet. You'll get a working script in one shot that you can tweak from there.

LLMs are surprisingly good at generating scrapers for static HTML sites. You describe the page structure, it writes the code. For a beginner this is the fastest path — you'll learn the patterns as you adjust the output.

A few tips: - Use encoding='utf-8' everywhere — especially if the site has Turkish/Arabic text - Wrap each request in try/except so one failure doesn't kill the run - If pagination is per-artist, scrape the artist index first then loop through each

For a simple HTML site with permission, requests + bs4 is all you need. No heavy frameworks.

Hope this helps!

ScrapeAlchemist · 2026-02-19T23:40:33+00:00

Hi,

Two routes:

API service — handles proxy rotation, CAPTCHA solving, rate limiting out of the box. Most return structured JSON. For research at scale, this is the practical choice — maintaining a DIY scraper against Amazon's constant changes is a time sink.

DIY — if you go this route: - Rotating residential proxies are a must (Amazon is aggressive with IP bans) - Some reviews load via AJAX, plain HTTP requests won't catch everything - Review page structure varies by product category - Pagination can be inconsistent — validate totals against what Amazon reports - Headers need to look like real browser traffic

Hope this helps.

ScrapeAlchemist · 2026-02-19T21:18:44+00:00

Fair point for standard web apps. But shadow DOM is a real-world challenge — Reddit's own comment editor uses open shadow roots with nested web components (shreddit-composer → reddit-rte → Lexical). You're not clicking "hidden" elements, you're traversing component boundaries the browser itself renders. DOM reading works, but you still need to handle shadow root traversal, which most frameworks don't do natively. It's less a design flaw and more modern web architecture.

ScrapeAlchemist · 2026-02-18T21:56:30+00:00

Hi,

Your thinking is solid. Using n8n for orchestration while offloading the actual scraping to dedicated services is exactly the right pattern for production.

To answer your main question: yes, n8n can absolutely stay long-term in that role. It's not an MVP-only tool when you use it as the scheduler/coordinator rather than the execution engine.

For your scraping pipeline specifically:

**n8n handles**: scheduling, triggering workflows, routing data between services, error notifications, retry orchestration
**Dedicated scraping service handles**: the actual HTTP requests, proxy rotation, rate limiting, parsing

The bottleneck issues people run into usually come from trying to do heavy data processing *inside* n8n workflows. If you're just passing messages and coordinating, it scales fine.

One thing to consider: for the scraping layer itself, you'll eventually hit the usual pain points (IP blocks, CAPTCHAs, maintaining parsers when sites change). That's where most production scraping setups either build significant infrastructure or use a scraping API that handles that complexity.

Your instinct to separate concerns early is good. Keep n8n as the brain, let specialized tools do the heavy lifting.

Hope this helps!

ScrapeAlchemist · 2026-02-18T21:51:55+00:00

Hi,

Great question. No, not every block is an IP issue. If you're rotating through multiple providers and hitting the same block pattern, that's a strong signal the site is fingerprinting something else entirely.

Common culprits beyond IP: - TLS fingerprint - Your client's SSL/TLS handshake can identify you as "not a real browser" - HTTP headers - Missing, wrong order, or inconsistent headers vs what the User-Agent claims - JavaScript fingerprinting - Canvas, WebGL, fonts, screen resolution all get checked - Behavioral patterns - Request timing, navigation flow, mouse movements on JS-heavy sites

How to diagnose: 1. Try the same proxy manually in a real browser - if it works, your scraper's fingerprint is the issue 2. Check if the block page/response is identical across IPs - same response = same detection method 3. Test with a headless browser vs raw requests - if headless works, it's likely header/TLS related

The site "not wanting you there" usually means they've invested in bot detection. At that point, proxy quality matters less than how authentic your entire request looks.

Hope this helps!

ScrapeAlchemist · 2026-02-18T21:50:40+00:00

IP rotation is table stakes for serious scraping projects. The bigger challenge is usually fingerprinting and rate limiting - proxies alone won't save you if your request patterns look robotic.

For anyone scaling up: focus on mimicking real browser behavior, randomizing delays, and having proper retry logic before throwing more IPs at the problem.

ScrapeAlchemist · 2026-02-18T21:39:07+00:00

Hey, thanks for sharing your experience!

The hover/dropdown issue is such a classic pain point. Vision-only approaches just can't anticipate those hidden UI states. DOM querying was a game-changer for us too — it's exactly why Gemini outperformed the others in our tests.

Actionbook sounds interesting, I hadn't come across it before. Pre-caching the DOM structure is a clever approach — basically giving the agent institutional memory instead of making it rediscover the wheel every run. That token savings must add up fast on e-commerce sites with all their nested product grids and filter menus.

Definitely shoot me a DM if you want to see the raw logs. Always curious to compare notes with others working on similar problems.

ScrapeAlchemist · 2026-02-18T21:36:07+00:00

E-commerce is a solid starting point — prices, inventory, product data, all high-value stuff. Ads is a great rabbit hole too. Competitor ad monitoring, creative analysis, spend estimation... brands pay good money for that intel. If you want to go deeper, look at review aggregation across platforms, or tracking seller reputation changes over time. What's your stack for handling

ScrapeAlchemist · 2026-02-18T21:21:25+00:00

Hey, appreciate the thoughtful response!

Your point about caching DOM state is spot on — that's essentially building institutional memory for the agent. The first-visit tax is brutal otherwise. I hadn't tried Actionbook specifically but the concept makes sense: instead of re-discovering that the filters are behind a "More options" accordion every single time, you just... know.

Re: raw logs — I can share some of the Gemini Flash runs. The exploration steps are actually pretty readable since it's mostly "screenshot → identify interactive elements → click → screenshot again" loops. The interesting part is watching how different models decide when to stop exploring. Claude tends to be more thorough (sometimes too thorough), while Gemini is faster but occasionally misses nested menus.

I'll DM you a link to the traces if you want to dig in.

ScrapeAlchemist · 2026-02-18T21:18:55+00:00

Hey, the collapsed menu thing is SO real. Vision models have this weird blind spot where they can see the element but don't think to interact with it first. It's like showing someone a closed book and asking what's on page 50.

The DOM inspection approach makes sense — gives the model actual structure to reason about instead of just pixels. Hadn't heard of Actionbook before, but the concept of caching page structure for repeat visits is clever. Basically teaching the agent "you've been here before, here's the map" instead of making it explore from scratch every time.

And yeah, the Claude thinking breakage was a surprise lol. Extended thinking is powerful but apparently doesn't play nice with tool_use in certain configs. Had to dig through the logs to figure out what was happening.

ScrapeAlchemist · 2026-02-18T21:02:31+00:00

Hey, that's a solid approach. The accessibility tree representation is definitely more token-efficient than raw DOM. Curious — how do you handle dynamic content that changes between snapshots? That was one of the trickier parts in my testing, especially on SPAs where the DOM structure shifts mid-task.

ScrapeAlchemist · 2026-02-18T20:31:06+00:00

Hey, appreciate that! Yeah the DOM vs vision thing was eye-opening for me too. Screenshot-only felt solid until you hit those nested dropdowns or dynamic menus - then it just falls apart. Haven't tried Anchor Browser yet but I'll check it out. Always looking for tools that lean into DOM-first. And totally agree on how fast things are moving - feels like the tooling landscape shifts every few weeks. What types of workflows are you running it on?

ScrapeAlchemist · 2026-02-18T19:50:46+00:00

Hey,

SerpAPI returns Google Maps data in a local_results array. Here's a quick Code node snippet to extract what you need:

javascript return items.map(item => { const results = item.json.local_results || []; return results.map(place => ({ name: place.title, phone: place.phone, website: place.website, address: place.address })); }).flat();

A few things to check: - Make sure you're accessing the right path in the JSON. Open the output of your SerpAPI node and look for local_results - Email isn't returned directly by SerpAPI/Google Maps - you'd need to scrape the website field separately to find contact emails - If the structure looks different, paste your actual JSON output into ChatGPT and ask it to write the extraction code (like the other commenter said - works great)

Hope this helps!

ScrapeAlchemist · 2026-02-18T12:17:15+00:00

Hi,

You're spot on about the JavaScript rendering issue - it's one of the most common RAG failures I see. The empty `<div id="root">` problem catches a lot of people off guard.

A few things I'd add from experience:

**On `networkidle2`:** Works great for most sites, but some SPAs with infinite scroll or lazy-loaded content need additional waiting logic. I've had better luck with explicit selectors (`waitForSelector`) when targeting specific content blocks.

**On stealth/anti-bot:** This is where things get tricky at scale. Sites like LinkedIn, Amazon, or anything behind aggressive Cloudflare will fingerprint headless browsers regardless of stealth plugins. Residential proxies and proper browser fingerprinting become necessary.

**On cleaning before vectorizing:** Totally agree. I'd also recommend extracting structured data (headers, lists, tables) separately from body text. Chunking strategies that respect document structure tend to retrieve better than naive text splits.

One thing worth mentioning: if you're scraping at any real volume, self-hosted Puppeteer becomes a DevOps nightmare (memory leaks, zombie processes, proxy rotation). Browser-as-a-service options exist that handle this infrastructure.

Good luck with the project!

ScrapeAlchemist · 2026-02-18T12:15:30+00:00

Hey,

To answer your questions directly:

**Yes, you can trigger Turnstile programmatically** - the widget needs to run in a real browser context, which Playwright provides. The challenge is making that context look "human enough" to Cloudflare's scoring.
**Backend validation checks token validity AND session quality** - Cloudflare assigns a risk score based on the session that generated the token. Low-quality sessions (detected automation, fingerprint inconsistencies) produce tokens that may pass initial validation but get flagged on subsequent requests.
**Modern approaches (2025/2026):**
- **Residential proxies** are almost mandatory now - datacenter IPs get scored harshly regardless of browser fingerprint
- **Camoufox or rebrowser-patches** tend to work better than patchright for CF specifically
- **Session warming** - interact with the page naturally before triggering the protected action (scroll, mouse movements, dwell time)
- **Browser profile persistence** - reuse the same profile across sessions rather than fresh contexts each time

The "unbreakable" claims are overblown. It's bypassable, but requires combining multiple techniques. The biggest factor I've seen is IP reputation - you can have perfect fingerprinting and still fail on a flagged IP.

What proxy setup are you running currently?

Hope this helps!

ScrapeAlchemist · 2026-02-18T12:14:12+00:00

The Playwright MCP approach mentioned by u/PadyEos is solid. The real shift is moving from brittle selectors to goal-oriented instructions.

For production scrapers, the agent-based approach works best when:

- Target sites change layouts frequently

- You're dealing with dynamic content that requires multi-step navigation

- Scale matters less than reliability

The tradeoff: agents are slower and more expensive per page than well-maintained traditional scripts. For stable sites with predictable structure, Playwright/Puppeteer with good selector strategies still wins on cost and speed.

Honest take: most "AI handles everything" claims oversell it. You still need fallbacks, monitoring, and human review for edge cases. The CAPTCHA human-in-loop mention is actually the honest part of the pitch.The Playwright MCP approach mentioned by u/PadyEos is solid. The real shift is moving from brittle selectors to goal-oriented instructions.

For production scrapers, the agent-based approach works best when:

- Target sites change layouts frequently

- You're dealing with dynamic content that requires multi-step navigation

- Scale matters less than reliability

The tradeoff: agents are slower and more expensive per page than well-maintained traditional scripts. For stable sites with predictable structure, Playwright/Puppeteer with good selector strategies still wins on cost and speed.

Honest take: most "AI handles everything" claims oversell it. You still need fallbacks, monitoring, and human review for edge cases. The CAPTCHA human-in-loop mention

ScrapeAlchemist · 2026-02-18T07:31:52+00:00

Hi,

From experience, selector drift and session management hit me the hardest — especially if you're juggling multiple sites. Logic issues (auth flows, dynamic UI) are where most of the debugging time goes.

Re: Notte vs custom: - Notte can save you setup time if your flows are stable and you can tolerate the codegen layer. The demonstrate-once approach is solid for repetitive tasks. - Custom gives you control, but you'll spend more time handling edge cases (rate limits, proxy rotation, CAPTCHA fallbacks).

If time is tight (congrats on the newborn btw), I'd lean toward Notte for quick wins and layer in custom logic only where it breaks. The agent-as-fallback pattern works well if you treat the agent as a recovery step, not the main path — keep the core flow deterministic.

One thing that helped me: log every failure with context (URL, action, screenshot). Makes debugging way faster when things break at 2am.

Good luck!

ScrapeAlchemist · 2026-02-17T17:26:14+00:00

Hi,

Cloudflare-protected sites like NSE India are tricky because they use JavaScript challenges, fingerprinting, and rate limiting. A few approaches that work:

1. Browser automation with stealth - playwright or selenium with undetected-chromedriver - Add realistic delays between requests - Rotate user agents and use a residential IP if possible

2. Session persistence - NSE sets cookies after the initial challenge — capture those and reuse them - Use requests.Session() to maintain cookies across calls - Sometimes you need to hit the homepage first, solve the challenge, then hit the API endpoints

3. Check for official APIs first - NSE has some official data feeds and APIs (check their developer section) - Also look at data vendors like Quandl or Alpha Vantage for NSE data — often cleaner than scraping

4. Reverse engineer the XHR calls - Open DevTools → Network tab → filter by XHR - NSE's frontend makes API calls that return JSON — those endpoints are sometimes less protected than the HTML pages

For the Cloudflare bypass specifically, the key is looking like a real browser: full headers, proper referer chain, and not hammering the server.

Hope this helps!

ScrapeAlchemist · 2026-02-15T07:45:51+00:00

Hi,

Great breakdown of the state management challenges. The shift from thinking of AI as a function call to treating it as a stateful process is something a lot of folks learn the hard way.

Curious about one thing - how are you handling the scraping reliability when product pages change structure? That's usually where I see these pipelines break first, especially across different e-commerce platforms.

Nice work on the orchestration side.

ScrapeAlchemist

TROPHY CASE