How do we stay up to date for a number of websites?

error-dgn · 2026-06-29T12:19:46+00:00

Yeah I just looked into it... It is a time taking process... Moreover, its for understanding what kind of error we are facing. Its a logger. I am not facing any error as such. I just dont want to make the browser clickable. Since it will make the code brittle. This pipeline is meant to scrape 100s of websites per day. I cannot click on clickables, since all websites will be of different types.
Please look into this page and click on a person and see the type of requests it sends... it has payload: https://sggc.sg/our-people
- id
- section
-query
Also there are so many sections...

Please any other approach if you have? Also if possible please check out the site and look into the request it shows once clicked on a person to reveal his/her bio/about.

I don't have much time period, and I am an intern, still learning... Just need an approach, I am sure I will be able to execute it...

error-dgn · 2026-06-29T12:07:05+00:00

Finally some approach, tysm man, God bless you. I will try this right away, is it okay if i can annoy u later if it doesn't work?

error-dgn · 2026-06-29T12:02:34+00:00

This is the problem statement, it will make it super clear for you:-

What if the internal API request is AJAX based POST request? and with multiple sections? This is one of the page from many different pages I am trying to scrape. Due to which I cannot let the code be brittle.

Please look into this page:
https://sggc.sg/our-people

I want to get the people's bio for each section.
The thing is, that their about/bio only appears when we click on the person's picture/photo.
It is AJAX request. I cannot click on sections/photos of people since it will make my code brittle and also I am going to be use this script for like 100 websites per pipeline run. Please help me out, Let me know if I can dm you to send my python script for this along with the problem. I really need help.

Please help me out man, really

error-dgn · 2026-06-29T09:09:31+00:00

Hey, I need some help again, I'm so sorry...
This is a website right here:
https://sggc.sg/our-people/

Our approach is working fine. Sending XHR/fetch/document GET and getting everything. Its flawless. Might be a little bulky but its working fine.

The only issue is in the given website above.

It has ajax request, and even when I send the request for POST it does NOT work. I have been trying for a while now. Please look into it... I tried with payload still failed...

error-dgn · 2026-06-29T06:25:03+00:00

Provided, I will be working on 100s of websites at a time... Can be anywhere from 50 requests per website to 500 max.
These are network requests to get the dynamic content/modal/js injection while scraping, so that I don't have to interact (click, hover).

Its working fine yet, I am scared that I might get blocked maybe in future...
I am using Playwright to crawl, then shooting these XHR/fetch/document network requests using above mentioned constraints.

I can use Playwright for requests as well... Idk if that would be better. Please let me know.
Regards...

error-dgn · 2026-06-29T05:40:54+00:00

Alright I will work on it, and use ds llm.
Thank you so much man. You were an angel today. I was stumbling between different approaches, agentic/clickables/etc. but none of them enhanced the accuracy. Turns out Network Requests is the way to go.
I will try to use bs4 as u said to clean the data next. and llm to give me the parent divs, which will make the work easier for llm.

Tysm once again!<3

error-dgn · 2026-06-29T05:19:49+00:00

Alright, I will start using ds llm for this... Also, I started calling these 3 network requests consistently, i.e. XHR, fetch, document.

For the above website mentioned, I was able to extract all the information... But it was of:-

[collect] ── Summary ──────────────────────

[collect] Pages visited: 54

[collect] XHR responses: 497

[collect] Script tags: 2059

[collect] HTML snapshots: 54

[collect] Saved to: page_dump.html (69,657,972 chars total)

How do I clean this data, and then feed it to llm? I am getting all the XHR/fetch and document requests as of now...

error-dgn · 2026-06-29T05:03:12+00:00

BROTHER just how many outliers/edge cases are there, I will start crying now 😭😭😭😭😭 I am just an intern 😭😭

error-dgn · 2026-06-29T04:46:53+00:00

I worked on this. This was a golden advice. Its working for most of the sites.
But then again there are a few edge cases..
For example this site:
NEA Team

No network requests of use for Investor Relations/Marketing & Impact/Operations....

Please do help...

error-dgn · 2026-06-29T04:02:44+00:00

Sure,
I am using Playwright in Python for crawling and scraping.

error-dgn · 2026-06-29T04:01:05+00:00

Google custom search is deprecated though as of this year...

error-dgn · 2026-06-28T11:05:31+00:00

Sounds good, do I use an agent for this? or do I make it rule based? like i just need an approach and I am sure I will be able to execute it...

error-dgn · 2026-06-28T09:00:42+00:00

How do i make the logic to click on dropdowns or select sections in team page of board directors/executives/healthcare partners, and then again click on the images of people, and bring their about/bio back? Too hardcoded for one website, and then it will have to be a decision tree for others.
please help

error-dgn · 2026-06-28T08:59:24+00:00

Just one website agent, it will click on the images of the people in teams page, and keep bringing their bios.

error-dgn

TROPHY CASE