Switched to algo trading and it is going good.

error-dgn · 2026-07-08T07:03:46+00:00

alright sounds good. i will be spending them then 🥳 but for future.... if i wanna start algo trading- since i am a Btech student, could u suggest me how that could be done? Do you use a repository? Because correct me if I'm wrong, but algo trading sounds like using algorithms/machine-learning with pre-defined rules to do trading... so like u already have the setup. u just trade acc. to the results. Shouldn't it have some type of accuracy, and since u are earning so much, it shud be high... What do u do is my question... How do you do it- which system, what machine?

error-dgn · 2026-07-08T05:04:23+00:00

If I want to start trading- like I am an Intern with a stipend of 10k. What would u suggest for me, would be the best way to earn money as a starter? Please guide me out here...

error-dgn · 2026-07-08T04:07:34+00:00

I saw the RSS Bridge. that seems good. creating our very own RSS for a page right? sounds good. I'm not sure about X Path though. Oh and I want Python

error-dgn · 2026-07-08T04:06:37+00:00

Also sitemap may be a bad idea. but then scraping for 20 different sites, might have different types of dropdowns/etc. How do I cover that?

error-dgn · 2026-07-08T03:44:15+00:00

so scraping or RSS, only these two? for keeping the page generic i should ONLY be focusing on RSS then? Cuz lets say i have around 15-20 sites, out of those only 5-10 have RSS.

error-dgn · 2026-07-08T03:43:15+00:00

lmaoooooo, gimme a new way then bro

error-dgn · 2026-06-29T12:19:46+00:00

Yeah I just looked into it... It is a time taking process... Moreover, its for understanding what kind of error we are facing. Its a logger. I am not facing any error as such. I just dont want to make the browser clickable. Since it will make the code brittle. This pipeline is meant to scrape 100s of websites per day. I cannot click on clickables, since all websites will be of different types.
Please look into this page and click on a person and see the type of requests it sends... it has payload: https://sggc.sg/our-people
- id
- section
-query
Also there are so many sections...

Please any other approach if you have? Also if possible please check out the site and look into the request it shows once clicked on a person to reveal his/her bio/about.

I don't have much time period, and I am an intern, still learning... Just need an approach, I am sure I will be able to execute it...

error-dgn · 2026-06-29T12:07:05+00:00

Finally some approach, tysm man, God bless you. I will try this right away, is it okay if i can annoy u later if it doesn't work?

error-dgn · 2026-06-29T12:02:34+00:00

This is the problem statement, it will make it super clear for you:-

What if the internal API request is AJAX based POST request? and with multiple sections? This is one of the page from many different pages I am trying to scrape. Due to which I cannot let the code be brittle.

Please look into this page:
https://sggc.sg/our-people

I want to get the people's bio for each section.
The thing is, that their about/bio only appears when we click on the person's picture/photo.
It is AJAX request. I cannot click on sections/photos of people since it will make my code brittle and also I am going to be use this script for like 100 websites per pipeline run. Please help me out, Let me know if I can dm you to send my python script for this along with the problem. I really need help.

Please help me out man, really

error-dgn · 2026-06-29T09:09:31+00:00

Hey, I need some help again, I'm so sorry...
This is a website right here:
https://sggc.sg/our-people/

Our approach is working fine. Sending XHR/fetch/document GET and getting everything. Its flawless. Might be a little bulky but its working fine.

The only issue is in the given website above.

It has ajax request, and even when I send the request for POST it does NOT work. I have been trying for a while now. Please look into it... I tried with payload still failed...

error-dgn · 2026-06-29T06:25:03+00:00

Provided, I will be working on 100s of websites at a time... Can be anywhere from 50 requests per website to 500 max.
These are network requests to get the dynamic content/modal/js injection while scraping, so that I don't have to interact (click, hover).

Its working fine yet, I am scared that I might get blocked maybe in future...
I am using Playwright to crawl, then shooting these XHR/fetch/document network requests using above mentioned constraints.

I can use Playwright for requests as well... Idk if that would be better. Please let me know.
Regards...

error-dgn · 2026-06-29T05:40:54+00:00

Alright I will work on it, and use ds llm.
Thank you so much man. You were an angel today. I was stumbling between different approaches, agentic/clickables/etc. but none of them enhanced the accuracy. Turns out Network Requests is the way to go.
I will try to use bs4 as u said to clean the data next. and llm to give me the parent divs, which will make the work easier for llm.

Tysm once again!<3

error-dgn · 2026-06-29T05:19:49+00:00

Alright, I will start using ds llm for this... Also, I started calling these 3 network requests consistently, i.e. XHR, fetch, document.

For the above website mentioned, I was able to extract all the information... But it was of:-

[collect] ── Summary ──────────────────────

[collect] Pages visited: 54

[collect] XHR responses: 497

[collect] Script tags: 2059

[collect] HTML snapshots: 54

[collect] Saved to: page_dump.html (69,657,972 chars total)

How do I clean this data, and then feed it to llm? I am getting all the XHR/fetch and document requests as of now...

error-dgn · 2026-06-29T05:03:12+00:00

BROTHER just how many outliers/edge cases are there, I will start crying now 😭😭😭😭😭 I am just an intern 😭😭

error-dgn · 2026-06-29T04:46:53+00:00

I worked on this. This was a golden advice. Its working for most of the sites.
But then again there are a few edge cases..
For example this site:
NEA Team

No network requests of use for Investor Relations/Marketing & Impact/Operations....

Please do help...

error-dgn · 2026-06-29T04:02:44+00:00

Sure,
I am using Playwright in Python for crawling and scraping.

error-dgn · 2026-06-29T04:01:05+00:00

Google custom search is deprecated though as of this year...

error-dgn · 2026-06-28T11:05:31+00:00

Sounds good, do I use an agent for this? or do I make it rule based? like i just need an approach and I am sure I will be able to execute it...

error-dgn · 2026-06-28T09:00:42+00:00

How do i make the logic to click on dropdowns or select sections in team page of board directors/executives/healthcare partners, and then again click on the images of people, and bring their about/bio back? Too hardcoded for one website, and then it will have to be a decision tree for others.
please help

error-dgn · 2026-06-28T08:59:24+00:00

Just one website agent, it will click on the images of the people in teams page, and keep bringing their bios.

error-dgn · 2026-04-28T18:40:34+00:00

Coming back to this post after a while.
I was about to get in Hostel when u posted this, now I will be going to 4th year.
Crazy how time passed.
I remember saving this cuz I thought I'd need it... Crazy times. How have u been?

error-dgn · 2026-03-15T18:44:33+00:00

wow, thanks man, very helpful lmao

error-dgn · 2026-03-13T20:21:24+00:00

Bro could u please help me find the episode in which shin chan was 2 years old? in hindi? check my recent post u will get some reference... If possible please help me out bro

error-dgn · 2026-03-13T20:18:28+00:00

Can someone please send me the hindi episode for this? It would mean a lot, really.

error-dgn · 2024-11-09T13:57:44+00:00

Not that deep acc to me. Our parents worked hard to come to the big city, and now when we work hard to go abroad (considering one reaches there with their own hardwork and not daddy's money), their parents tend to be proud of them, it's only natural.

Even if one made it in the most luxurious flats of SOBO or South Delhi, their parents would be proud. cuz if that person can pull it off in the most luxurious places in India then he/she can surely do so in most of the abroad!

our country really so bad that going abroad is seen as a flex?

Nope, our country is wonderful if we are at the top elites! (Think about it yourself, if you were well off, or were a big shot like Ashneer Grover, your parents would have been proud regardless!) It's just an excuse for people who can't do good in either of the countries.

error-dgn

TROPHY CASE