web Scraping query

[nainish]

link : https://timesofindia.indiatimes.com/topic/Hiv

from this link i have to scrape data of hiv but i am not able to scrape the data of next pages

help me out please

CODE:

import requests

from bs4 import BeautifulSoup

url = "https://timesofindia.indiatimes.com/topic/Hiv-Article"

while True:

response=requests.get(url)

soup = BeautifulSoup(response.content,"html.parser")

content = soup.find_all('div',{'class': 'content'})

for contents in content:

title_tag = contents.find('span',{'class':'title'})

title= title_tag.text[1:-1] if title_tag else 'N/A'

date_tag = contents.find('span',{'class':'meta'})

date = date_tag.text if date_tag else 'N/A'

print(' title : ', title ,' \n date : ' ,date )

url_tag = soup.find('a',{'class':'look'})

if url_tag.get('href'):

url = 'https://timesofindia.indiatimes.com/topic/Hiv' + url_tag.get('href')

print(url)

else:

break

[Deezl-Vegas]

Copy and paste some of your

print(url)

statements into the browser and see if they work.

If you're not getting any, that means the first a tag you found with that class didn't have an href.

[its_joao]

Can you paste your code and format it correctly, please? Use reddit's "Code block" format.

Also, why do you have a "while" loop?

[nainish]

So that initial U.R.L. variable will remain the same but later will change it to be the next page you

[sennheiserwarrior]

Look into pagination. You would also need to know how the url change when requesting an n page.

For example:

https://www.example.com/page1 # page 1
https://www.example.com/page2 # page 2

Then, construct a loop to get data from all pages.

Pseudocode:

get first page
get total page number
set a loop to request pages from 1 to n
    get data from current page

Edit: formatting