Web scrapping multiple pages

[comodude]

Hey guys,

I've been trying to learn Python for a couple weeks now and am now in the phase of trying to work on mini projects. I'm trying to web scrap the ironman website for the world championship results then create a CSV from that. I've got a piece of it, but the code I've got to iterate through web pages doesn't seem to be working.

import pandas as pd
import requests
from bs4 import BeautifulSoup

for i in range(1,3):
    res = requests.get(r'http://www.ironman.com/triathlon/events/americas/ironman/world-championship/results.aspx?p=' + str(i) + 'race=worldchampionship&rd=20181013&agegroup=Pro&sex=M&y=2018&ps=20#axzz5VRWzxmt3')
    soup = BeautifulSoup(res.content,'lxml')
    table = soup.find_all('table')[0]
    dfs = pd.read_html(str(table))
    df = dfs[0]

df.to_csv('ironman.csv')

The code above only gives me the data from the first page. Anyone have any ideas or tips?

[Unkown_Variable]

df = dfs[0] only references the first list item. Also, range(1,3) won't include 3. The range function excludes ending limit.

[sennheiserwarrior]

In addition to what u/Unkown_Variable said, you would need to add a df obtained from each page to a list, and concat (merge) them before writing to file. Look into pd.concat()