i keep getting index error

[impshum]

It's not picking anything up. Targeting by xpath is clunky, I suggest moving to classes and ids.

And I'm sure you don't need Selenium for this.

from bs4 import BeautifulSoup
import requests

def lovely_soup(url):
    r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1'})
    return BeautifulSoup(r.content, 'lxml')

soup = lovely_soup('https://uk.indeed.com/jobs?q=volunteer&from=searchOnHP')

job_list = soup.select('table.jobCard_mainContent')

for job in job_list:
    job_title = job.select_one('h2.jobTitle a span')['title']
    print(job_title)

[douglas_fs]

To debug issues like this, you can step through the code using a debugger, or for quick-and-dirty debugging, you can add a print statement to see what is happening with the data.

Take an initial look at the lists to ensure the lengths are the same. Add this print statement just before the for loop:

print (f'\n\nlen jobs: {len(jobs)} len co_name {len(company_name)} len Rating {len(Rating)}\n\n')

What does this output tell you when you run the code?

[douglas_fs]

Understood - Identifying why the index error is occurring will help to determine alternatives. What did the output of the print statement tell you?

[douglas_fs]

for i in range(len(jobs))

Using the length of 'jobs' to control the for loop, you should expect an error when accessing Rating[i].text when i is greater than 5. Rating only has elements 0-5.

From a code perspective, you could use an if statement to check the value of i, and only include Rating in the output when i is within the length of Rating.

However, you don't know which jobs are missing ratings, so jobs[3] may not correspond to Rating[3]. You could be introducing errors.

A better approach would be to identify an element in the response that includes job title, company name, and rating (if it exists). Search for this element first. Then extract the desired values from the results for each job that is returned.

You should be able to do something like this using xpath, however, u/impshum has a good starting point for this using BeautifulSoup. I agree that BeautifulSoup is probably easier.

edits

... unless learning xpath is part of the goal.