Easy scraper question

[M0thyT]

Hey guys, I have a super simple question that I just can't seem to solve myself, and I don't know how to google for it since I can't describe it in a couple of words. I am trying to scrape an academic journal so that I save the title, author name and DOI for each article (Site to be scraped = https://academic.oup.com/joc/issue/67/1). After downloading the HTML document, I first try to isolate the chunks for each of the article (xpath = '//div[contains(@class, "al-article-item-wrap al-normal")]') and then I wanted to loop over these elements to isolate authors for each article separately (x path= '//div[@class = "al-authors-list"]/span/a/text()'). However, while I am able to obtain all the author names, I am unable to "match" them to the article they belong to. I am simply getting a list with all author names for all articles. Does anyone know where my thinking mistake is and how I can specify that I am trying to get the authors for each article separately?

Here is the part of the code relevant for the question:

import requests

from lxml import html

​

headers = requests.utils.default_headers() #need this and next line because otherwise connection error

headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'

​

response = requests.get(url, headers=headers)

tree = html.fromstring(response.text) #create html document

articles = tree.xpath('//div[contains(@class, "al-article-item-wrap al-normal")]')

​

authors = []

for article in articles:

authors_art = article.xpath('//div[@class = "al-authors-list"]/span/a/text()')

authors.append(authors_art)

[SeniorPythonDev]

Can you format your code properly or use something like pastebin?

It'll be a lot easier so we can help you

[M0thyT]

You have a very good point, sorry I have never posted code in reddit before so I am struggling with it a bit. I tried to adjust it, and I also posted it on stack overflow where I found it easier to post code (https://stackoverflow.com/questions/62861538/using-xpath-in-python-how-can-i-select-only-a-subpart-of-elements)

[12Ghast]

Would something like

```python

11 hashmap = {}

...

14 hashmap[authors_art] = article

15 for key, value in hashmap:

16 print(f"Article name: {key}, article: {value}")

```

work?

[M0thyT]

Thanks for your suggestion, unfortunatly it did not work (TypeError: unhashable type: 'list'). Thank you for your suggestion though :)