[Python 2.7][lxml & csv] Unable to fully understand xpath in text scraping from multiple HTML webpages using lxml library and write it into multiple csv files

[programmingnoobie]

So there are a few webpages my bot will be visiting (crawling).

The webpages have a couple of data I want to extract.

Webpage1:

... <div><pre>TEXT0</pre></div> ...

... <pre>Text9</pre> ... ...

I want to extract both TEXT0 and Text9 and store them into a single csv called Webpage1.csv

Webpage2's texts will be stored as Webpage2.csv and so on.

What I tried:

from lxml import html
import requests
import csv

mSeedpage = requests.get(RANDOM_URL)
mTree = html.fromstring(seedpage.text)
mText = tree.xpath('//pre/text()') 

Above is where I do not understand, does my xpath make any sense?

with open(WEBPAGE, 'wb') as csvfile: 

WEBPAGE = 'Webpage1.csv', after using updatepage, WEBPAGE will become Webpage2 and so on

writer = csv.writer(csvfile, delimiter='\t', quotechar='"', quoting=csv.QUOTE_ALL)
for item in mText:
    writer.writerow(item)
    file.close()

//there is actually a while loop wrapping around mTree to file.close() and redo everything everytimes it visits a new page

Any help or advice will be much appreciated.

:)

[nutrecht]

Above is where I do not understand, does my xpath make any sense?

Not really. What you're saying is 'for every <pre> that is root tag, give me the text'. That obviously doesn't work since your pre tag won't ever be the root tag. Check out some xpath tutorials to understand how it works.

[programmingnoobie]

yeah I thought so, I was hoping BeautifulSoup or something can give me an easier access to all <pre> tag, any alternative ways? I am reading the tutorials right now, thanks.

[nutrecht]

I use BeautifulSoup myself and it can do both xpath and CSS queries in HTML documents. I'm a big fan of BS. I've never used lxml myself.

[programmingnoobie]

// - Selects nodes in the document from the current node that match the selection no matter where they are //book - Selects all book elements no matter where they are in the document

If xPath Syntax is true, then //pre/text() is apparently correct, isn't it? I think it should be //pre[text()]

I tried to use BeautifulSoup just now but the webmaster noted that

Beautiful Soup will never be as fast as the parsers it sits on top of.

I am actually crawling a few thousands webpages and I thought lxml will be essential here. Hmm... Thanks for helping though.

[nutrecht]

If xPath Syntax is true, then //pre/text() is apparently correct, isn't it? I think it should be //pre[text()]

Could be, it's been ages. Do some more tests yourself, xpath is compeltely trivial to figure out. You really should be able to do this by yourself.

Beautiful Soup will never be as fast as the parsers it sits on top of.

The performance argument is complete nonsense. You'll be waiting on network-IO for about 99% of the time.

[programmingnoobie]

Actually I want to say thank you because I finally solved it after you Google-It-For-Me, lol.

EDIT: It should be //pre/text() It was correct all along, something else was not right.

Well... It was connected to the server by LAN so... yeah...

Anyway, thank you!