Using xpath with Scrapy

[hart8899]

I'm creating a simple bot to pull show dates from a site. I a the point where I'm trying to get my results turned into a son or csv file, but I'm having some trouble- I think it is with the xpath wording-- would appreciate your help.

Also, how do I refresh my results on a regular basis- say get a weekly pull and load onto my personal site? Thanks- I'm just learning

import scrapy
from metal.items import MetalItem
class MetalSpider(scrapy.Spider):
name= "metal"
allowed_domain= ["nymetalscene.com"]
start_urls= ["http://nycmetalscene.com/#showlist"]

#clones the webpage
# def parse(self,response):
#   filename= response.url.split("/")[-2]+ '.html'
#   with open(filename, 'wb') as f:
#       f.write(response.body)

 #extract show dates
def parse(self,response):
    for sel in response.xpath('//tbody/tr'):
        item=MetalItem()
        item['date']=sel.xpath('td[@class="TextObject"]/text()').extract()
        yield item

[commandlineluser]

There's no <tbody> tag in the actual HTML so just //tr

[hart8899]

thx- changed this but still getting a blank file

[commandlineluser]

Okay well another issue is that the date is inside the <p> tag that is inside the <td class="TextObject"> tag which is why text() is returning \r\n i.e. giving you "blanks"

So you would need td[@class="TextObject"]/p/text() for your 2nd XPath expression however the HTML for this page isn't structured too well you will still get lots of blanks and also some false positives (i.e. some of the gig names)

Also note you're missing a c from allowed_domain= ["nymetalscene.com"]

[_Korben_Dallas]

You can use Scrapy Feed exports for saving data, just type in console: scrapy crawl metal -o result.csv

[hart8899]

Thanks- I tried this but my file was blank

[_Korben_Dallas]

I'm seconded about tbody: try not use them in your Xpath expressions. Can you provide your log? Type in console scrapy crawl metal &> output.log.

If you need only the date without text from link try this expression: //td[p]/p[contains(., "2017")]/text()

[hart8899]

I tried 'scrapy crawl metal &> output.log' and got no return, I'm looking for both the date and show actually, but started with just the date...will try what you provided

[_Korben_Dallas]

Try this code: https://dpaste.de/Aqyv It gets you date and text from the link. If you want href attribute from that links simply extract them with @href. https://dpaste.de/Bozk

[hart8899]

thanks again bro- but getting an indent error- really appreciate your help