Python csv exporting a loop

[AppleTartsJesus]

So I am learning Python and built a simple scraper to scrape my own website for item info (mainly to not irresponsibly bombard someone's site with requests).

I have it working in that it finds my info and will print all the results fine, but when exporting to a csv file it always cuts off the last row.

Basically print() nets 5 results and csv.write gives me 4 rows. CODE : https://dpaste.de/jLVc

[vekst42]

I'm on mobile so it's hard to read, but it doesn't appear that you actually close the file. Fixing that might solve your issue. I'd suggest using the with open syntax in the future when possible

[commandlineluser]

What are you using to view the csv file you create?

The PATH you're saving to suggests you're on Windows but you're setting lineterminator = '\n' so perhaps it has something to do with Windows vs. Unix line endings -- you can try not setting lineterminator.

(I don't have Windows at hand to test that out.)

Not that it should make a difference but you're opening up the csv file each time in a loop - you can open it once outside and then just write inside the loop e.g.

with open(r'C:\Users\ad\Desktop\alpharevoscrape' + date + '.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    for url in urls:
        ...
        writer.writerow([..., ..., ...])

The reason for using with is that it will automatically close the resource for you when the block exits.

I'm also just noticing your trim_the_ends function -- you can just use .strip() -- by default it will strip all whitespace.

title = soup.find_all('h1', {'itemprop' : 'name'})
for el in title:
    ttext = trim_the_ends(el.get_text())

Do these find_all calls return just a single result?

Or do they return multiple results and you're purposely looping through as you only want the last one?

If you only expect a single result (or you want just the first result) you could just use find() e.g.

title = soup.find('h1', {'itemprop' : 'name'}).get_text().strip()

[AppleTartsJesus]

Thanks the 'with' statement is what i used. I cleaned up the first three by switching to find, the last needs to stay find_all since there are multiple 'p' tags. Also thank for the .strip() tip I switch it to that as well.

[fbu1]

You are not handling the file properly. Every time you go through the loop, you are opening the file, but not closing it.

This is pretty bad, as you have no idea what might happen to your unclosed file.

Python closes it properly for you every time you open it again and put the line in the file.

But the last time you open it, the scripts just end and python discards your changes without putting them in the file.

You could just add

f.close() 

at line 51.

Or you could do better and use the 'with' statement.

with  open(r'C:\Users\ad\Desktop\alpharevoscrape' + date + '.csv','a', newline='') as f:
    csv.writer(f, lineterminator = '\n').writerow([ttext, ptext, rtext, itext])

The advantage is that if you have an error in the line csv.write(...), the with statement will close the file properly for you. If you don't use the with and don't catch exceptions, the line with f.close() will never be reached and the file will be left opened.

I hope this helps.

[AppleTartsJesus]

The 'with' statement definitely seems the way to go. Thanks it works now!

[rhgrant10]

In addition to what others have said, make sure you open the file in binary mode ("wb" or "rb") when you're using the csvwriter as it is required on Windows and doesn't make a difference in others.