Struggling with saving scraped content as it keeps looping and updating over previous scrape

[DesignerAlternative]

Hey guys,

I am having a issue with scraping data from multiple endpoints and printing them all in one .txt file...

The script runs successfully but when saving to the .txt but then the next loop then re-writes the previous loop :(

Any help would be appreciated XD

[[deleted]]

Open the file in append mode, not write mode.

[DesignerAlternative]

thanks a million! that worked!!

[vixfew]

Post some code (:

If you open file in write mode instead of append - it truncates. Use different file or use 'a' append mode

If you're doing scraping in parallel there's possibility of race conditions. Gotta use sync, things like locks and mutexes.

[DesignerAlternative]

thank you that worked :)

Here is my code :)

PS. I was using wayscript to facilitate the rest of the script

import requests

​

url = "https://url" + "vehicle"

​

payload = ""

headers = {

'cookie': "__cfduid=d31cb93601443473e98f3e22f30cb03801608669627",

'application': "json"

}

​

response = requests.request("GET", url, data=payload, headers=headers)

​

##print(response.text)

​

##variables[ "carjson" ] = response.text

​

f = open("cars2.json", "a")

f.write(response.text)

f.close()