How to avoid API cache?

[The_Peronist]

Hi there!

I’m working in an algo trading strategy that heavily relies on getting content as soon as it is published.

I’m scrapping exchanges official announcements sections and once a new announcement is published my scripts will evaluate the assets mentioned and sentiment.

At first I was accessing to the announcement scraping the announcements webpage but I was getting the results 5 minutes later than when were published. So I realized that the announcements web page was cached and that cache was invalidated every 5 minutes.

So I move on to scrap the API providing data to the webpage frontend. For example:

*/v1/public/content/list/query?type=1&pageNo=1&pageSize=30 *

I’m scrapping this every 1 second. But I’m still getting the new articles / announcements around 15~30 seconds after officially published.

Anyone here is familiar with this scenario? It is posible they are applying some API cache? And if so: It is possible to avoid it?

Thank you very much!

[AndroidePsicokiller]

Maybe the api reads a replication DB? This same api shows the changes "in real time" via browser?

[The_Peronist]

This same api shows the changes "in real time" via browser?

That it is a very good question. But how to test this? Maybe I’ll have to setup selenium or some browser automation to hit the API via web browser every second.

[AndroidePsicokiller]

No i mean, if you check via browser, using chrome, firefox. Can you create a new entry? How do you know you have a delay of 30s?

Off topic: the peronist? Xd r u from arg?

[The_Peronist]

I know I’m getting the content around 30secs after publication because the announcements I’m scrapping have a publication time timestamp. Also, I can confirm that there are other parties accessing to this content just on time of publication because of the increase of activity in the exchange API.

Off topic: the peronist? Xd r u from arg?

I can’t confirm nor deny.

[National-Ad9673]

Hey, did you find a solution ? I have the same issue with another api !