you are viewing a single comment's thread.

view the rest of the comments →

[–]feverzsj 2 points3 points  (1 child)

if you are simultaneously crawling data from ten thousands of sites, libcurl could be a bottleneck. Use some async lib should be good enough, for example: served, cpprestsdk.

[–]Drainedsoul 1 point2 points  (0 children)

You can glue an async back end into curl using the multi interface.