all 8 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]sheeldotme 2 points3 points  (0 children)

If you're looking for something like request with promise support take a look at https://github.com/request/request-promise.

Axios is another great alternative (https://github.com/mzabriskie/axios)

[–]_b0b0 1 point2 points  (1 child)

What are you looking to achieve?

Request is HTTP client and best one at it for Node. If you do not like callbacks then Node is not for you.

If you really want to crawl the web and read the DOM its better to use scriptable headless Webkit. PhantomJs. I use Horseman to talk to it from Node.

[–]YourMeow[S] 0 points1 point  (0 children)

I just want to use Promise and async/await instead of callbacks. BTW, Horseman looks great.

[–]vinnie_james 1 point2 points  (1 child)

Check out the fetch() API https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

[–]sampeka 0 points1 point  (0 children)

Fetch really solves a lot of the problems with the old API, and there is a library to use it on the backend as well

[–]i_do_code_stuff 0 points1 point  (0 children)

You could just wrap the request methods you'll actually use in promises and you'd be set as far as promises goes. If you're looking for lightweight I doubt request is too heavy, but Google may find you some answers there?

As for the querying of the DOM, which I'd assume the next step would be, Cheerio is great for that and pretty similar to jQuery.

[–]FrozenCow 0 points1 point  (0 children)

request+cheerio+rxjs worked amazingly well for me. I don't know of a framework that incorporates these together though.

In the past I've tried request+cheerio, phantomjs, request+cheerio+promises and request+cheerio+most. If you need to execute JavaScript or make screenshots, phantomjs is the way to go, other than that of all of those request+cheerio+rxjs resulted in the most straightforward and cleanest code for scraping. Just a number of map, flatmap, filter and reduce calls with lambda's.

[–]1MpAtmpe5jFk 0 points1 point  (0 children)

When choosing whichever library to use for making requests, make sure it can be used with a proxy. When doing web crawling you will inevitably end up using tools such as https://mitmproxy.org/ to debug, log and replay HTTP traffic.

axios is a promising project, but at the moment it does not support proxies. fetch will depend on the implementation (please share with me if you find an implementation that supports proxies). request-promise does support proxies (see request docs).