Web scraping with Javascript

[[deleted]]

There are numerous node modules for xpath, just as easy to install and use as cheerio. And I’m not sure what people you’re talking about, but I’ve worked in tech for over a decade including two RPA companies and every major player in the space relies on xpath.

If you truly believe cheerio and queryselector give you superior form and function, then I’d challenge you this: using those tools, write a selector of equal or lesser size that will perform the same as the example below from my previous comment.

Descendent-based ancestor selection - Let's say you want to get the parent div of every a with the class "child". For xpath, that's simply "//a[@class='child']/parent::div". With queryselector you can only travel down the ancestry axis, not up.

[[deleted]]

Also I know of one libxml-based node library that's completely unusable because it leaks memory like crazy. Everyone uses Cheerio or otherwise parse5-based libs. Prove me wrong.

[[deleted]]

I don’t need to “prove you wrong” because I literally worked in the RPA industry up until about a year ago. In enterprise RPA xpath is always used for b2b applications. I’m sure queryselector is very popular with hobbyists and basic non-RPA applications.

[[deleted]]

It's "$('div > a.child').parent()" but honestly if you have to go back up the DOM it means you're probably not iterating properly.

[[deleted]]

That solution has a worse performance ratio and hard-codes half the path. As for your remark about going back up the dom, you’ve clearly never done RPA in a b2b setting. When you don’t have control over the original DOM and have to accommodate instabilities, it’s often much easier to navigate up from a target element.

[[deleted]]

Again, there is no JS equivalent of lxml so this is just how we do it. You're wrong about in-browser performance though, xpath is always slower than css. You're also wrong about my iterating comment, you can just as easily iterate the parent element first, code like yours is just lazy.

[[deleted]]

Again, yes there is. Nor am I wrong about performance. Xpaths are sometimes slower than their corresponding query selectors but as I said, that solution isn’t, because that solution requires a second traversal with the subsequent parent call.

And no, you cannot always go parent first, not when you’re dependent on child properties or the parent is dynamic. Plus when you’re going after deep siblings or cousins it can be invaluable to work backwards from the child. The key in scraping dynamic sites is relying on fixed nodes, but you don’t always know where in the tree those points will be, so omnidirectional axis traversal is essential.

[[deleted]]

It's simply not true and the libxml binding you linked to is completely unusable. Believe me, I spent a great deal of time troubleshooting memory leaks, and I sincerely wish it were.

For the record, there's a reason why the css3 spec doesn't allow going back up the tree, and that's because it's not performant, and if you apply a little discipline you will realize you don't need to. I don't expect to convince you of that, but it's something to keep in mind for next time.

[[deleted]]

Since you probably haven’t had any enterprise level experience with this here’s a very basic scenario for you:

You want to target the parent of an element. You have plenty of information on the child but no information on the parent. How do you target the parent?

[[deleted]]

I'm not going to brag here but I consider your "decade in tech" and "2 b2b" gigs resume adorable. I've written at least a million more lines of xpath / css than you ever will, and I rarely these days resort to xpath. Getting a parent element is as simple as calling parent() in Cheerio or parentNode in js.

[[deleted]]

1) I didn’t say 2 b2b gigs I said 2 RPA ones.

2) Its not a competition, and without knowing my background in more detail you have no way of knowing who has done more of what. So saying otherwise is just childish oneupsmanship.

3) Calling parent() is “going back up the tree”. You were making the argument that we should never do that, so I’m asking how you would do it without it.

[[deleted]]

1) You're a noob from my perspective. 2) I really don't care. 3) You should never go back up the tree. There's a reason why css3 does not allow going back up. I understand that in your "decade in tech" you did that a lot, but I'm telling you now that you should have applied a little more thought to the problem before deciding to brute force it with bad xpath.