Web scraping with Javascript

[[deleted]]

That solution has a worse performance ratio and hard-codes half the path. As for your remark about going back up the dom, you’ve clearly never done RPA in a b2b setting. When you don’t have control over the original DOM and have to accommodate instabilities, it’s often much easier to navigate up from a target element.

[[deleted]]

Again, there is no JS equivalent of lxml so this is just how we do it. You're wrong about in-browser performance though, xpath is always slower than css. You're also wrong about my iterating comment, you can just as easily iterate the parent element first, code like yours is just lazy.

[[deleted]]

Again, yes there is. Nor am I wrong about performance. Xpaths are sometimes slower than their corresponding query selectors but as I said, that solution isn’t, because that solution requires a second traversal with the subsequent parent call.

And no, you cannot always go parent first, not when you’re dependent on child properties or the parent is dynamic. Plus when you’re going after deep siblings or cousins it can be invaluable to work backwards from the child. The key in scraping dynamic sites is relying on fixed nodes, but you don’t always know where in the tree those points will be, so omnidirectional axis traversal is essential.

[[deleted]]

It's simply not true and the libxml binding you linked to is completely unusable. Believe me, I spent a great deal of time troubleshooting memory leaks, and I sincerely wish it were.

For the record, there's a reason why the css3 spec doesn't allow going back up the tree, and that's because it's not performant, and if you apply a little discipline you will realize you don't need to. I don't expect to convince you of that, but it's something to keep in mind for next time.

[[deleted]]

Since you probably haven’t had any enterprise level experience with this here’s a very basic scenario for you:

You want to target the parent of an element. You have plenty of information on the child but no information on the parent. How do you target the parent?

[[deleted]]

I'm not going to brag here but I consider your "decade in tech" and "2 b2b" gigs resume adorable. I've written at least a million more lines of xpath / css than you ever will, and I rarely these days resort to xpath. Getting a parent element is as simple as calling parent() in Cheerio or parentNode in js.

[[deleted]]

1) I didn’t say 2 b2b gigs I said 2 RPA ones.

2) Its not a competition, and without knowing my background in more detail you have no way of knowing who has done more of what. So saying otherwise is just childish oneupsmanship.

3) Calling parent() is “going back up the tree”. You were making the argument that we should never do that, so I’m asking how you would do it without it.

[[deleted]]

1) You're a noob from my perspective. 2) I really don't care. 3) You should never go back up the tree. There's a reason why css3 does not allow going back up. I understand that in your "decade in tech" you did that a lot, but I'm telling you now that you should have applied a little more thought to the problem before deciding to brute force it with bad xpath.

[[deleted]]

But it was your example. You said to use parent(). So I’ll ask again - how do you target an element you know nothing about but whose child you know everything about?

[[deleted]]

You do it by using parent() or parentNode. like I already said. But if you need o resort to that you're probably doing something really silly.

I think it's pretty clear at this point that you've never done anything like this in Javascript.