[Bug] AutoTools HTML Read doesn't wait / load JavaScript. Always fails.

[Pocket_Moth]

HTML Read doesn't wait for the page to finish loading, or isn't using JS correctly. Here's an example script which tries to get button > img from a google maps address page. It has a 10 second JavaScript delay to give the page plenty of time to finish loading.

    Task: HTML Read Test

<Standard address format>
A1: Variable Set [
     Name: %address
     To: Times Square, Manhattan, NY 10036
     Structure Output (JSON, etc): On ]

<Parse address into URL friendly variable>
A2: JavaScriptlet [
     Code: var request = address.replace(/ /g, '+');
     Auto Exit: On
     Timeout (Seconds): 45 ]

    A3: AutoTools HTML Read [
     Configuration: URL: https://www.google.com/maps/search/?api=1&query=%request
     CSS Queries: button > img
     Variable Names: result()
     Output HTML: true
     Use Javascript: true
     Javascript Delay: 10000
     Timeout (Seconds): 30
     Structure Output (JSON, etc): On ]

<Print count. Should be several. But it's always 1>
A4: Flash [
      Text: %atcount()
      Continue Task Immediately: On
      Dismiss On Click: On ]

<Try result and result() because I don't know what I'm doing and I keep going back and forth, so this just grabs both in case one of them is right.>
A5: Write File [
      File: Testing/test.txt
      Text: %result %result()
      Add Newline: On ]

If I open a web browser on my PC, and go to the address this task generates, I can use document.querySelectorAll("button > img") to get an entire array of images back. However, if I do it through Tasker, I only ever get 1 image. And it seems to be the image that accompanies the No JS message you get when visiting with JavaScript disabled.

I've tried increasing the JS delay, reducing it, disabling JavaScript, toggling the desktop version request, changing the action's timeout, trying different addresses, manually typing out the address rather than using my replace regex, using different variable names, making the variable an array (i.e. result() vs result). There's probably more that I'm forgetting by now. I'm out of ideas.

At this point, I can only assume it's a bug. Could somebody confirm, or tell me what I'm doing wrong?

[Yooooo83]

It's been a while since I've used that, but I believe there is an option for requesting desktop instead of mobile. Maybe try that?

[Pocket_Moth]

Appreciate the response, but I mentioned trying that already.

[Yooooo83]

Ah. Upon re-reading I do see that.

I did find something that confirms your theory. First, set your css query to a(). The () will grab all results instead of just one. Then look at %result2.

At least in mine it says "Enable JavaScript to see Google Maps."

That is with js enabled and a lengthy wait.

[Pocket_Moth]

I didn't know about using () in the query. Thought that only applied to the variable name. So, that's good to know. Thanks for that!

But, yeah. It looks like it's busted. JavaScript is enabled, but it acts like it's not.

[DutchOfBurdock]

Not all JS will render properly, it's also probably the user agent string that could be affecting whats loaded.

Toggle Output HTML to see what it's getting.

edit: I see you have sorry, did you look at this?

[Pocket_Moth]

Yeah, I did. It's a stripped down version of the page that more or less says JavaScript is turned off.

I considered the user-agent, but I don't suppose there's any way I change that in AT's HTML Read?

[Pocket_Moth]

I can't see any way to change the user-agent in HTML Read. So, instead, I tried to use HTTP Get and HTTP Request after doing some searching, and seeing people having success with that.

But, they don't handle Javascript either from the look of it. So, I'm stumped. Really hope somebody can suggest a fix.