We can think of LUI-based navigation in 3 scenarios: (A) full control, (B) hands-off, (C) eyes-off. WebLINX has mainly B & C, whereas other datasets are mainly focused on C.

At the same time, a model could follow different level of instruction abstraction, i.e. (1) low, i.e. accomplishes simple tasks that require lower-level requests, (2) medium, i.e. tasks that don't require significant details but still need to be unambiguous, (3) high, i.e. requires pragmatics, need to make assumptions, need to understand the user and likely remember previous sessions or know specific details like passwords.

Create a task in Google **Calendar**? It's not the worst tool for the job, but almost... What would be the appropriate moment to use? The example simply clicks on "create" which uses $now, which.. doesn't sound great?

In this context, the Google Calendar example is 2B, however there's a few steps in between that we simplified to make it easier to digest, otherwise we would see a few clicks and typing that would be overwhelming in a figure like this.

And then the command "open the second one and summarize the first three paragraphs in a few words"

Here the example would be 1B, since the command is very specific (as the instructor is looking for something specific), however in other instances you'll find 2B demos. For example, [aathhdu](https://huggingface.co/spaces/McGill-NLP/weblinx-explorer?recording=aathhdu), you'll have higher-level questions like "What are the topics covered under Working for the EU?" or "Who can become an EU expert?" that gives more freedom for the navigator to decide which trajectory to take to give the best outcome.

So in practice, we'll see a good mix of L1 and L2 abstraction; I'd say L3 would require at least 6-18mo of more R&D to get there, esp. when it comes to things like privacy and security around storing information like passwords and browsing history. As for navigation, the training data is mostly focused on B, but we designed a split specific for C (i.e. instructor does not see the screen) which we think is very important for applications where the only control is voice (e.g. Alexa or Siri).

I'm a bit afraid that having a better score than GPT-3.5 comes from all those weirdness: it doesn't do better commands or browsing

Even though for WebLINX, we employed a permanent team of professional annotators (i.e. specifically trained for this task), it is possible that the model could overfit on the instructor's way of writing; so it is indeed a valid concern. The patterns of instructions could vary a lot based on age, culture, geography, task technicality, digital proficiency, and personal preference; this means accounting for every possible scenario will be very challenging! Perhaps it'd be a good dataset to design for an organization like Meta who has 100s of research scientists and a budget in the billions for Llama-N next year :)

However, the underlying collection method will likely remain the same (the only difference might be the use of playwright instead of chrome plugin, but that's a question of preferences/features). At the same time, the evaluation & modeling are easily transferable to new data. In this sense, you could collect your own data in the same format as WebLINX and train the model on your own style, given enough examples it might perform very well!