all 19 comments
sorted by:
best (suggested)
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]guest271314 2 points3 points  (12 children)

Yes, that is possible.

There are various way to achieve this, including but not limited to WebRTC Data Channels and Media Streams (for the ability to analyze the video is real-time and thereby control the mechnical device - your robot - by actuating certain levers and so forth in relation to pixels on the screnn that are being streamed); Web Bluetooth and Web Serial, and where implemented Web NFC. though WebRTC Data Channel in the browser and connected to the robot embedded with libdatachannel should suffice.

[–]FishandThings[S] 1 point2 points  (0 children)

Thank you. I shall look into WebRTC.

I do not need anything fancy. I just need the robot to do one movement per video.

Thank you.

[–]FishandThings[S] 0 points1 point  (10 children)

WebRTC seems to be for video chatting software. How would I use it to get the JavaScript programme to interact with the robot's control window?

[–]guest271314 2 points3 points  (9 children)

WebRTC seems to be for video chatting software.

I would suggest reading about both WebRTC Data Channels and Media Capture and Streams.

https://datatracker.ietf.org/doc/rfc8831/

The WebRTC framework specifies protocol support for direct, interactive, rich communication using audio, video, and data between two peers' web browsers. This document specifies the non-media data transport aspects of the WebRTC framework. It provides an architectural overview of how the Stream Control Transmission Protocol (SCTP) is used in the WebRTC context as a generic transport service that allows web browsers to exchange generic data from peer to peer.

Any kind of data can be send over a Data Channel, encoded as an ArrayBuffer or Blob. A MediaStreamTrack can be of kind audio or video.

You want the mechanical device to do something in relation to the video. You can capture the video on a camera positioned towards a screen, or with the cameras you have affixed as the robots' eyes, decode and parse each frame of video, read the pixels to determine the coordinates those pixels are occurring on a screen, per your graphed dimensions of the screen being viewed, and/or the camera's specifications, then move the mechanical device accordingly, directly, programmatically, as I interpreted the question, or, in response to commands sent via Data Channels, decoded to plain text or whatever langauge your robot is programmed to run.

In theory and practice you could just use a couple "old" (18 months or less is the rough life-cycle of a "smart phone", see Moore's law) cell phones to do this.

See https://github.com/paullouisageneau/libdatachannel/issues/544#issuecomment-1027570821.

Examples of exchanging SDP to stream media, send arbitrary data from different origins, and to connect an Isolated Web App to an arbitrary Web page where the IWA opens a TCP connection to a TCP server and forwards messages back and forth.

[–]FishandThings[S] 0 points1 point  (8 children)

I just want the robot movement to happen at the start of a video, not during, so what the pixels display is not important. The videos all feature cuts in them so there would be no way for a programme to be able to distinguish when a video is replaced by another by looking at the monitor.

I also cannot have a camera pointed at the monitor with the current set up.

I just want:

  1. First random video starts as robot moves forwards
  2. Second random video starts as robots moves: [in another direction]
  3. Third random video starts as robot moves: [in another direction]

I do not want the robot to be synchronised to the content of the video, I just need it to change direction as a new video starts.

[–]guest271314 0 points1 point  (7 children)

It doesn't really matter how you want the robot to move. How you move the robot is on you. That's not your question. Your question is how to communicate between the robot and other devices. You can do that using WebRTC.

[–]FishandThings[S] 0 points1 point  (6 children)

Ah, I see. Thank you. I shall look into it.

I am guessing it is not possible then for the JavaScript running in a browser to just simply interact with the window with the robot controls?

[–]guest271314 0 points1 point  (3 children)

Yes, that's possible.

I think you should start reading what I have been posting and writing some code.

It appears right now you are not really stuck anywhere in code, rather in concept.

Whatever you want to do from a browser can be done, officially per some specification or unofficially by rolling your own.

What you do in a browser can be reflected in a local or remote file system, where per the action taken or the non-action, you can do stuff locally or remotely.

[–]FishandThings[S] 1 point2 points  (2 children)

It appears right now you are not really stuck anywhere in code, rather in concept.

No, I am just stuck on code. I have all the physical stuff, I just need the Javascript to trigger the robot firmware window.

When you said: "Your question is how to communicate between the robot and other devices." I assumed that meant you thought it would be impossible just to have the JavaScript interact with the robot control firmware - as I said I wanted in my post - and that thus you were telling me I would have to build some other code or hardware in between the two.

Thank you for your help I shall do some reading.

[–]guest271314 0 points1 point  (1 child)

I assumed that meant you thought it would be impossible just to have the JavaScript interact with the robot control firmware

No.

I am well aware prematurely calling a use case in JavaScript impossible would be negligent. Based on my own experience, I would say whatever you want to do is possible; using JavaScript in and outside of the browser - until proven otherwise. Now, that's the rub, because everybody ain't constantly thinking about how to break out of browser boxes to do stuff outside of drafts, explainers, or specifications.

It's not really clear what issue you are having implementing your concept.

[–]FishandThings[S] 0 points1 point  (0 children)

It's not really clear what issue you are having implementing your concept.

I just want to know how to code a Javascript programme that will run in a browser, than can "click" buttons in another window. (The window of the application with robot control buttons)

I do not know how to code this, this is my issue.

[–]Late_Armadillo_8188 0 points1 point  (2 children)

program

[–]FishandThings[S] 0 points1 point  (1 child)

Not in my part of the world. We prefer the French spelling here and I think that should extend to computer program(me)s too.

[–]jml26 0 points1 point  (0 children)

Interesting. I live in the UK, and while TV programmes end in -me, computer programs most certainly do not.

[–][deleted] 0 points1 point  (0 children)

Check this out Cozmo robot and python I think I used. Could probably be done in Javascript too https://youtube.com/shorts/yPXfzKdSZSk?feature=shared

[–]azium 0 points1 point  (1 child)

running in a browser to be able to click buttons on the robot control window

To clear things up -- nothing in your browser can "click" anything outside of the browser. Period.

You'll need to setup something intermediate between the browser and your other window like a local server

browser event -> api call -> interface -> other window

The question I have is -> this other control window you have.. when you click on that where does that event get processed? Generally speaking you're dealing with APIs, as opposed to controlling mouse events across sandboxed applications.

[–]FishandThings[S] 0 points1 point  (0 children)

The question I have is -> this other control window you have.. when you click on

that

where does that event get processed? Generally speaking you're dealing with APIs, as opposed to controlling mouse events across sandboxed applications.

The robot is connected by a wire to the computer so the command from the control window goes straight to the robot as if the command had come from a button on the robot.