WebRTC iPad Testing #36

kirsle · 2023-08-31T02:35:08Z

kirsle commented

2023-08-31 02:35:08 +00:00

This branch begins to get webcam sharing to work with iPads.

Current Status

With this branch you can get an iPad to share webcam with a desktop Firefox browser in very specific circumstances. The iPad must be sharing its own video before it can open someone else's video (mutual calling seems required).

Log onto chat
Share your own webcam (on iPad side) first
Then click to open somebody else's webcam who is sharing

But it does not (yet) work in reverse: if the Firefox user has their webcam on and they click to open the iPad user's webcam, the iPad user will see a black video screen appear from Firefox but the Firefox user will not see any camera open at all for the iPad.

Things Learned

Apple's implementation of WebRTC is "very modern" and only Promise-based syntax is supported

How this was discovered: when watching the server logs about the WebRTC negotiation messages I noticed:

When a Firefox browser was trying to open my cam, it sent an "SDP" message first followed by a flurry of "Candidate" messages.
When an iPad was trying to open my cam, it would fire off six "candidate" messages to Firefox and then stop.
- When Firefox received these, it logged a browser error about "No remoteDescription" being set.
- The remoteDescription is set in the "On SDP" handlers, and the "On Candidate" one was being fired.
So clearly the iPad was failing to start off the conversation with an SDP message, so Firefox was not ready to receive the candidates.

It turns out that in the WebRTC spec, functions such as setLocalDescription() have deprecated the callback functions in favor of Promises in the modern API. BareRTC used the legacy style but Safari supports only the modern Promise-based: so fixing this, Safari was able to send the SDP message before candidates and get us further along.

// Legacy callback syntax (still supported in Firefox and Chrome)
// setLocalDescription(desc, onSuccess, onFailure)
pc.setLocalDescription(
    new RTCSessionDescription(desc),
    () => {
        this.ws.conn.send(JSON.stringify({
            action: "sdp",
            username: username,
            description: JSON.stringify(pc.localDescription),
        }));
    },
    console.error,
);

// Newer Promise-based syntax: the only kind Safari supports
pc.setLocalDescription(desc).then(() => {
    this.ws.conn.send(JSON.stringify({
        action: "sdp",
        username: username,
        description: JSON.stringify(pc.localDescription),
    }));
}).catch(e => {
    this.ChatClient(`Error sending sdp: ${e}`);
});

Also: not sure if important, but in the original code I gave a new RTCSessionDescription(desc) object to setLocalDescription and in the new code I pass the desc object directly without the RTCSessionDescription wrapper. Firefox and iPad seem happy without the wrapper anyway.

Currently, per this Guide to WebRTC with Safari in the Wild, iPad does not support receive-only or send-only channels: the async video support where an iPad user is not sharing video but wants to watch video, or vice versa, does not seem currently possible.

// iPad test: if we are the offerer and are already broadcasting, add our cam.
// TODO: only do this if the answerer has auto-open videos enabled, because adding
// our video on the offer will force open our video on their side
if (isOfferer && this.webcam.active) {
    this.ChatClient("Adding my camera pre-emptively to the call now");
    let stream = this.webcam.stream;
    stream.getTracks().forEach(track => {
        pc.addTrack(track, stream)
    });
}

The downside: if the offerer adds their video track at the time of connection, then their video will automatically appear on the screen of the answerer when the connection goes through. This is fine if the answerer had the "automatically open their video when they open mine" setting on, so they see the same behavior they would expect. But for users who do not want to auto-open their watcher's camera, the iPad camera would force itself open on their screen anyway, if the iPad adds their own local video to the offer.

Super early on when prototyping BareRTC I discovered the above behavior and removed it because I wanted webcam watching to be asynchronous; but for iPads it may be that asynchronous video is just not going to be supported. (See "Next Ideas", below).

Current Issues

As mentioned above: while mutual video calling seems kosher for the iPad, it doesn't currently work if the non-iPad is the one offering to connect.

Other jank noticed was:

If I (Firefox) close out of the iPad camera on my screen: their video goes away for me, but the iPad still sees and hears me on their side!
- Sort of makes sense because the same would happen with a non-iPad partner: cams are usually asynchronous, me closing your cam doesn't mean you close mine too (unless my require mutual video setting is on).

Next Ideas

Add a dummy data channel to work around the send-only/receive-only channels?

Very early in the BareRTC prototyping, when I wanted async camera sharing to work, I noticed that if the offerer was not attaching their video to the connection, they also would not receive video from the answerer, so they couldn't watch the camera. One attempt at a work-around was to add a Data channel to the WebRTC connection, thinking that by adding "something" useful, the offerer could receive the video from the answerer.

(The solution was to add "offerToReceiveVideo: true" on the offer; then the answerer could send their video correctly, and the data channel code was left dangling and not used).

It may be possible to make iPad's experience better by utilizing the data channel: iPads don't support receive-only channels but maybe by adding a data channel to the connection, it can be tricked into thinking it's a two-way call and the iPad can watch a camera without having their own cam added too.

(Hopefully not) iPads may need to be limited to only specifically bidirectional calls

If the data channel idea doesn't pan out -- it will be very unfortunate but the iPad may just need to be limited to only mutual video calls.

How it may have to look, is:

On the click to view someone's webcam, detect if the user's device is a Safari browser
Then check whether the camera they are opening has the "auto-open my viewer's cam" setting enabled.
If it is, then add the iPad's video to the offer: this will allow the call to connect and the iPad video will auto-open on the answerer's side, working as expected.

But if the iPad is not sharing its own webcam, or the answerer does not auto-open cameras, to give an error message like: you can't watch this camera from an iPad.

This branch begins to get webcam sharing to work with iPads. ## Current Status With this branch you can get an iPad to share webcam with a desktop Firefox browser in **very specific** circumstances. The iPad must be sharing its own video before it can open someone else's video (mutual calling seems required). 1. Log onto chat 2. Share your own webcam (on iPad side) first 3. Then click to open somebody else's webcam who is sharing But it does not (yet) work in reverse: if the Firefox user has their webcam on and they click to open the iPad user's webcam, the iPad user will see a black video screen appear from Firefox but the Firefox user will not see any camera open at all for the iPad. ## Things Learned ### Apple's implementation of WebRTC is "very modern" and only Promise-based syntax is supported How this was discovered: when watching the server logs about the WebRTC negotiation messages I noticed: * When a Firefox browser was trying to open my cam, it sent an "SDP" message first followed by a flurry of "Candidate" messages. * When an iPad was trying to open my cam, it would fire off six "candidate" messages to Firefox and then stop. * When Firefox received these, it logged a browser error about "No remoteDescription" being set. * The remoteDescription is set in the "On SDP" handlers, and the "On Candidate" one was being fired. * So clearly the iPad was failing to start off the conversation with an SDP message, so Firefox was not ready to receive the candidates. It turns out that in the WebRTC spec, functions such as setLocalDescription() have deprecated the callback functions in favor of Promises in the modern API. BareRTC used the legacy style but Safari supports only the modern Promise-based: so fixing this, Safari was able to send the SDP message before candidates and get us further along. ```javascript // Legacy callback syntax (still supported in Firefox and Chrome) // setLocalDescription(desc, onSuccess, onFailure) pc.setLocalDescription( new RTCSessionDescription(desc), () => { this.ws.conn.send(JSON.stringify({ action: "sdp", username: username, description: JSON.stringify(pc.localDescription), })); }, console.error, ); // Newer Promise-based syntax: the only kind Safari supports pc.setLocalDescription(desc).then(() => { this.ws.conn.send(JSON.stringify({ action: "sdp", username: username, description: JSON.stringify(pc.localDescription), })); }).catch(e => { this.ChatClient(`Error sending sdp: ${e}`); }); ``` Also: not sure if important, but in the original code I gave a `new RTCSessionDescription(desc)` object to setLocalDescription and in the new code I pass the `desc` object directly without the RTCSessionDescription wrapper. Firefox and iPad seem happy without the wrapper anyway. ### iPad must share its local video on the initial offer for the call to connect at all Currently, per this [Guide to WebRTC with Safari in the Wild](https://webrtchacks.com/guide-to-safari-webrtc/), iPad does not support receive-only or send-only channels: the async video support where an iPad user is not sharing video but wants to watch video, or vice versa, does not seem currently possible. ```javascript // iPad test: if we are the offerer and are already broadcasting, add our cam. // TODO: only do this if the answerer has auto-open videos enabled, because adding // our video on the offer will force open our video on their side if (isOfferer && this.webcam.active) { this.ChatClient("Adding my camera pre-emptively to the call now"); let stream = this.webcam.stream; stream.getTracks().forEach(track => { pc.addTrack(track, stream) }); } ``` The downside: if the offerer adds their video track at the time of connection, then their video will automatically appear on the screen of the answerer when the connection goes through. This is fine if the answerer had the "automatically open their video when they open mine" setting on, so they see the same behavior they would expect. But for users who do not want to auto-open their watcher's camera, the iPad camera would force itself open on their screen anyway, if the iPad adds their own local video to the offer. _Super early on_ when prototyping BareRTC I discovered the above behavior and removed it because I wanted webcam watching to be asynchronous; but for iPads it may be that asynchronous video is just not going to be supported. (See "Next Ideas", below). ## Current Issues As mentioned above: while mutual video calling seems kosher for the iPad, it doesn't currently work if the non-iPad is the one offering to connect. Other jank noticed was: * If I (Firefox) close out of the iPad camera on my screen: their video goes away for me, but the iPad still sees and hears me on their side! * Sort of makes sense because the same would happen with a non-iPad partner: cams are usually asynchronous, me closing your cam doesn't mean you close mine too (unless my require mutual video setting is on). ## Next Ideas ### Add a dummy data channel to work around the send-only/receive-only channels? Very early in the BareRTC prototyping, when I wanted async camera sharing to work, I noticed that if the offerer was _not_ attaching their video to the connection, they also would not _receive_ video from the answerer, so they couldn't watch the camera. One attempt at a work-around was to add a Data channel to the WebRTC connection, thinking that by adding "something" useful, the offerer could receive the video from the answerer. (The solution was to add "offerToReceiveVideo: true" on the offer; then the answerer could send their video correctly, and the data channel code was left dangling and not used). It _may_ be possible to make iPad's experience better by utilizing the data channel: iPads don't support receive-only channels but _maybe_ by adding a data channel to the connection, it can be tricked into thinking it's a two-way call and the iPad can watch a camera without having their own cam added too. ### (Hopefully not) iPads may need to be limited to only specifically bidirectional calls If the data channel idea doesn't pan out -- it will be very unfortunate but the iPad may just need to be limited to only mutual video calls. How it may have to look, is: * On the click to view someone's webcam, detect if the user's device is a Safari browser * Then check whether the camera they are opening has the "auto-open my viewer's cam" setting enabled. * If it is, then add the iPad's video to the offer: this will allow the call to connect and the iPad video will auto-open on the answerer's side, working as expected. But if the iPad is not sharing its own webcam, or the answerer does not auto-open cameras, to give an error message like: you can't watch this camera from an iPad.

kirsle added 1 commit 2023-08-31 02:35:09 +00:00

28d4768406 WebRTC iPad Testing

kirsle added 2 commits 2023-09-01 00:20:26 +00:00

81a4d0cd0c Merge branch 'master' into ipad-testing

f3a917ec04 Code cleanup

kirsle merged commit 0607fac724 into master

2023-09-01 00:24:08 +00:00

kirsle referenced this issue from a commit

2023-09-01 00:24:08 +00:00

Merge pull request 'WebRTC iPad Testing' (#36) from ipad-testing into master

kirsle referenced this pull request

2023-09-01 01:54:34 +00:00