Webkit browsers can't share video (Safari, all iOS browsers) #14

New Issue

kirsle · 2023-04-10T04:26:34Z

kirsle commented

2023-04-10 04:26:34 +00:00

Browsers using the Webkit engine all seem to have difficulty connecting to peers to enable webcam sharing. Some browsers that currently have issues:

Safari on macOS
DuckDuckGo browser for macOS
Every web browser on iPad/iPhone (because every browser on iOS is really Safari, aka Webkit, regardless of branding)
GNOME Web and Midori for Linux

What's known about the issue

A user on Safari is able to turn on their own camera (and so their cam icon lights up blue for others to click), but when others click on their camera, nothing happens; similarly, when the Safari user clicks on somebody else's camera, nothing happens.

From my Firefox browser on Linux when I try and connect to the Safari user, my console says that peer-to-peer WebRTC connection fails and there is no TURN server to carry the video instead (the latter of which is expected - the chat room does not have a TURN server). update: I added TURN server support but it has not improved the situation.

I have not seen what the error is on Safari's side. At one point I saw a console error that Safari was unhappy that the ordering of lines for the WebRTC handshake was being altered (because the server parsed and re-serialized the json messages), but fixing that (passing the messages verbatim) did not resolve the issue, and Safari on a macOS Ventura virtual machine logged no error at all in the console for me.

Linux Webkit Browsers

In both GNOME Web (Epiphany) and Midori (Xfce's minimalist browser), opening a user's webcam fails because seemingly these Webkit browsers do not support WebRTC at all? They give a console error: "Variable not found: RTCPeerConnection"

Things to do

The front-end should more safely check whether RTCPeerConnection is available in the JavaScript API, and give a nice ChatClient error message when it is not (rather than just "not doing anything" when clicking on a webcam)
In case the peer-to-peer WebRTC connection fails, this error should be caught and something sent by ChatClient so the user can get feedback rather than nothing happening.

Additional Context

How webcam sharing generally works in BareRTC (see also: chat server protocol):

When a user turns on their camera to make it available for sharing
- The frontend calls navigator.mediaDevices.getUserMedia() to prompt the user to share their webcam and microphone.
- On success, their own loopback preview video appears on the page: a <video> element directly sourced by their webcam stream, with its audio muted so you don't hear your own microphone echo from it.
- The front-end posts a "presence update" message (sendMe()) to the chat server to say: my video is active, and whether it's marked NSFW. This is what makes the blue button appear to other chatters that they could click on to try and connect and see your video.
- Code is around here in startVideo(): https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L989
- This much works on Safari browsers: it isn't doing any WebRTC peer connection stuff yet, just showing your local video feed to your browser.
When a viewer clicks on your blue video button to see your camera
- The openVideo() function
- The viewer sends an "open" message to the chat server which the chat server forwards to the broadcaster to let them know that the viewer wants to connect.
  - When the broadcaster receives it, this is what triggers the "Soandso has opened your camera." message
  - Note: if you're on Safari you might see this message, but the viewer doesn't actually connect and doesn't go on your Watching list, and vice versa: if a Safari user clicks your camera, you see this notification but they don't actually go on to connect.
- The WebRTC function is here: startWebRTC() - both the offerer (the one wanting to connect) and the answerer (the one sharing video) will go into this function.
  - At the chat protocol level: the offerer sent an "open" to the server, and the server echoes the "open" back to the offerer here: onOpen()
  - To the answerer, the chat server sent a "ring" message in response to that open, handled here: onRing()
The two peers exchanged WebRTC negotiation messages
- In here is where the Safari issue occurs.
- Using the WebSocket connection with the chat server, the chat server acts as the "signaling server" to basically allow the two peers to exchange messages to each other and negotiate features and how they'll establish the peer connection.
- For this stage the chat server just relays the ICE candidate messages back and forth. Usually, the two peers exchange enough info to be able to connect to each other but Safari browsers fail in this step.
The two web browsers connect peer-to-peer and add a video stream to that connection, and the broadcaster's video appears on the requestor's screen.

At one point in step 3 I saw Safari desktop's error console complain that the ICE messages (json blobs) were rearranged from how it sent them (because the chat server would parse the json message, and re-serialize it when sending it to the peer - the data was the same but keys got reordered, but I updated it so the chat server just forwards the json blob verbatim without messing with it - this resolved that Safari error, but then Safari still doesn't connect and I couldn't get another error message from it yet).

Browsers using the Webkit engine all seem to have difficulty connecting to peers to enable webcam sharing. Some browsers that currently have issues: * Safari on macOS * DuckDuckGo browser for macOS * Every web browser on iPad/iPhone (because every browser on iOS is really Safari, aka Webkit, regardless of branding) * GNOME Web and Midori for Linux ## What's known about the issue A user on Safari is able to turn on their own camera (and so their cam icon lights up blue for others to click), but when others click on their camera, nothing happens; similarly, when the Safari user clicks on somebody else's camera, nothing happens. From my Firefox browser on Linux when I try and connect to the Safari user, my console says that peer-to-peer WebRTC connection fails and there is no TURN server to carry the video instead (the latter of which is expected - the chat room does not have a TURN server). **update:** I [added TURN server support](https://git.kirsle.net/apps/BareRTC/commit/4be18ea3a2e221e252a5e6e97b5d520bd28aa7f1) but it has not improved the situation. I have not seen what the error is on Safari's side. At one point I saw a console error that Safari was unhappy that the ordering of lines for the WebRTC handshake was being altered (because the server parsed and re-serialized the json messages), but fixing that (passing the messages verbatim) did not resolve the issue, and Safari on a macOS Ventura virtual machine logged _no error at all_ in the console for me. ## Linux Webkit Browsers In both GNOME Web (Epiphany) and Midori (Xfce's minimalist browser), opening a user's webcam fails because seemingly these Webkit browsers do not support WebRTC at all? They give a console error: "Variable not found: RTCPeerConnection" ## Things to do * [x] The front-end should more safely check whether RTCPeerConnection is available in the JavaScript API, and give a nice ChatClient error message when it is not (rather than just "not doing anything" when clicking on a webcam) * [x] In case the peer-to-peer WebRTC connection fails, this error should be caught and something sent by ChatClient so the user can get feedback rather than nothing happening. --- ## Additional Context How webcam sharing generally works in BareRTC (see also: [chat server protocol](https://git.kirsle.net/apps/BareRTC/src/branch/master/Protocol.md)): 1. When a user turns on their camera to make it available for sharing * The frontend calls `navigator.mediaDevices.getUserMedia()` to prompt the user to share their webcam and microphone. * On success, their own loopback preview video appears on the page: a `<video>` element directly sourced by their webcam stream, with its audio muted so you don't hear your own microphone echo from it. * The front-end posts a "presence update" message ([sendMe()](https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L371)) to the chat server to say: my video is active, and whether it's marked NSFW. This is what makes the blue button appear to other chatters that they could click on to _try_ and connect and see your video. * Code is around here in startVideo(): https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L989 * This much works on Safari browsers: it isn't doing any WebRTC peer connection stuff yet, just showing your local video feed to your browser. 2. When a viewer clicks on your blue video button to see your camera * The [openVideo()](https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L1057) function * The viewer sends an "open" message to the chat server which the chat server forwards to the broadcaster to let them know that the viewer wants to connect. * When the broadcaster receives it, this is what triggers the "Soandso has opened your camera." message * Note: if you're on Safari you might see this message, but the viewer doesn't actually connect and doesn't go on your Watching list, and vice versa: if a Safari user clicks your camera, you see this notification but they don't actually go on to connect. * The WebRTC function is here: [startWebRTC()](https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L667) - both the offerer (the one wanting to connect) and the answerer (the one sharing video) will go into this function. * At the chat protocol level: the offerer sent an "open" to the server, and the server echoes the "open" back to the offerer here: [onOpen()](https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L470) * To the answerer, the chat server sent a "ring" message in response to that open, handled here: [onRing()](https://git.kirsle.net/apps/BareRTC/src/branch/master/web/static/js/BareRTC.js#L470) 3. The two peers exchanged WebRTC negotiation messages * In here is where the Safari issue occurs. * Using the WebSocket connection with the chat server, the chat server acts as the "signaling server" to basically allow the two peers to exchange messages to each other and negotiate features and how they'll establish the peer connection. * For this stage the chat server just relays the ICE candidate messages back and forth. Usually, the two peers exchange enough info to be able to connect to each other but Safari browsers fail in this step. 4. The two web browsers connect peer-to-peer and add a video stream to that connection, and the broadcaster's video appears on the requestor's screen. At one point in step 3 I saw Safari desktop's error console complain that the ICE messages (json blobs) were rearranged from how it sent them (because the chat server would parse the json message, and re-serialize it when sending it to the peer - the data was the same but keys got reordered, but I updated it so the chat server just forwards the json blob verbatim without messing with it - this resolved that Safari error, but then Safari still doesn't connect and I couldn't get another error message from it yet).

kirsle added the

bug

label 2023-04-10 04:26:40 +00:00

kirsle commented

2023-09-01 01:54:34 +00:00

With PR #36 we got iPads able to connect!

Both parties need to be sharing their videos first
If iPad initiates the call, it works

But if the non-iPad initiates the call, it doesn't go thru and they get this error from ChatClient:

Error sending WebRTC negotiation message (SDP): OperationError: Failed to execute 'setLocalDescription' on 'RTCPeerConnection': Failed to set local offer sdp: The order of m-lines in subsequent offer doesn't match order from previous offer/answer.

This looks similar to an error seen long ago by Safari, and seemed caused by the chat server parsing and re-serializing the SDP and Candidate messages and Safari being unhappy about it. The server switched to just passing the messages along verbatim, and Safari stopped throwing this error.

With PR #36 we got iPads able to connect! * Both parties need to be sharing their videos first * If iPad initiates the call, it works But if the non-iPad initiates the call, it doesn't go thru and they get this error from ChatClient: > Error sending WebRTC negotiation message (SDP): OperationError: Failed to execute 'setLocalDescription' on 'RTCPeerConnection': Failed to set local offer sdp: The order of m-lines in subsequent offer doesn't match order from previous offer/answer. This looks similar to an error seen _long_ ago by Safari, and seemed caused by the chat server parsing and re-serializing the SDP and Candidate messages and Safari being unhappy about it. The server switched to just passing the messages along verbatim, and Safari stopped throwing this error.

kirsle commented

2024-04-17 03:29:55 +00:00

"Apple compatibility mode"

In 8e87c377e8 a breakthrough was found that enabled Webkit browsers to open other peoples' cameras, "most of the time."

The chat added an "Apple compatibility mode" checkbox that enabled this new behavior - on by default if the chat page could detect you were on an iPad or iPhone, and enable-able manually if the auto-detection failed.

When enabled, how it would work is this:

The Webkit browser has to turn its own camera on first, because Apple only supports two-way video calls period.
The Webkit browser clicks to view somebody else's camera.
- And it always attaches the local (Webkit) webcam to that request, to send it to the other party.
- If the other person wanted to "when someone opens my camera, I also open their camera automatically", then this results in the Webkit camera appearing on their screen as expected - two way video call working.
- If the other person did not want to auto-open your video: it received it but it ignores it and doesn't show it on the screen. From Webkit's perspective, it's a two-way video call (it sends video, and gets yours) and they still get to watch you.

The Apple compatibility mode controls whether in step 2 the local video is always sent on the request - it is more efficient not to send your video if the receiver is going to ignore it, and is only a necessary workaround for Webkit viewers.

But: there is still a bug (likely in my own code) where connecting in to view a Webkit browser's camera doesn't work; the Webkit browser has to initiate the call for it to go through. And Webkit-to-Webkit connections probably don't work either.

New workaround ideas for better functionality

Scenario 1

Say you are on a Firefox-on-Windows browser and:

You are on webcam
You don't auto-open your viewer's camera
A Webkit browser connects to see yours (you get their video feed, but ignore it, and the Webkit browser sees your video)
Later, you do want to view the Webkit browser's camera.

Normally, you opening their cam won't work but since your page was technically already receiving their video (but ignoring it), the page could check if a video stream is already coming in and just show it on the page (and it doesn't even need to begin WebRTC negotiations, it can skip straight to showing the video feed it previously was ignoring).

Scenario 2

Say we have two users on camera in the chat:

Alice is on Firefox-on-Windows
Bob is on an iPad browser (Webkit)

And both users are on camera. Alice wants to open Bob's camera, but this won't normally work because Webkit needs to be the one to start the connection. How to work around it?

By sending a message via the chat server, Alice could tell Bob's page to start the connection from his side instead, so that the video connect might actually go through, e.g.:

Alice clicks to open Bob's camera
The chat server (knowing* that Bob has "Apple compatibility mode" enabled), with the "ring" message that gets sent to him to begin WebRTC it includes an indicator saying Bob should turn this around and connect out to Alice instead.
- The chat server would send a similarly custom message back to Alice, so she won't attempt to begin WebRTC negotiation but to wait for Bob to start it. The video icon spinner may still appear on her side while she waits.
Bob receives the custom "ring" message and will attempt to open Alice's camera from scratch as normal, as though he had clicked to open her camera to begin with. He also attaches his video to that request.
- If Bob has the "auto-open my viewer's camera" option enabled, his page will display Alice's video feed. If he did not: it will ignore the video feed from Alice, in a similar way that other users would ignore Webkit videos if they don't want to auto-open.
- Alice's page would remember that she originally wanted this connection to happen, so she would display Bob's video even if her auto-open option wasn't enabled.

This would basically emulate the regular workflow of: Alice needing to enable "auto-open my viewer's cam" and then asking Bob to open her camera. But Alice wouldn't need to auto-open videos, and from her perspective she was able to "just open" Bob's camera like normal as if he wasn't on a Webkit browser at all.

* The chat server would need to know who has Apple compatibility mode enabled, which is currently a front-end-only concern. So when a non-Webkit browser wants to connect in to a Webkit browser, the chat server can send specialized messages to both sides in order to reverse the connection workflow to make it happen.

This would probably still not solve Webkit-to-Webkit connections, but should enable non-Webkit users (who are on camera themselves - this is always a requirement) to open a Webkit user's camera from their side (with possible auto-open on the Webkit end working too).

### "Apple compatibility mode" In 8e87c377e8c28edd6a0a291a3768a78fd5483a59 a breakthrough was found that enabled Webkit browsers to open other peoples' cameras, "most of the time." The chat added an "Apple compatibility mode" checkbox that enabled this new behavior - on by default if the chat page could detect you were on an iPad or iPhone, and enable-able manually if the auto-detection failed. When enabled, how it would work is this: 1. The Webkit browser has to turn its own camera on first, because Apple only supports two-way video calls period. 2. The Webkit browser clicks to view somebody else's camera. * And it _always_ attaches the local (Webkit) webcam to that request, to send it to the other party. * If the other person wanted to "when someone opens my camera, I also open their camera automatically", then this results in the Webkit camera appearing on their screen as expected - two way video call working. * If the other person _did not_ want to auto-open your video: it received it but it ignores it and doesn't show it on the screen. From Webkit's perspective, it's a two-way video call (it sends video, and gets yours) and they still get to watch you. The Apple compatibility mode controls whether in step 2 the local video is _always_ sent on the request - it is more efficient _not_ to send your video if the receiver is going to ignore it, and is only a necessary workaround for Webkit viewers. But: there is still a bug (likely in my own code) where connecting _in_ to view a Webkit browser's camera doesn't work; the Webkit browser has to initiate the call for it to go through. And Webkit-to-Webkit connections _probably_ don't work either. ### New workaround ideas for better functionality #### Scenario 1 Say you are on a Firefox-on-Windows browser and: * You are on webcam * You _don't_ auto-open your viewer's camera * A Webkit browser connects to see yours (you get their video feed, but ignore it, and the Webkit browser sees your video) * Later, you do want to view the Webkit browser's camera. Normally, you opening their cam won't work _but_ since your page was technically already receiving their video (but ignoring it), the page could check if a video stream is already coming in and just _show it_ on the page (and it doesn't even need to begin WebRTC negotiations, it can skip straight to showing the video feed it previously was ignoring). #### Scenario 2 Say we have two users on camera in the chat: * Alice is on Firefox-on-Windows * Bob is on an iPad browser (Webkit) And both users are on camera. Alice wants to open Bob's camera, but this won't normally work because Webkit needs to be the one to start the connection. How to work around it? By sending a message via the chat server, Alice could tell Bob's page to start the connection from his side instead, so that the video connect might actually go through, e.g.: 1. Alice clicks to open Bob's camera 2. The chat server (knowing* that Bob has "Apple compatibility mode" enabled), with the "ring" message that gets sent to him to begin WebRTC it includes an indicator saying Bob should turn this around and connect out to Alice instead. * The chat server would send a similarly custom message back to Alice, so she won't attempt to begin WebRTC negotiation but to wait for Bob to start it. The video icon spinner may still appear on her side while she waits. 3. Bob receives the custom "ring" message and will attempt to open Alice's camera from scratch as normal, as though he had clicked to open her camera to begin with. He also attaches his video to that request. * If Bob has the "auto-open my viewer's camera" option enabled, his page will display Alice's video feed. If he did not: it will ignore the video feed from Alice, in a similar way that other users would ignore Webkit videos if they don't want to auto-open. * Alice's page would remember that she originally wanted this connection to happen, so she would display Bob's video even if _her_ auto-open option wasn't enabled. This would basically emulate the regular workflow of: Alice needing to enable "auto-open my viewer's cam" and then asking Bob to open her camera. But Alice _wouldn't_ need to auto-open videos, and from her perspective she was able to "just open" Bob's camera like normal as if he wasn't on a Webkit browser at all. \* The chat server would need to know who has Apple compatibility mode enabled, which is currently a front-end-only concern. So when a non-Webkit browser wants to connect in to a Webkit browser, the chat server can send specialized messages to both sides in order to reverse the connection workflow to make it happen. This would probably still not solve Webkit-to-Webkit connections, but should enable non-Webkit users (who are on camera themselves - this is always a requirement) to open a Webkit user's camera from their side (with possible auto-open on the Webkit end working too).

Sign in to join this conversation.