In this article, we will create a simple video conferencing application using Java & pure JavaScript (WebRTC).
Video Conferencing Application in this article
- We will create simple 2 person video conferencing application.
- First user will join video conference from a desktop. His own audio/video will start showing up on browser.
- Other participant will also join video conference from a android mobile device.
- Then both participants will be able to see & hear each other via video conference.
- Participants will have option to leave video conference using a provided button.
Technologies
For this application, we will be using below technologies
- WebRTC (Web Real-Time Communication) using HTML5 & Javascript (API specification on Mozilla)
- Websocket using Javascript & Java server for signaling. (Learn Websocket with simple example here)
Here are few ideas/technologies that were discarded in favor of above technologies
- Desktop application client – For this, we need Java based API to connect to web cam & audio etc. There are Java provided API like JMF (Java Media Framework) etc. which have been without update from long time. There are other alternatives (like bytedeco javacv, QuickTime for Java & more) which were not considered/explored as goal was to create basic example without external libraries.
- Web Application with Java server to transmit audio/video between participant browsers – Javascript can do job of capturing webcam/audio & send to server in chunks. We can use web socket server endpoint to exchange audio & video between participants in chunks. But this approach won’t be real time as server will add lag due to latency, processing time etc.
Hence finally WebRTC is chosen for this article since it provides real time Peer-To-Peer streaming between web pages. Basically with this approach, your browser will directly stream audio & video to other participant’s browser without any server in between. Server will only do job of connecting both participant’s browsers to each other.
Design
This video explains the design & approach we will be taking for this example.
Lets Code
Now that you know the design approach, lets get coding. We will create our own signaling server endpoint using Java websocket API. Below is the Java Websocket server endpoint code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
package com.itsallbinary.tutorial.vconf; import java.io.IOException; import java.util.Collections; import java.util.HashSet; import java.util.Set; import javax.websocket.EncodeException; import javax.websocket.OnClose; import javax.websocket.OnMessage; import javax.websocket.OnOpen; import javax.websocket.Session; import javax.websocket.server.ServerEndpoint; /** * Signaling server to WebRTC video conferencing. */ @ServerEndpoint("/signal") public class WebRtcSignalingEndpoint { private static final Set<Session> sessions = Collections.synchronizedSet(new HashSet<Session>()); @OnOpen public void whenOpening(Session session) throws IOException, EncodeException { System.out.println("Open!"); // Add websocket session to a global set to use in OnMessage. sessions.add(session); } @OnMessage public void process(String data, Session session) throws IOException { System.out.println("Got signal - " + data); /* * When signal is received, send it to other participants other than self. In * real world, signal should be sent to only participant's who belong to * particular video conference. */ for (Session sess : sessions) { if (!sess.equals(session)) { sess.getBasicRemote().sendText(data); } } } @OnClose public void whenClosing(Session session) { System.out.println("Close!"); sessions.remove(session); } } |
This is the simple HTML where both participant’s video will be rendered. You can have another separate HTML with “Join Conference” link which can simply redirect to below HTML.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
<!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" /> <title>WebRTC Video Conferencing Application</title> </head> <body> <!-- Title & header of demo aplication. --> <div> <img style="float: left; width: auto; height: 50px" src="https://itsallbinary.com/wp-content/uploads/2017/03/final_itsallbinary.gif" /> <h3 style="position: relative; left: 10px;">WebRTC Video Conferencing <br />Application Demo </h3> </div> <!-- Other person's camera video will show up here --> <div> <h3 style="margin: 5px">Other Person</h3> <video style="width: 50vh; height: 50vh;" id="remoteVideo" poster="https://img.icons8.com/fluent/48/000000/person-male.png" autoplay></video> </div> <!-- Your camera video will show up here. --> <div> <h3 style="margin: 5px">You</h3> <video style="width: auto; height: 20vh;" id="localVideo" poster="https://img.icons8.com/fluent/48/000000/person-male.png" autoplay muted></video> </div> <!-- Button to leave video conference. --> <div class="box"> <button id="leaveButton" style="background-color: #008CBA; color: white; ">Leave Video Conference</button> </div> <script type="text/javascript" src="conf.js?reload=true"></script> </body> </html> |
Now comes the main part i.e. JavaScript. Notice that we are using Google’s free public STUN server for this example. Even though this looks like a big script, have made it pretty simple with code comments explaining all steps. This JavaScript does below steps,
- On load, prepares websocket connection for signaling server endpoint.
- Captures media devices i.e. webcam & microphone using navigator.mediaDevices then renders in UI as local video.
- Performs offer-answer handshake using Java Signaling websocket server & Google’s STUN server so that all participants get connectivity information of each other.
- Setup peer connection with other participant.
- Then adds local video & audio stream as track to peer connection so other participant can hear & see you.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 |
var peerConnection; /* * Setup 'leaveButton' button function. */ const leaveButton = document.getElementById('leaveButton'); leaveButton.addEventListener('click', leave); function leave() { console.log('Ending call'); peerConnection.close(); signalingWebsocket.close(); window.location.href = './index.html'; }; /* * Prepare websocket for signaling server endpoint. */ var signalingWebsocket = new WebSocket("ws://" + window.location.host + "/video-conf-tutorial/signal"); signalingWebsocket.onmessage = function(msg) { console.log("Got message", msg.data); var signal = JSON.parse(msg.data); switch (signal.type) { case "offer": handleOffer(signal); break; case "answer": handleAnswer(signal); break; // In local network, ICE candidates might not be generated. case "candidate": handleCandidate(signal); break; default: break; } }; signalingWebsocket.onopen = init(); function sendSignal(signal) { if (signalingWebsocket.readyState == 1) { signalingWebsocket.send(JSON.stringify(signal)); } }; /* * Initialize */ function init() { console.log("Connected to signaling endpoint. Now initializing."); preparePeerConnection(); displayLocalStreamAndSignal(true); }; /* * Prepare RTCPeerConnection & setup event handlers. */ function preparePeerConnection() { // Using free public google STUN server. const configuration = { iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] }; // Prepare peer connection object peerConnection = new RTCPeerConnection(configuration); peerConnection.onnegotiationneeded = async () => { console.log('onnegotiationneeded'); sendOfferSignal(); }; peerConnection.onicecandidate = function(event) { if (event.candidate) { sendSignal(event); } }; /* * Track other participant's remote stream & display in UI when available. * * This is how other participant's video & audio will start showing up on my * browser as soon as his local stream added to track of peer connection in * his UI. */ peerConnection.addEventListener('track', displayRemoteStream); }; /* * Display my local webcam & audio on UI. */ async function displayLocalStreamAndSignal(firstTime) { console.log('Requesting local stream'); const localVideo = document.getElementById('localVideo'); let localStream; try { // Capture local video & audio stream & set to local <video> DOM // element const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true }); console.log('Received local stream'); localVideo.srcObject = stream; localStream = stream; logVideoAudioTrackInfo(localStream); // For first time, add local stream to peer connection. if (firstTime) { setTimeout( function() { addLocalStreamToPeerConnection(localStream); }, 2000); } // Send offer signal to signaling server endpoint. sendOfferSignal(); } catch (e) { alert(`getUserMedia() error: ${e.name}`); throw e; } console.log('Start complete'); }; /* * Add local webcam & audio stream to peer connection so that other * participant's UI will be notified using 'track' event. * * This is how my video & audio is sent to other participant's UI. */ async function addLocalStreamToPeerConnection(localStream) { console.log('Starting addLocalStreamToPeerConnection'); localStream.getTracks().forEach(track => peerConnection.addTrack(track, localStream)); console.log('localStream tracks added'); }; /* * Display remote webcam & audio in UI. */ function displayRemoteStream(e) { console.log('displayRemoteStream'); const remoteVideo = document.getElementById('remoteVideo'); if (remoteVideo.srcObject !== e.streams[0]) { remoteVideo.srcObject = e.streams[0]; console.log('pc2 received remote stream'); } }; /* * Send offer to signaling server. This is kind of telling server that my webcam & * audio is ready, so notify other participant of my information so that he can * view & hear me using 'track' event. */ function sendOfferSignal() { peerConnection.createOffer(function(offer) { sendSignal(offer); peerConnection.setLocalDescription(offer); }, function(error) { alert("Error creating an offer"); }); }; /* * Handle the offer sent by other participant & send back answer to complete the * handshake. */ function handleOffer(offer) { peerConnection .setRemoteDescription(new RTCSessionDescription(offer)); // create and send an answer to an offer peerConnection.createAnswer(function(answer) { peerConnection.setLocalDescription(answer); sendSignal(answer); }, function(error) { alert("Error creating an answer"); }); }; /* * Finish the handshake by receiving the answer. Now Peer-to-peer connection is * established between my browser & other participant's browser. Since both * participants are tracking each others stream, they both will be able to view & * hear each other. */ function handleAnswer(answer) { peerConnection.setRemoteDescription(new RTCSessionDescription( answer)); console.log("connection established successfully!!"); }; /* * Add received ICE candidate to connection. ICE candidate has information about * how to connect to remote participant's browser. In local LAN connection, ICE * candidate might not be generated. */ function handleCandidate(candidate) { alert("handleCandidate"); peerConnection.addIceCandidate(new RTCIceCandidate(candidate)); }; /* * Logs names of your webcam & microphone to console just for FYI. */ function logVideoAudioTrackInfo(localStream) { const videoTracks = localStream.getVideoTracks(); const audioTracks = localStream.getAudioTracks(); if (videoTracks.length > 0) { console.log(`Using video device: ${videoTracks[0].label}`); } if (audioTracks.length > 0) { console.log(`Using audio device: ${audioTracks[0].label}`); } }; |
Video conference in action
Now that we have code ready, we can deploy above application as a war file to any server like Tomcat. Here are few things to note before you test the application.
- Please check browser compatibility of navigator.mediaDevices.getUserMedia() & choose appropriate browser to test. Below demo is using chrome browser.
- For the first time when you run this application, browser will prompt you to give permission to use webcam & microphone. Provide permissions to test further.
- Since we are testing this using “http://” & not “https://, android chrome browser won’t let webcam & microphone to be accessed. It won’t even prompt for permissions. So you will have to whiltelist local URL in settings of
unsafely-treat-insecure-origin-as-secure
by openingchrome://flags
. More explanation in this stackoverflow discussion.
Here is the video which shows our video conferencing application in action.
Below is the screenshot of the setting required in android to run this application in local.
GitHub Code
All above code can be found in GitHub repository of itsallbinary.
Further Reading
Checkout below articles which might be of interest to you.
Create your own screen sharing web application using Java and JavaScript (WebRTC)
Things to try further
You can take this as base & try to play around by adding some features on top of this. Below are few ideas.
- Make application multi participant video conference.
- Currently it sends signal to all participants. Change it so that user can pick who to add to video conference. Signal to only those participants.
- Experiment with navigator.mediaDevices & see what all you can do with it like choose specific camera on mobile etc., try mute/unmute on audio device etc.
- Try running outside local network & see if you can get it to work. You might need TURN server which is not explored in this article.
References
- Thanks to icons8 for free icon. https://icons8.com/icons/set/person
- Thanks to https://gist.github.com/zziuni/3741933 for list of free STUN servers.
- https://stackoverflow.com/questions/34197653/getusermedia-in-chrome-47-without-using-https
- https://webrtc.org/
How did you run it, am not able to start the project
Hi, I am able to run it on local machine and mobile phone as well. The only problem am facing is its not showing both participant only one participant video is visible. Any pointes for that?
Thanks
If everything went well then it should show both user’s video. You can first try testing you application in same computer with 2 different browsers windows. Both browser windows should show your video from same webcam.
If abvoe itself is not working, something might be wrong. You can use ‘Developer Tool’ in your browser & see if websocket communication is happening correctly. Also verify that there is no error in javascript. If you see any error, then will need more info on error to analyze further. Hope this helps.
conf.js?reload=true:19 WebSocket connection to ‘ws://127.0.0.1:8081/video-conf-tutorial/signal’ failed: Error during WebSocket handshake: Unexpected response code: 404
getting this error. remote video not shown
404 means websocket endpoint itself is not reached. But since you are seeing this error, I assume that you did see HTML successfully which means war is deployed correctly. So I think somehow websocket endpoint is not being detected by your server.
Not sure which server you are using. Please check if your server is latest & updated. Also verify server supports websocket endpoints. Also verify that your server supports “annotation” based websocket endpoints.
We tested this in Apache Tomcat version 9.0 & it worked perfectly. Please try this.
Hope this helps.