DDD, a company that has been involved with stereoscopic 3D for over 20 years, has released a new product called TriDef SmartCam. It is essentially a virtual green screen application that allows you to remove the background and insert any background you want (like with a real green screen application). However, the SmartCam does this all with an ordinary web camera and software.
How is this useful? According to DDD CEO Chris Yewdall, the initial applications are focused on video conferencing and gamecasting. For video conferencing for example, “We took a few lessons learned in the 3D market and made it a ‘middleware’ app so that it can be used with a host of popular apps like Skype, Google Hangouts, Facebook chat etc. without the need for the software developer to add any TriDef SmartCam specific code to their products”, commented Yewdall.
I tried it out by downloading the free trial app at: www.tridef.com/smartcam. Then, I went into the Skype setting where SmartCam was already loaded and ready to go. You initially see yourself with the web cam with a new background image inserted. I easily uploaded a new image and placed myself in front of an old building I photographed in Ieper, Belgium earlier in the year. (The town was completely rebuilt in the medieval style stone by stone after being completely demolished by the Germans in WW1.)
SmartCam did a great job of removing the background, but as you can see, there are still some issues with removing the edges around my head and neck. Yewdall says good lighting is important, but this is one area they are working on and will improve in later editions. “This is especially hard to do in real time, but for non-real time applications, we can do some additional processing to get good clean edges”, he said.
I also noticed that if I stuck only my hand in front of the web cam, it didn’t do anything. “This is the ‘just me’ mode”, explained Yewdall. “We spent a lot of time learning how to use the algorithms to focus on face and torso detection in the near field. This is important if you are using the SmartCam in an environment where there are other people moving in the background, so that the people are properly removed from the frame”.
However, the ‘normal’ mode allows for the tracking of hand and arm movements. In this mode, the system works well up to 6-8 feet away from the webcam which means you can stand and give a lecture or presentation and insert your image over the powerpoint slides, for example. “This was one of the many suggestions we had, in this case, from a Professor in Italy”, said Yewdall. “We have had downloads from over 45 countries already!”.
Yewdall also noted it can be used to improve picture quality over lower bandwidth connections since the video encoder can be steered to apply the detail to the area of the image represented by the end user, thereby reducing the amount of video data that is used by areas which are not of interest to the other participants. This is particularly useful for video conferences conducted over mobile networks from mobile devices.
The other major initial area of adoption is in gamecasting. This is where gamers record video of themselves playing a video game and then post this on sites like Twitch for others to enjoy. With the SmartCam, they can remove the background and even superimpose their head over the game playing. Twitch has 100M monthly viewers so is a huge potential market opportunity. SplitMediaLabs, whose XSplit products are widely used in the webcasting and gamecasting communities, is a new SmartCam affiliate, so look for this to roll out shortly. You can see a demo in this video: https://vimeo.com/132993580
A third major area is what Yewdall calls “social photography.” Here, the idea is to use the SmartCam app with your mobile device to make a “smart selfie.” To do this, you use the rear camera to snap a photo of a landmark, for example, and the rear-facing camera to take a picture of your face. The App can remove the background your face and place it anywhere in the landmark photo you want. You then share this with friends, who can place themselves in front of the landmark as well. Cool. Yewdall says this capability should be available in late Q3 or early Q4 this year. You can see more in this video: https://vimeo.com/121422478
The SmartCam can even work with a RealSense 3D sensor suite from Intel using the extra sensor information for better decision making and a more refined background removal. In fact, for many applications Yewdall sees its SmartCam and similar software products, like the hand-gesture tracking software from PointGrab, as a viable alternative to an Intel RealSense solution. “RealSense is wonderful, but it requires hardware integration and consumes power and dissipates heat. SmartCam uses the web cam already built into mobile devices and software, so it is an easier sell for the applications we are targeting”, concluded Yewdall. “And down the road, we see intelligent video being possible where you will able to insert yourself into live TV shows and share with your friends”.
The Windows version of the SmartCam software can be downloaded with a free 7 day trial from https://www.tridef.com/products/smartcam and DDD was kind enough to offer Display Daily readers a 50% discount on the SmartCam purchase price before the end of August using the code MEKO50 during checkout.
