A team of researchers headed Robert Xiao within the Human-Computer Interaction Institute at Carnegie Mellon University (Pittsburgh, PA) is developing a smartwatch with an integrated pico-projector. The system uses the skin of the wearer’s arm as the surface on to which the image is projected.
The self-contained prototype device, named LumiWatch, also has the capability to detect the point in the projected image touched by the wearer’s finger when interacting with the content in the image. The researchers believe that this approach has the potential to “expand the display area of smartwatches and provide a more intuitive form of touch interactivity.”
A recent article on LumiWatch technology was presented by the team this past April at the 2018 Conference on Human Factors in Computing Systems (CHI 2018) held in Montréal, Canada. The article was entitled “LumiWatch: On-Arm Projected Graphics and Touch Input.” A copy of the article is available on-line and can be found here.
The LumiWatch projection capability is contained in a module with a physical size of 25.8 x 16.6 x 5.2 mm. The projector outputs 15 lumens and operates in a raster-scan mode using red, green and blue lasers. Two MEMS mirrors are used to steer the laser beams. The image is projected at 60 Hz and has a pixel resolution of 1024 x 600 and a 39º horizontal x 22.5º vertical field of view.
The LumiWatch touch detection capability uses a 7 x 38 x 3 mm module containing ten STMicro VL6180X sensor elements. The sensors act as a one dimensional, time of flight, depth sensing array. The module is used to determine the wearer’s 2D finger position on or near the surface of the skin when the wearer is interacting with the projected image. The result is an interactive area is about 40 cm² – more than five times the area found in a typical smartwatch display. The LumiWatch interactive area supports common touchscreen operations such as tapping and swiping. These capabilities enable the LumiWatch to offer touch interactions similar to that of a conventional single touch smartphone. The touch tracking frame rate of is 27.5 Hz.
The figure below and to the left shows the prototype LumiWatch. In the figure below and to the right is an image of the wearer using their finger to interact with a projected image.
Right: close up of the LumiWatch module. Left: the wearer interacting with a projected image.
A video at the end of this article further illustrates and discusses the LumiWatch.
Other features and specifications of the prototype LumiWatch include the following:
- Bluetooth 4.0 and WiFi.
- Qualcomm APQ8026 system-on-chip.
- 1.2 GHz quad-core CPU.
- 450 MHz GPU.
- Overall physical size of the prototype LumiWatch is 50 x 41 x 17 mm.
- Runs on Android 5.1.
- A 740 mAh, 3.8 V lithium-ion battery is incorporated in the casing.
- Draws 2.7 W at maximum brightness.
- One hour continuous operation in the projection mode.
- 768 MB of RAM.
- 4 GB of flash memory.
- Contains an inertial measurement unit.
- Contains an ambient light sensor.
Owing to the shallow angle of projection and the complex shape of the forearm surface, the researchers report that it was necessary to develop a graphics pipeline to “rectify” or correctly shape the projected image. In addition, calibration was required to enable continuous tracking of the 2D position of the wearer’s finger touch on the skin. It was also necessary to perform luminance correction since pixels near to the projector are brighter than those that are further away.
Part of the means developed to accomplish the rectification process used a swipe unlock gesture.which actually served two functions. First, to verify the user’s intent to interact with the system (mitigating false inputs). Second, to allow the angle of the wearer’s arm to be seamlessly and intuitively calibrated without the need for a separate and dedicated calibration routine. Beyond the initial calibration, the researchers found that it was necessary to dynamically align the projector and touch sensor systems.
In their paper, the researchers report on two studies that were conducted to evaluate the performance of the LumiWatch system. Details of the tests and the test results are fully presented in their article. The first study evaluated input touch performance. The results were, in general, quite good. The second study measured qualities of the projected image. Interestingly but, perhaps not surprisingly, both ambient lighting and skin color were found to significantly affect the visibility of the projected image.
In closing, it can be noted that Meko subscription newsletters have previously reported on projection watch technology. The reader is referred to an article entitled “ASU Brings Big Backers, but Fails with Us” that reports on the ASU CastOne device. The article can be found here. (Haier is also reported to have shown a similar device at the MWC show in February – and we found a recent review here Man. Ed.) -Arthur Berman
Carnegie Mellon University, Robert Xiao, [email protected]
