In display related applications, a useful definition of latency is “the delay between what a finger does and how fast coordinated graphics are drawn.” It is important that latency be minimized because excessive delay can “degrade precision in tasks such as drawing, and, in general, break the realism and responsiveness of direct manipulation interfaces.” A recent effort to reduce latency was undertaken by a team of researchers led by Craig Shultz of Carnegie Mellon University (Pittsburgh, PA).
The team addressed the latency issue by the use of an approach that has been given the name TriboTouch. A recent article on this topic by the team is entitled “TriboTouch: Micro-Patterned Surfaces for Low Latency Touchscreens.” The article will be published in the Proceedings of the 40th Annual SIGCHI Conference on Human Factors in Computing Systems (April 30 – May 6, 2022) CHI ‘22. ACM, New York, NY. This Display Daily article is paraphrased from the team’s article.
Here are a few words of background information. The effort to reduce latency has included some approaches based on hardware and others based on software. Up to now, success has been modest. There are two reasons why this is the case. First, in software-based approaches, “extrapolation is inherently limited by old and ambiguous positional data, and closing the gap to zero perceptible latency with acceptable jitter may be impossible.” Second, in hardware based approaches, “it has been found difficult to simply increase system scan rate due to factors such as sensor noise, timing constraints, data overhead/computation and power consumption.”
In their article, the team explain that the TriboTouch approach combines innovations both in hardware and software. A top level description of the approach is that conventional touch data is combined with acoustic data and that the combined data is fed into a machine learning model to produce predictions of future touch location.
More specifically, TriboTouch is based on a principle of texture induced vibrations from frictional interaction. When two surfaces rub together, vibrations are created which are a function of each surface’s spatial roughness distribution and the velocity at which they are sliding. This principle is implemented in the TriboTouch approach by a film that is embossed with a 2D pattern of small bumps with a pitch of 5 microns. In the display application, the TriboTouch film is positioned on top of the 2D touch sensor and the upper surface of the display. “When a finger or stylus is drawn across the patterned surface, interaction with the micro-features induces an acoustic vibration with a fundamental frequency that directly encodes x and y velocities. This vibration is low amplitude and, at many velocities, ultrasonic, meaning it is imperceptible to the user.”
The signal is sampled using a conventional audio pipeline at 192 kHz with a mean latency of 28 ms. This is less than half the time of most touch pipelines. The high bandwidth, low latency, 1D vibroacoustic signal is fused with the low framerate but high spatial accuracy 2D touch data. The touch data derives from a conventional projected capacitance touchscreen which has a latency of 69 ms. All the data is fed into a machine learning model.
A block diagram of the TriboTouch system is presented in the figure below.
Block diagram of the TriboTouch system. The TriboTouch layer sits on top of a 2D touch sensor and upper surface of the display.
The article reports the results of experimental evaluations of a TriboTouch enabled device. It was found that latency could be reduced from 96 ms to 16 ms with a mean distance error of 5.13 mm. Stating the results more generally, the team claims having demonstrated that, by incorporating TriboTouch data into the touch prediction pipeline, their combined model substantially improves upon touch only models.
In the technical article, the researchers go on to describe how to build a TriboTouch system with inexpensive, readily available materials and methods. The team concludes their article with the assessment that a TriboTouch approach could also work on resistive, self-capacitive, optical, acoustic and other touchscreen technologies. -Arthur Berman
Carnegie Mellon University, Craig Shultz, [email protected]
