The first of the main SID keynotes was given by Steven Bathiche of Microsoft who said that things had changed a lot since he attended his first SID event in 2001. He keeps coming back because SID is the place to see what the next computer will look like. He is ‘effectively’ CTO of Microsoft’s hardware devices.
New categories of products have new displays and interaction technologies – almost every new category has followed this kind of change in display and interaction. There has been increasing diversification in the kinds of technology delivered. Each new computer class meets different needs in entertainment or at work. New devices don’t drive old technologies away – Batiche said that every device is ‘best for something’ and ‘worst for something’.
For Microsoft, the new category hardware development started with the SUR40 which it co-developed with Samsung in 2005 – it was a multiple user table. The system could create mixed reality experiences by detecting real objects that were placed on the display surface. The display was truly bi-directional – with image recognition in each pixel.
Each device from Microsof is intended to be unique. The Surface Book looks like a notebook, but it can undock and has two GPUs – one in each half. All of the newer Microsoft displays in the category are 3:2 aspect ratio – which is more like paper, Bathiche said. Books and content are not often 16:9 – vertical height is important. Pen use and support is also a big focus for the firm. You can’t type much of the content that you create. Keyboards are left brain – but creative content comes from the right. Keyboarding is less effective than typing when note taking.
How do you make a really good pen experience, he asked? A simple pen is easy – but a good pen is very difficult. For example, Microsoft’s first decision when it looked at the display was to bond the panel to the cover to avoid parallax – and Bathiche said that the Surface book is the notebook with the thinnest parallax on the market,
Microsoft has worked hard to reduce parallax for better pen input
Paper has zero latency – and response time down to 1ms is noticeable and difficult. His group has tested a 1ms pen system which is difficult to make but feels a lot more natural. Paper also doesn’t get confused by palms on the surface – the system must recognise and differentiate between a finger, or pen or palm if it is to be intuitive. Creating the sensing hardware without trading off the display performance is difficult.
The Surface Book has the thinnest large tablet module for LCD on the market, but in terms of resolution, Microsoft believes that 300 ppi is enough.
Once you can recognise pen and finger you can get rid of menus and other instructions, you effectively use gestures entirely. Microsoft has developed software for ‘ink beautification’. Bathiche showed a demonstration of the software taking a user’s handwriting and just ‘tidying it up’ to look much better. (I found a video showing this work from 2013 at http://tinyurl.com/jgopgla)
He also showed work from a user study that helped users create drawings, both by recognising geometric objects and by using ‘shadows’ to help guide the artist. For example, he started to draw a bike and the system showed broad shadowed lines roughly where a drawn line should be. The technology could be used to perform searches. He gave an example of the use of drawing to search in Vizio, which uses a lot of templates. A simple drawing starts to show the objects with the same kind of shape.
Surface Hub is a New Device Class
The next class of new device is the Surface Hub – and in this case, the key innovation was to make a controller for high speed, low latency touch and stylus. Then you have to build the systems – which was not trivial as the cover glass is optically bonded and Microsoft had to design its own production equipment to do this.
Microsoft had to develop its own equipment for Surface Hub Assembly
With the hub, there are novel interactions which are both multi-device and multi-user. It’s important to understand which other PCs are in the room when you are running a meeting room display. For several years, Microsoft has a ‘Magic Window’ concept, where users collaborate on either side of a transparent display, but this is still under development. (I found a short video from 2011 at http://tinyurl.com/zb44e97)
Bathiche talked about Continuum, which exploits the power that Moore’s law is delivering to mobile devices – there’s a lot of power in your hand, these days. Windows 10 Continuum adds the display to match the power by allowing Windows phones to use a full sized monitor.
Another class is the Microsoft Band – a new category of computing device for the wrist that uses a curved plastic OLED and bonded touch – it includes 11 sensors to track a lot of information.
The Hololens A/R system has all new silicon and includes 3D camera technology which can look down at the hands and also out into the world – the system reconstructs in real time the world around you. It also watches your hands to know what you are doing. Latency has to be very low ‘and world locked; this was part of the original specification. There are two projectors using diffraction gratings which inject the images into the device and it has a ‘floating’ display using a diffraction grating in front of the screen.
Bathiche also talked about “The Wedge” display that Microsoft has been working on for a long time (we first reported on it from the EID show in the UK in 2000). An image is injected from the edge and uses TIR to be distributed over the screen. However, it can also be used to capture light so could act as a way of making conferencing more intuitive by enabling eye contact. The development is working on ‘touch and hover’ and a camera sees through the screen. The system helps to understand gesture and can distinguish between hover and touch. Check out the paper “Pre-touch sensing – for mobile interaction” by Ken Hinckley et al in Microsoft Research. http://tinyurl.com/gtv5ykt
Microsoft is also developing its own eye tracker technology – what could you do if you can see where the user is looking, Bathiche asked?
One of the key concepts for Microsoft is that ‘every surface becomes a computer’ (hence the product names) and this is particularly true once you have a Hololens-type headset with real world reconstruction.
Finally, Bathiche suggested that the SID should be renamed the “Society for Interactive Display”. Display is only half of the solution – there should be a relationship between output and input pixels. (BR)
