Skip to main content

A Modest Proposal: Virtual Keyboards via Kinect Eyeglasses

This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American


In the series "A Modest Proposal," my colleagues and I will propose inventions and projects that I think are eminently doable and would love made real.

I love my iPad. I scan vast amounts of information daily for work and play, and tablet computers are ideal for displaying it. But tablets and mobile devices in general have a key weakness — as much as they help people consume information, they are not very good at helping us produce it. This is why I am proposing virtual floating keyboards via Kinect-loaded eyeglasses.

A virtue and a vice of iPads is that they use the same surface for inputting and outputting data. Since the display is the interface, all you need are your fingers to manipulate information, which means figuring out how to use them comes naturally to infants, senior citizens and even cats. The problem is that when it comes to writing, the keyboard that pops up to input data competes with the visual real estate needed for the iPad to output data.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


The way that conventional computers solve this problem is through peripherals such as keyboards and mice. Tablets can use wireless versions of those too, but those detract from the streamlined experience they offer. The new iOS 5 that Apple has for its latest generation of mobile devices can split the iPad's keyboard up into two smaller halves so it takes up less visual real estate, but the problem there is that you can only use your two thumbs to input data, as opposed to all 10 fingers, limiting your input bandwidth.

The iPhone 4S has Siri, which allows users to interact with applications by voice, which to me shows that Apple is trying to come up with new interfaces to overcome the size limits that mobile devices have when it comes to keyboards. This is very promising, but speech recognition still has a ways to go before it can always accurately transcribe what you are saying, and even if it was perfect, keyboards and mice are still better at helping people cut, paste, copy and otherwise edit data — 10 fingers are better than one voice in that regard.

So what to do? Ideally you would have a virtual keyboard you could use without actually having a physical keyboard to lug around. Inventors have developed many techniques over the years that allow any surface to be turned into keyboards — a recent version can be seen here — but to me it seems a bit laborious to have to try and find a surface to project a keyboard onto. Virtual reality gloves also seem cumbersome, and are not a one-size-fits-all peripheral.

https://www.youtube.com/watch?v=Pz17lbjOFn8

I suggest eyeglasses that display virtual keyboards. This would essentially be an augmented reality system that overlays a floating keyboard onto your field of vision. I also suggest a Kinect sensor mounted on top of these glasses.

After the sensor recognize your 10 fingertips, you can then type in the air onto the virtual keyboard to input data into your mobile device. I suggest these glasses only be used to overlay a keyboard instead of a full screen — a virtue of tablets is that they only take up as much of your field of view as you want, and you can easily put them down if you want.

https://www.youtube.com/watch?v=f4xeB_duBoY#!

Kinect sensors are currently rather clunky to fit onto eyeglasses, but I wager optimized sensors could be much smaller and integrated into the glasses. I imagine the system would also require a rather healthy amount of computing power, but perhaps that could be exported onto the tablet in question, with data transmitted wirelessly to and from the glasses.

I suspect the main bottleneck of such augmented reality glasses would be power. There's not a lot of space in glasses to support batteries for what must be a rather power-hungry application.

You can email me regarding A Modest Proposal attoohardforscience@gmail.comand follow the series on Twitter at #modestproposal.

Charles Q. Choi is a frequent contributor to Scientific American. His work has also appeared in The New York Times, Science, Nature, Wired, and LiveScience, among others. In his spare time, he has traveled to all seven continents.

More by Charles Q. Choi