Gameface Project: Google makes mouse cursors controllable with facial gestures

Along with the mobility-impaired game streamer, Google has developed open source software called Project Gameface that allows users to control their mouse cursor with a simple webcam and occasionally play video games on it.

Google replaces the gaming mouse with a webcam and software

As part of its I/O 2023 internal developer conference, Google introduced a new artificial intelligence project called Project Gameface, which enables players to control video games through head movements and facial gestures. In this context, the group also talks about “An AI-controlled gaming mouseWhich gives users full control over their computer mouse cursor. Clicks and drag-and-drops can also be implemented, for example by the user raising their eyebrows or opening or contorting their mouth.

To do this, the webcam first records the user’s head movements and facial gestures. The recorded footage is then processed via the open source machine learning platform Google media tube processed. The program displays a grid of a total of 468 points on the user’s face and finally translates their movements into mouse cursor actions. So no special hardware is required. It is a pure software solution that works with a standard webcam.

An open source solution for people with disabilities

Google cites gamer Lance Carr, who suffers from muscular dystrophy, as the inspiration for Project Gameface. It is a genetic disease that leads to the progressive breakdown of muscle tissue. Over time, this is associated with significant motor impairments for those affected, which also means that in the case of Lance Carr, he cannot use any commercially available input devices to control his computer.

However, the operator did not let this stop him from playing PC games directly on his phone YouTube channel Transfer to. In order to control his mouse pointer, he sometimes uses a special head-tracking mouse. However, in November 2021 it came during the live broadcast His house caught fire, after which he lost all of his gaming equipment.

For Google, this story was reason enough to create Project Gameface and work with Lance Carr to develop a hands-free gaming mouse. Since users with physical limitations often have very individual requirements, the project provides the possibility to define different sizes of gestures. For example, a click can be triggered by a very slight movement of the eyebrow.

As Google points out in the announcement, Project Gameface is currently still in development. However, a first preview of the open source solution is now available Ready on github.