Teaching a computer how to lip read isn’t science fiction, it’s a reality and it’s happening right now in Malaysia. Here, researchers at the International University in Selangor are teaching a computer to interpret human emotions based on lip patterns in order to improve the way people interact with computer.

The scientists developed their system using a genetic algorithm that improves with each iteration to match irregular ellipse fitting equations to the shape of a human mouth displaying different emotions. The team used photos of individuals from South-East Asia and Japan to train the computer to recognize the six commonly accepted human emotions — happiness, sadness, fear, anger, disgust, surprise — and neutrality. The algorithm then analyzed the upper and lower lip as two separate ellipses.

 In the current study, the researchers’ algorithm successfully classified all seven emotions, along with a neutral expression. The researchers suggested that initial applications of such an emotion detector could involve, for instance, helping disabled patients lacking speech to interact more effectively with computer-based communication devices.

Image Credit: Dmitriy Kiryushchenkov / Shutterstock