Building Accessible Apps with Eye Tracking

The Challenge of Accessibility

Millions of people around the world live with motor disabilities that make it difficult or impossible to use traditional computer interfaces like keyboards and mice. For these individuals, simple tasks that most of us take for granted—browsing the web, writing emails, or even just clicking a button—can be incredibly challenging.

Eye tracking and head movement detection technologies offer a revolutionary alternative, enabling hands-free computer control that can dramatically improve quality of life and independence.

Technology Stack

For this project, I chose a combination of powerful, open-source libraries that work together seamlessly:

How It Works

1. Face Detection with MediaPipe

MediaPipe's Face Mesh solution detects 468 facial landmarks in real-time. We specifically use the eye landmarks to calculate the Eye Aspect Ratio (EAR), which helps us detect blinks.

2. Eye Aspect Ratio (EAR)

The EAR is calculated using the distances between specific eye landmarks. When the eye is open, the EAR is relatively constant. When the eye closes (blink), the EAR drops significantly.

Core Implementation

import cv2
import mediapipe as mp
import pyautogui
import numpy as np

# Initialize MediaPipe Face Mesh
mp_face_mesh = mp.solutions.face_mesh
face_mesh = mp_face_mesh.FaceMesh(
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5
)

# Eye landmark indices
LEFT_EYE = [362, 385, 387, 263, 373, 380]
RIGHT_EYE = [33, 160, 158, 133, 153, 144]

# EAR threshold for blink detection
EAR_THRESHOLD = 0.2
CONSECUTIVE_FRAMES = 3

def calculate_ear(eye_landmarks, landmarks):
    """Calculate Eye Aspect Ratio"""
    # Get coordinates
    coords = np.array([(landmarks[i].x, landmarks[i].y) for i in eye_landmarks])
    
    # Calculate vertical distances
    v1 = np.linalg.norm(coords[1] - coords[5])
    v2 = np.linalg.norm(coords[2] - coords[4])
    
    # Calculate horizontal distance
    h = np.linalg.norm(coords[0] - coords[3])
    
    # EAR formula
    ear = (v1 + v2) / (2.0 * h)
    return ear

# Main loop
cap = cv2.VideoCapture(0)
blink_counter = 0

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    
    # Convert to RGB
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = face_mesh.process(rgb_frame)
    
    if results.multi_face_landmarks:
        landmarks = results.multi_face_landmarks[0].landmark
        
        # Calculate EAR for both eyes
        left_ear = calculate_ear(LEFT_EYE, landmarks)
        right_ear = calculate_ear(RIGHT_EYE, landmarks)
        avg_ear = (left_ear + right_ear) / 2
        
        # Detect blink
        if avg_ear < EAR_THRESHOLD:
            blink_counter += 1
        else:
            if blink_counter >= CONSECUTIVE_FRAMES:
                # Trigger mouse click
                pyautogui.click()
                print("Click!")
            blink_counter = 0
    
    cv2.imshow('Eye Tracking', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Key Features

Real-World Impact

This project represents more than just a technical achievement—it's a tool that can genuinely change lives. For individuals with conditions like ALS, spinal cord injuries, or severe arthritis, this system provides:

• Independence: The ability to use a computer without assistance
• Communication: Access to email, messaging, and social media
• Employment: Opportunities for remote work and productivity
• Entertainment: Browsing, gaming, and media consumption