Everything runs client-side from a webcam or uploaded clip. One MediaPipe Face Landmarker pass supplies geometry for three independent estimators, then a fusion step produces the experimental integrity reading shown beside markets and the Integrity Room.

1. 01
   Landmarks

   478-point face mesh + ARKit-style blendshapes every video frame.
2. 02
   Physiology

   Forehead / cheek RGB → POS rPPG → BPM + RMSSD; EAR → blinks + PERCLOS fatigue.
3. 03
   Expression

   Blendshape activations mapped to seven expression probabilities (heuristic, not a trained FER net).
4. 04
   Fusion

   Stress from HR/HRV + expression heuristics → suspicion meter, gated by signal quality.

Implementation lives under lib/emote/ (face.ts, rppg.js, fatigue.js, stress.js, emotion.js, integrity.ts). Live demo: /analyze.

Emote AI uses Google's MediaPipe Face Landmarker (Tasks Vision). One model drives every downstream signal: forehead / cheek ROIs for rPPG, eyelid landmarks for Eye Aspect Ratio, and 52 blendshapes for expression.

• Forehead and cheek index sets sample skin for mean RGB over time ([10], [11]).
• Left / right eye horizontal and vertical pairs compute EAR each frame ([1]).
• Processing stays on-device; video is never uploaded.

Fatigue is the most directly research-grounded panel in the analyzer. We do not invent a novel drowsiness classifier; we implement published eyelid metrics.

Eye Aspect Ratio (EAR)

Soukupová & Čech showed that facial landmarks are precise enough to estimate eye openness from a single scalar: the ratio of vertical eyelid distances to eye width. EAR stays roughly constant when the eye is open and collapses toward zero during a blink ([1], [2]).

Blink counting

A blink is detected on the closed → open transition. Typical blinks last roughly 100–400 ms ([2]). Consecutive closed frames beyond a drowsiness window are flagged as a micro-sleep candidate rather than a normal blink.

PERCLOS

PERCLOS (PERcentage of eyelid CLOSure) is the fraction of recent time the eyes are closed. Dinges & Grace validated it against psychomotor vigilance as a psychophysiological alertness measure for transportation safety ([3]; earlier vehicle monitoring work in [4]). Emote AI keeps a rolling window of closed/open frames and blends PERCLOS with a micro-sleep term into a 0–100 fatigue score:

Why this matters for Integrity Markets: fatigue is a real, published biometric - useful as match context (e.g. late-game officiating load) without claiming intent or guilt.

Blood volume pulses change how skin reflects light. A webcam pointed at a stable skin ROI can recover heart rate without contact. Verkruysse et al. demonstrated remote PPG from ambient light, with the green channel carrying the strongest pulse ([7]).

Emote AI implements POS (Plane-Orthogonal-to-Skin) from Wang et al.: temporally normalize RGB, project onto a plane orthogonal to skin tone, overlap-add short windows, then band-pass (0.7–4 Hz ≈ 42–240 BPM) and FFT for BPM plus peak-interval analysis for HRV ([5], [6]). This is the method named in lib/emote/rppg.js - not a mock waveform.

• Sliding ~12 s analysis window; first estimate after ~6 s of buffer.
• Signal quality gates downstream stress / integrity so bad lighting does not invent drama.
• Accuracy still depends on stillness, illumination, and visible skin - the quality meter exists for that reason.

Sympathetic arousal tends to raise heart rate and suppress short-term heart-rate variability. The ESC/NASPE Task Force standardized HRV metrics including RMSSD for short-term parasympathetic activity ([8]). Meta-analytic work links reduced HRV to stress-related states ([9]).

Emote AI calibrates a personal resting baseline over ~30 s of good-quality samples, then scores deviations:

• HR component: how far BPM sits above the baseline (soft-clamped).
• HRV component: how far RMSSD sits below the baseline.
• Blend ≈ 60% HR + 40% HRV → 0–100 arousal index (Calm / Elevated / High).

This is a coarse arousal proxy - deliberately not labelled as guilt, lying, or match-fixing.

MediaPipe exposes ARKit-style blendshape activations. Emote AI maps those to seven expression buckets (neutral, happy, sad, angry, surprised, fearful, disgusted) with hand-tuned rules and a softmax. That is an interpretable expression readout - not a trained facial-emotion-recognition network, and not a measure of felt emotion.

The Action Unit tradition behind modern face coding is FACS ([12]). We cite it as conceptual background; our mapping is a lightweight heuristic on top of MediaPipe outputs ([11]).

Track 1 Integrity Markets (see the pitch: trade the result and whether officiating reads clean) needs a single chip beside each fixture. The composite is honest about what it is: a transparent entertainment heuristic, shown as context, never as the settlement source.

Markets settle from the TxODDS result oracle and a recomputable sha256 evidence hash (/verify). The integrity chip is context - "built to be audited, not to accuse."

Facial and physiological deception detection is not scientifically reliable as evidence of wrongdoing. Large reviews find human deception judgments barely above chance ([13]), and the National Academies concluded that polygraph-style physiological inference is not a dependable test of lying ([14]).

• Detection of HR, HRV, EAR, and expression geometry is real engineering.
• The integrity / suspicion score is an experimental heuristic for entertainment.
• It is not a lie detector, not forensic evidence, and not an accusation of match-fixing.
• It never settles a market; settlement is the verifiable result hash.

Primary papers and standards behind the measures above. Prefer DOIs / official PDFs when citing Emote AI methods externally.

1. [1]Soukupová, T. & Čech, J.
   Real-Time Eye Blink Detection using Facial Landmarks ↗
   CVWW 2016

   Defines the Eye Aspect Ratio (EAR) used for blink / closure detection.

2. [2]Soukupová, T. & Čech, J.
   Real-Time Eye Blink Detection using Facial Landmarks (technical report) ↗
   CTU CMP TR-2016-05

   Extended treatment of EAR thresholds, temporal windows, and blink timing (~100–400 ms).

3. [3]Dinges, D. F. & Grace, R.
   PERCLOS: A Valid Psychophysiological Measure of Alertness As Assessed by Psychomotor Vigilance ↗
   FHWA-MCRT-98-006

   PERCLOS - proportion of time eyes are closed - as a validated drowsiness metric.

4. [4]Wierwille, W. W. et al.
   Research on vehicle-based driver status/performance monitoring ↗
   NHTSA DOT HS 808 247 (1994)

   Foundational driver-drowsiness work linking eyelid closure metrics to alertness.

5. [5]Wang, W., den Brinker, A. C., Stuijk, S. & de Haan, G.
   Algorithmic Principles of Remote-PPG ↗
   IEEE Trans. Biomed. Eng. 64(7), 2017

   Introduces Plane-Orthogonal-to-Skin (POS) - the pulse extraction method used in Emote AI.

6. [6]Wang, W. et al.
   Algorithmic Principles of Remote-PPG (open preprint) ↗
   Eindhoven University of Technology

   Open PDF of the POS / rPPG principles paper.

7. [7]Verkruysse, W., Svaasand, L. O. & Nelson, J. S.
   Remote plethysmographic imaging using ambient light ↗
   Opt. Express 16(26), 2008

   Shows that ambient-light camera video carries a usable pulse signal, strongest in the green channel.

8. [8]Task Force of the ESC and the NASPE
   Heart rate variability: standards of measurement, physiological interpretation and clinical use ↗
   Circulation 93(5), 1996

   Canonical HRV standards; RMSSD as a short-term parasympathetic marker.

9. [9]Thayer, J. F., Åhs, F., Fredrikson, M., Sollers, J. J. & Wager, T. D.
   A meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health ↗
   Neurosci. Biobehav. Rev. 36(2), 2012

   Links reduced HRV to stress / arousal states relevant to our stress index.

10. [10]Kartynnik, Y., Ablavatski, A., Grishchenko, I. & Grundmann, M.
    Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs ↗
    CVPR Workshop / arXiv:1907.06724, 2019

    MediaPipe Face Mesh geometry that underpins landmark indices for ROI, EAR, and blendshapes.

11. [11]Google MediaPipe
    Face Landmarker task documentation ↗
    MediaPipe Tasks Vision

    Product docs for the Face Landmarker model Emote AI loads in-browser.

12. [12]Ekman, P. & Friesen, W. V.
    Facial Action Coding System (FACS) ↗
    Consulting Psychologists Press, 1978

    Action-unit vocabulary that ARKit / MediaPipe blendshapes loosely approximate.

13. [13]Bond, C. F. & DePaulo, B. M.
    Accuracy of deception judgments ↗
    Pers. Soc. Psychol. Rev. 10(3), 2006

    Meta-analysis: human deception detection accuracy is barely above chance.

14. [14]National Research Council
    The Polygraph and Lie Detection ↗
    National Academies Press, 2003

    Authoritative review: physiological signals alone are not reliable evidence of deception.