◆ Research Problem 02

Behavioral-Biometric Emotion Inference for Adaptive Recommendation

Can a page infer how you feel from nothing but how you move, scroll, click and type — no camera, no microphone? The literature says the signal is real but weak. The contribution is not the model. It is the dataset that does not exist yet.

01 — The problem
Reading affect from interaction telemetry alone
Webcam and voice emotion recognition is intrusive, needs consent for a sensitive stream, and fails the moment the user looks away. Interaction telemetry is already available to any web page — the question is whether it carries enough signal to act on.
2 / 15
Emotional states Epp et al. could classify reliably from keystrokes — anger and excitement
CHI 2011
10–20%
Affect variance explained from cursor motion, for unseen participants
Yamauchi & Xiao 2018
25
Participants in SWELL-KW — the only public multi-channel corpus
Small by ML standards
0
Public datasets fusing mouse + scroll + click + keystroke under one affect label
← your contribution
Set expectations from the literature, not from the demo. Published cross-subject results in this field sit far below the 90%+ that student projects usually report. When you see a very high number, the cause is almost always participant leakage in the train/test split — covered in section 03 below.
End-to-end pipeline — raw events to recommendation
Fig. 8 · system
Behavioral emotion inference pipeline Browser events feed a raw event log, which is windowed into overlapping 30-second segments, converted to a feature vector, classified by a model, and mapped to a recommendation policy. A separate experience-sampling channel supplies the ground-truth emotion labels used to train the model. RUNTIME PATH Browser events pointermove, scroll, click, keydown/up, visibility, idle Raw event log JSON, millisecond timestamps — never pre-aggregated Windowing 30-second windows, 50% overlap Feature vector curvature, jerk, entropy, flight time, idle fraction … Model XGBoost baseline, then LSTM on the raw sequence Recommendation state + confidence → content policy, neutral on low conf. TRAINING-TIME LABEL CHANNEL Experience sampling · every 2–3 minutes 5-emoji picker + valence and arousal on a 1–5 SAM scale, so labels stay re-mappable ground truth Capture a neutral baseline per participant at the start of every session and z-score against it — absolute typing speed varies more between people than between emotions.
The label channel is the expensive half. Events are free; ground truth costs participant attention. Sampling every 2–3 minutes with a valence/arousal scale — rather than a single categorical tap — lets you re-map to 5 classes or 3 later without re-collecting.
02 — Data
What exists online, and the gap you have to fill
Four public datasets are relevant. Each covers part of the signal you want; none covers all of it under a single affect label. That gap is the reason this is a research project rather than a tutorial.
Dataset coverage — which channels each public corpus actually contains
Fig. 9 · the gap
Coverage of public datasets across interaction channels EmoSurv has keystroke data and emotion labels only. The Attentive Cursor Dataset has mouse and click data with an attention proxy label. SWELL-KW has keystroke, mouse, click and physiological data with a stress-focused label, but only 25 subjects. WESAD has physiological data and emotion labels but no interaction telemetry. No public dataset covers all interaction channels under a full affect label — the corpus you build would be the first. DATASET KEYSTROKE MOUSE CLICKS PHYSIO AFFECT LABEL SUBJECTS EmoSurv induced emotion, fixed + free text 5 classes not reported The Attentive Cursor Dataset search-task cursor traces attention proxy ~3,000 SWELL-KW knowledge work under stressors stress-focused 25 WESAD wearable sensors, lab protocol 3 states 15 The corpus you build 5 classes + SAM 30–100
● channel present ○ channel absent ◍ partial — proxy label, not full affect ● your contribution
Read the bottom row against the four above it. Physiological signals are deliberately absent from your corpus — the whole premise is that no extra hardware is required. Everything else lines up in a single row for the first time, which is what makes the dataset publishable on its own.

💾 Download these first

  • EmoSurv — keystroke timing with induced-emotion labels: anger, happiness, calmness, sadness, neutral. Fixed and free text, physical keyboards only. Open access. IEEE DataPort · DOI 10.21227/eae6-pk42
  • SWELL-KW — the closest existing match. 25 knowledge workers under email-interruption and time-pressure stressors, with mouse and keyboard logging, facial video, Kinect posture, ECG and skin conductance, plus validated task-load, mental-effort, emotion and stress questionnaires. Ships raw data, preprocessed data and extracted features. Radboud University · ICMI 2014 paper
  • The Attentive Cursor Dataset — roughly 3,000 subjects and ~2 hours of cursor interaction on a search task, with ground-truth labels and demographics. Attention rather than affect, but by far the largest trajectory corpus — ideal for pretraining mouse features. Leiva & Arapakis · Frontiers in Human Neuroscience 2020
  • WESAD — 15 subjects, neutral/stress/amusement, chest and wrist physiological signals. No interaction telemetry; use it only to sanity-check that your induced-stress labels behave like established ones. Academic, non-commercial use only. UCI ML Repository · Schmidt & Reiss 2018

🧪 Build the custom dataset

  1. Collapse the label spaceFive classes (happy, relaxed, neutral, sad, stressed) or three (positive, neutral, negative). Every reviewed study degrades badly past five.
  2. Instrument the pageCapture pointermove, scroll, click, keydown/keyup and visibilitychange with millisecond timestamps. Store raw streams as JSON — you will want to re-window later.
  3. Label by experience samplingPrompt every 2–3 minutes. Record valence and arousal on a 1–5 SAM scale alongside the emoji, not just one category.
  4. Induce, and baselineOptionally elicit with validated stimuli (film clips, IAPS images) as EmoSurv does. Always capture a neutral baseline per participant first.
  5. Window and extract30-second windows at 50% overlap; roughly 25 features per window (see the table in section 03).
  6. Recruit 30–100 peopleSeveral sessions each. A few thousand labelled windows is a viable first corpus.
  7. Split by participantNever by window. Report within-subject and cross-subject scores separately.
  8. Publish features, not keystrokesRaw keystroke timing is personally identifying and can reconstruct typed text. Release the anonymised feature-level dataset only.
Ethics and consent are not optional here. You are building a behavioural biometric — the same feature family used for continuous authentication, which means your raw data can identify people. Get informed consent for telemetry capture, never surface a mental-health inference to the user, treat low-confidence windows as neutral, and keep the recommendation layer suggestive rather than corrective.
03 — Modelling
Features, models, and the one mistake that fakes 95% accuracy
Behavioural features are strongly person-specific. That is what makes the split strategy — not the model architecture — the single decision that determines whether your results mean anything.
Participant leakage — why a random split reports a number you cannot publish
Fig. 10 · evaluation
Random window split versus grouped participant split With a random window split, windows from the same four participants appear in both the training and test sets, so the model can identify the person rather than the emotion and reports an inflated score. With GroupKFold grouped by participant, participants one to three are in training and participant four is held out entirely, producing an honest cross-subject score. ✗ Random window split train_test_split(X, y, shuffle=True) TRAIN P1 P2 P3 P4 P1 P3 P2 TEST P2 P4 P1 P3 Every test participant was also seen in training. The model learns "this is P2's typing rhythm", not "this is stress". Reports ~95% · means nothing ✓ GroupKFold by participant GroupKFold(...).split(X, y, groups=participant_id) TRAIN P1 P1 P2 P2 P3 P3 TEST P4 P4 P4 P4 is never seen during training. The score answers the question a reviewer will actually ask. Lower number · publishable Report both, and always against a majority-class baseline — neutral will dominate your labels.
This is the single most common way this class of project reports a fake result. Behavioural biometrics are used for identity precisely because they are person-specific — so a random split lets the model win by recognising the person. Group by participant, and report macro-F1 against a majority-class baseline, not raw accuracy.

Feature set per 30-second window

ChannelFeaturesWhy it carries affective signal
Mouse trajectory path curvature, acceleration, jerk, movement entropy, area under curve, direction changes Yamauchi & Xiao found area-under-curve and direction change to be the informative pair for affect.
Scroll mean and max speed, burst count, burst-to-pause ratio Bursty scrolling followed by long pauses is the pattern averages destroy — a key argument for sequence models.
Click clicks per minute, hesitation time before click, rage-click count, click duration Hesitation and rage-clicks are established frustration proxies in behavioural analytics.
Keystroke key hold time, flight time, WPM, backspace ratio, pause distribution The channel Epp et al. used; hold and flight times are the standard keystroke-dynamics primitives.
Session-level idle fraction, per-feature standard deviation and entropy Variability often separates states better than the mean does.
Normalisation z-score every feature against that participant's neutral baseline Without this, the model learns who is typing instead of how they feel.
Model order: tabular first, sequence second — and report both. Random Forest, XGBoost or LightGBM on windowed features give you a defensible baseline plus feature importances that tell you which channels actually matter. Only move to an LSTM or Transformer over the raw event sequence once you have beaten that baseline; the sequence models exist to capture patterns like "rapid scrolling then a long pause" that a 30-second average erases.

Papers to cite

CHI 2011 · Foundational

Identifying Emotional States Using Keystroke Dynamics

Epp, Lippold & Mandryk collected keystrokes with self-reported affect and built classifiers for 15 emotional states. Usable accuracy was reached for anger and excitement only — the result that should calibrate your own target.

Record on dblp →
Cognitive Science 2018

Reading Emotion From Mouse Cursor Motions

Yamauchi & Xiao extracted 16–26 trajectory features during choice-reaching tasks with music, film and image elicitation. Models trained on known participants explained 10–20% of the variance in positive-affect and attentiveness ratings for unknown participants.

Wiley Online Library →
Survey · Start here

A Review of Emotion Recognition Methods Based on Keystroke Dynamics and Mouse Movements

Kołakowska's survey of features, elicitation protocols and reported accuracies across both channels. Pair it with Does Keystroke Dynamics Tell Us About Emotions?, a systematic review that documents how earlier dataset constructions went wrong — read it before you collect anything.

IEEE HSI 2013 · SLR 2020
Recommender half

A Survey of Affective Recommender Systems

Classifies affective recommenders into attitude-aware, emotion-aware, mood-aware and hybrid. Pair with the heterogeneity-aware Bayesian music recommender, which models emotion variation across and within users — the failure mode a naive mood-to-playlist lookup runs straight into.

arXiv:2508.20289 · arXiv:2406.14090
← Research Problem 01 MCP Tool Poisoning & Defense Architecture How malicious metadata in third-party MCP tool descriptions tricks LLM agents into silent data exfiltration — and how to build a 3-layer runtime watchdog proxy.