- shivanand pegu
- prof sabyasachi paldas
- prof gourab kar
- department of design, IIT Delhi
Designing gesture interaction from the measured motor abilities of elderly users
with my own two hands
Design Project 2 at the Department of Design, IIT Delhi. The project designs mid air gesture interaction for elderly people from what their own hands can do, for the everyday controls that have become difficult for them to operate.
Guided by Prof. Sabyasachi Paldas and Prof. Gourab Kar, from August to November 2026. This page covers the work up to the second review in September. The third review is in November.
The title is taken from a song by Ben Harper.
Everyday controls assume steady hands and near vision
The controls of daily life are dense and small. A set top box remote carries several dozen buttons within a hand sized area, and a phone keypad, an appliance panel and a wall switchboard make comparable demands. Each target is small and identified by printed text a few millimetres high.
Operating any of them requires locating the control, orienting toward it, reading its labels and acting accurately. Each of these operations depends on fine motor control, near vision, or both. Where those capacities have declined, the control ceases to function as a control and its operation is delegated to another person.
Commercial gesture control on television, 2012
Samsung Smart Interaction on the ES8000 provided camera based gesture control on a consumer television. It is cited here as the best documented deployment of mid air gesture to a general consumer population, not as the context this project selects. Contemporary reviews recorded three recurring problems.
The camera needed a bright room. With the curtains drawn it did not register hands at all.
The cursor on screen was difficult to control with any precision.
Holding the arm up to wave was tiring. Later models cut the movement down to a finger wave.
From reviews of the UE55ES8000 and UE55H8000 in TechRadar, and of the voice and gesture controls in Trusted Reviews, listed with the references at the end of this page.
Reading the failure as a question of user selection
Lighting, cursor precision and fatigue are accurate descriptions of what went wrong. They do not on their own account for abandonment, because comparable levels of friction are tolerated in other interfaces without being abandoned. Tolerance for interaction cost is determined by the alternative available to the user.
Gesture control was offered to users whose remote already served them. Measured against a functioning remote, no level of friction was acceptable. Measured against delegating the task to another person, the same interaction costs are weighed differently. The population to which the system was offered, rather than the system itself, accounts for the outcome.
Relation to Design Project 1
Design Project 1, Post-Touch, argued that gesture research had concentrated on functional command and had little to say about expressive use. It built a browser based hand tracking instrument with per frame kinematic logging.
This project applies the same critical move, to users rather than to use. Gesture interface design has standardised on an assumed body which is young, steady and able. That assumption is rarely stated and has not been tested against the range of motor ability present in the intended user population.
Scope, and why elderly users rather than motor impairment generally
Full dexterity
Small buttons are usable. The remote already works, so a gesture system competes with a tool that fits. This is who Samsung sold to.
Age related decline
Broad movement holds. Small targets, fine positioning and dense layouts fail. This is the zone this project designs for.
Significant clinical impairment
Switch access, eye gaze and voice are the established answers. Gesture would likely be worse than what it replaces.
Research questions
In which everyday situations do elderly people find existing controls unusable, and which of those are suited to mid air gesture?
Additional questions
- What recognition tolerances keep gesture dependable for hands affected by tremor, stiffness and fatigue?
- What changes in the design of mid air gesture interaction when the gesture set is derived from the measured motor abilities of elderly users?
- What makes mid air gesture dependable enough for elderly people to use for everyday tasks in place of controls they can no longer operate?
Gesture elicitation with kinematic capture
In the established method a participant is given a task and asked what gesture they would make for it, and agreement is counted across participants. The task is a referent, the effect to be produced, such as a light coming on, rather than the action the participant performs.
The extension in this study asks whether that person can produce the same gesture again. Because Post-Touch already built the kinematic logger, every proposal is captured as motion data across repetitions, giving a consistency measure alongside the agreement score.
- Agreement
- Computed between participants for each referent, as the share of participant pairs who propose the same gesture, following Vatavu and Wobbrock.
- Consistency
- Computed within each participant across three repetitions of their own proposal, as the mean distance between repetitions, in hand lengths for the path of the movement and in radians for orientation.
Agreement indicates which gesture a group of people proposes. Consistency indicates whether each person can reproduce the gesture they proposed. Both are reported for every referent.
Session runner
A browser instrument for recording elicitation sessions in a participant’s home, without a network. It captures and nothing more. Analysis happens afterwards, against the files it writes.
- Twenty one landmarks per hand, recorded about thirty times a second.
- World coordinates in metres, centred on the hand, for shape and orientation.
- Normalised image coordinates, for the path of the movement.
- Each production is written to disk as it ends, so a failure loses one attempt.
- Runs with no network. The tracking model is stored on the laptop.
- Referent order is randomised per participant, and the seed recorded.
Session protocol
Each session takes forty five minutes of the participant’s time, seated throughout, in their own home. The setup happens before the clock starts, and the camera is placed where a device would plausibly live.
- Consent5 min
- Context conversation8 min
- Task2 min
- Practice2 min
- Elicitation20 min
- Debrief5 min
- Close3 min
The consent sheet is read aloud rather than handed over to be read, since small print is the subject of the project. The context conversation that follows asks what the participant uses every day, what they have stopped using, and what they ask somebody else to do for them. Gesture is not mentioned until it ends.
Each referent describes an effect and never an action. The light comes on, rather than wave to switch on the light, so that the vocabulary is the participant’s and not the researcher’s. No gesture is ever demonstrated, because the first gesture a participant sees becomes the template for everything after it. Each referent is performed three times, and the participant is told so in advance, so that a second request does not sound like a correction.
Imagine that the television, or the light, can see your hand. You do not need to touch anything.
मान लीजिए यह टीवी या यह बत्ती आपका हाथ देख सकती है। आपको कुछ छूना नहीं है।
Agreement rate by referent
Agreement is the share of participant pairs who propose the same gesture for a referent. With three participants the rate can only be 1, 0.33 or 0.
- 1.00
All three proposed the same gesture
fan faster, volume down
- 0.33
Two of the three agreed
call answer, call end, volume up, channel next, light off
- 0.00
All three proposed something different
fan slower, light dim, fan off, mute, screen off, light on, ac cooler, confirm
Within participant inconsistency by referent
The three repetitions of each participant’s gesture are compared in pairs, after the rise from the lap and the return have been removed. A higher value means the repetitions differ more.
Two referents with equal agreement
All three participants propose the same gesture for fan faster and for volume down, so both have an agreement rate of 1.00. The repetitions of fan faster vary about 2.6 times as much in path and nearly five times as much in orientation. A study that reports agreement alone would treat the two as equivalent.
fan faster
- Participant 1
- Participant 2
- Participant 3
- Agreement
- 1.00
- Path inconsistency, hand lengths
- 1.36
- Orientation inconsistency, radians
- 0.39
volume down
- Participant 1
- Participant 2
- Participant 3
- Agreement
- 1.00
- Path inconsistency, hand lengths
- 0.53
- Orientation inconsistency, radians
- 0.08
Reading agreement and consistency together
Agreement says whether people converge on one gesture. Consistency says whether each person can repeat their own. A referent is read by where it falls on both, and each combination suggests a different design response.
- Candidate for the vocabulary
- People converge on one gesture and can repeat it.
- volume down, 1.00 agreement, 0.53 inconsistency
- Needs a tolerant recogniser or a different gesture
- People converge on one gesture but cannot repeat it reliably. Agreement alone would accept it.
- fan faster, 1.00 agreement, 1.36 inconsistency
- Candidate for a personal gesture
- Each person has their own gesture and repeats it well, which suggests letting people set their own.
- ac cooler, 0.00 agreement, 0.26 inconsistency
- Unlikely to suit gesture
- People neither converge on a gesture nor repeat their own reliably.
- No referent fell here.
| Referent | Gesture | Share | Inconsistency |
|---|---|---|---|
| fan faster | circle clockwise | 3 of 3 | 1.36 |
| volume down | lower open palm | 3 of 3 | 0.53 |
| call answer | hand to ear | 2 of 3 | 1.29 |
| call end | close to a fist | 2 of 3 | 0.91 |
| volume up | raise open palm | 2 of 3 | 0.70 |
| channel next | tap in the air | 2 of 3 | 0.21 |
| light off | tap in the air | 2 of 3 | 0.15 |
No gesture was repeated for the other eight, fan slower, light dim, fan off, mute, screen off, light on, ac cooler, confirm.
Share is the number of participants who proposed the gesture. Inconsistency is the mean distance between repetitions of the path, in hand lengths, where higher means less consistent. The gesture marked in red was proposed for two referents.
Gesture vocabulary
The referents for which at least two participants proposed the same gesture. The same gesture, a tap in the air, was proposed for two different referents. A vocabulary cannot keep both as they stand, so one would need a different gesture.
Properties of the measurement found in testing
These follow from the camera and the tracking model, so they apply to every session.
- Handedness labels arrive inverted
- The tracking model assumes a mirrored image. The runner does not mirror its input, so a right hand is reported as left.
- World coordinates do not contain the movement
- They are centred on the hand. A swipe or a raise appears only in image coordinates, which are divided by the apparent size of the hand before comparison.
- Orientation needs three dimensions
- A palm down hand points its fingers at the camera, where an angle measured in the image plane is undefined. Palm normal and finger direction are used instead.
- Each trial contains the rise from the lap
- The rise and the return are nearly identical across repetitions and mask the gesture. Both are removed at the slowest frames either side of the raised span.
Tremor resolution at seated distance
Post-Touch operated at arm’s length. Daily life presents a range, from a handset held close to a switch across the room, with domestic seating commonly at 2.5 to 3 metres. Hand size in frame falls with distance, and evening light and backlighting reduce contrast further.
At two metres one pixel of a 1280 by 720 webcam spans about two millimetres at the hand, so small tremor cannot be resolved. A postural hold of five to ten seconds is proposed for each session, to test whether larger tremor can be detected.
1.64 mm per pixel
at 1.5 m
2.19 mm per pixel
at 2.0 m
2.74 mm per pixel
at 2.5 m
3.28 mm per pixel
at 3.0 m
If tracking does not hold at the distance a room requires
- Move the camera nearer, onto the table rather than under the screen.
- Use pose detection to locate the person, then crop before running hand tracking.
- Lean on larger and slower gestures, which ageing kinematics favour anyway.
- Move to a nearer context if the distance proves impossible.
Candidate domains and selection criteria
The brief covers daily life rather than a single device. The contexts for prototyping are chosen from these domains, against these criteria.
- Television and set top boxes
- Lights, fans and air conditioning
- Telephone calls, answering and dialling
- Kitchen and laundry appliances
- Switches and controls placed out of reach
- Summoning help or attention
- Frequency of the task in daily life
- Consequence of a recognition failure
- Whether the existing control is the actual barrier
- Whether the interaction can be performed seated
- Whether actuation is achievable within the semester
Limitations of the study design and of the measurement
- Small sample
- With three participants, agreement can only take three values and supports no statistical comparison. Consistency is measured within each person and is less affected.
- Convenience sample
- Participants come through family and friends, which favours urban, educated households in one city. Findings may not extend to elderly people generally.
- No comparison group
- The contrast with gesture designed for an able body is documentary. It draws on published studies and shipped vocabularies, and there is no control group.
- No working system
- Participants gesture at devices that do not respond. Proposals made without feedback may differ from gestures used with a working system over time.
- Resolution of one webcam
- At two metres one pixel spans about two millimetres, so small tremor cannot be resolved. Tracking also weakens with distance and poor light.
- One coder
- Gesture proposals are coded by the researcher alone. A second coder for a share of the videos would allow agreement between coders to be checked.
- Untested analysis choices
- Removing the rise from the lap, and the boundary between consistent and inconsistent repetition, still need checking on further sessions.
- Changed instructions
- Early sessions used a shorter instruction than later ones. Each session records which was used, so the two groups can be compared.
Expected contributions
- Design guidelines for mid air gesture interfaces intended for users with age related decline in dexterity and near vision.
- An elicitation dataset that records gesture agreement alongside per participant kinematic consistency, which is not standard in the elicitation literature.
- A working camera based probe in one domestic context, evaluated against the control the participant currently uses.
The third review, on 27 and 28 November 2026, adds the working probe, its testing with participants against their own controls, and the guidelines that follow from it.
References
- Wobbrock, J. O., Morris, M. R. and Wilson, A. D. 2009. User defined gestures for surface computing. Proceedings of CHI 2009, 1083 to 1092. doi.org/10.1145/1518701.1518866
- Vatavu, R. D. and Wobbrock, J. O. 2015. Formalizing agreement analysis for elicitation studies. Proceedings of CHI 2015. doi.org/10.1145/2702123.2702223
- Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S. and Froehlich, J. 2011. Ability based design. ACM Transactions on Accessible Computing 3, 3. doi.org/10.1145/1952383.1952384
- TechRadar. Samsung UE55ES8000 review. techradar.com
- TechRadar. Samsung UE55H8000 review. techradar.com
- Trusted Reviews. Samsung Smart TV voice and gesture control systems review. trustedreviews.com