Post-Touch
Expressive gesture in touchless creative interfaces · Design Project 1 · Department of Design, IIT Delhi
Long before language we made meaning with our hands. This project asks what that knowledge looks like when the hand, and not a learned device grammar, becomes the interface.
The instrument,
in use
A painter stands a metre from the screen and works in the open air. The hand alone moves the brush, picks the colour, and clears the canvas. There is no mouse, no pen, and nothing worn on the body. What the camera sees becomes paint in the same instant.
Everything that follows, the question, the system, and the data it leaves behind, begins from this one plain fact, that a hand in front of a webcam is enough.
To create
is human.
The oldest marks we have are handmade. People pressed pigment around their own hands on a cave wall. The gesture came first and the image followed.
We point before we can speak. We push away what we fear and shape the air to describe what we mean. The hand is, as Kant put it, the visible part of the brain.
It began as a tool.
It became an instrument.
Post-Touch started as a proposal for a tool, a fluid medium that would mirror the fluid nature of the hand. A canvas with no mouse and no stylus, where brush size and colour mapped to movement. The refinement turned that tool into a research instrument. The question was no longer whether we could paint without hardware. It was what expressive gesture actually is, and why no one had measured it.
Functional gesture
Studied as a command. Swipe, pinch, point. Success is measured by accuracy, agreement, and error rate. The body is asked to be precise and repeatable.
Expressive gesture
Modulated on purpose. Precision is sometimes abandoned and rhythm carries meaning. No prior study used a generative canvas to capture it, so it stayed invisible. The gap is methodological as much as empirical.
What does expressive gesture look like when the body, and not a learned device grammar, is the interface for creative mark making?
- 01
What kinematic parameters, velocity, depth, and dwell, characterise expressive gesture in mid air drawing?
- 02
Which spatial zones of the interaction volume are preferentially used, and how does this relate to fatigue?
- 03
What spontaneous gestural vocabularies do participants develop without prior instruction?
- 04
How should kinematic data be encoded visually so the resulting mark preserves expressive intent?
- 05
What design guidelines can be derived for inclusive expressive touchless interfaces?
The work rests on four ideas. Phenomenology, that the body is the site where perception and meaning happen and not a tool the mind drives. Kinematics, that speed, depth, and dwell describe how a gesture is made and not only what it points at. Creativity, that expressive work depends on uninterrupted attention an interface can support or break. And ergonomics, that every movement made in the air carries a physical cost the design has to account for.
Reading the hand used to
mean wearing the machine.
Each generation removed a layer of hardware until nothing was left between the hand and the camera.
Interaction meant suiting up. The Power Glove was expensive and tethered, and the machine, not the hand, was the focus.
The Kinect dropped the glove but still read the room with a special depth sensor.
A standard webcam is enough. Machine learning finds the hand in plain RGB video, which is what Post-Touch runs on.
The canvas is the instrument,
not the product.
Post-Touch tracks both hands in real time and turns bare movement into painted marks. It runs at thirty frames per second, logging position, velocity, and depth for each hand. The painting is not the goal. It is the trace the research reads.
- 01
Camera
Laptop webcam, 30 fps, 1280×720
- 02
Landmarks
MediaPipe Hands, 21 points per hand
- 03
Kinematics
Per frame speed, depth, dwell
- 04
Gesture
Open, pinch, fist, point
- 05
Brush
Width, alpha, pigment, then canvas
A vocabulary the
hand already knows.
Nothing here has to be learned. The system listens for gestures already in the body, so a first time participant can begin without instruction. This is the principle of primal mimesis.
Draw
The index fingertip is tracked in real time and its position drives the paint mark.
Pinch
Sliding a pinched hand left and right maps to the colour wheel, shifting the active hue.
Fist
Both hands closing at once changes the brush, the left held still while the right advances through the set.
Open palm
The palm becomes an eraser. Any stroke within its radius is lifted from the canvas.
Closing the
proprioceptive gap
Mid air has no surface to rest against, so the hand cannot feel where a mark will land. A ring anchored between the index and the thumb projects the exact point of contact and scales with the pinch aperture.
It is a small piece of pseudo haptics, a stand in for the resistance that is missing, and it resolved the disorientation almost immediately.
Eight brushes are organised into three families, with a ninth neutral line kept as a control. Each answers the same movement differently, mapping depth to weight, velocity to energy, and dwell to bloom. The mapping is where kinematics becomes image.
One gesture, many responses.
| Brush | Family | Depth | Velocity | Dwell |
|---|---|---|---|---|
| Glow | Mark | Stroke weight | Blur radius | Core opacity |
| Blossom | Mark | Branching count | Petal scale | Bloom radius |
| Ribbon | Mark | Nib width | Nib angle | Ink density |
| Neon | Mark | Glow spread | Strobe frequency | Core thickness |
| Scatter | Wet | Particle size | Spread radius | Particle count |
| Watercolor | Wet | Wash opacity | Edge graining | Bloom delay |
| Electric | Atmospheric | Branch depth | Arc frequency | Residual glow |
| Constellation | Atmospheric | Node density | Edge length | Star magnitude |
What the instrument records
Session data schema · one row written every frame
| timestamp | number | ms since session start |
| index.x / .y | number | canvas space position, 0 to 1 |
| index.z | number | depth relative to the wrist |
| velocity | number | px per frame, Euclidean |
| gesture | string | draw, lift, pause or fist |
| brushType | string | the active brush |
| pinchAperture | number | thumb to index span, normalised |
| strokeId | number | monotonic stroke counter |
Export · JSON and CSV · about 1800 frames a minute at 30 fps
Fig. 04 · one session as a ribbon · brush, gesture and velocity across sixty seconds
Nothing is summarised at capture. Every frame is written whole, so a session can be replayed, measured, and set beside another. Read back across early sessions, the move from draw to lift is by far the most frequent, the open palm pause for erasing is occasional, and a closed fist always marks a deliberate change of brush. The painting is only the surface. The record beneath it is what the study reads.
Velocity and dwell pull
the brushes apart.
Each brush settles into its own corner of the movement space. How fast the hand travels and how long it lingers are enough to tell them apart.
Fig. 02 · velocity and dwell by brush, fastest first
| Brush | Mean velocity | Dwell |
|---|---|---|
| Electric | ||
| Neon | ||
| Constellation | ||
| Glow | ||
| Ribbon | ||
| Blossom | ||
| Scatter | ||
| Watercolor | ||
| Vapor |
−0.81
velocity against dwell. The faster a brush runs, the less it lingers.
- Depth carries weightUse of the z axis runs about three times higher in Blossom than in Glow sessions.
Where the hand goes,
and for how long.
Fig. 03 · normalised landmark density across the canvas · N = 54,000 frames
Ergonomicsweet spot Extended
reach zone
Plotting every tracked landmark over a long session draws a clear map of effort. The hand returns again and again to a comfortable zone a little left of centre and around chest height. Reach to the far corners is brief and falls away as fatigue sets in, which is exactly where a reach guideline has to do its work.
Fig. 06 · typical stroke duration by brush, fastest first
Claude as a witness
When a session ends, the finished canvas and a compact kinematic summary are sent to a vision language model. It returns a short metaphorical title and a two sentence reading grounded in the body and its movement. The human gesture makes the work and the machine reads it afterward.
- Witness. It attends to the gesture as mark. It observes, it does not intervene.
- Interpreter. It reads only the finished trace, never the live hand.
- Authorship preserver. The expression precedes and determines the output. The reading cannot alter it.
- Generator. Post-Touch is not an AI art tool. The model adds no marks of its own.
Expressive gesture is not functional gesture performed artistically. It is a distinct register, marked by continuous modulation, spatial exploration, and rhythmic variation.
Participants consistently moved beyond the functional gesture space. They invented compound gestures the classifier never anticipated, like a slow pinch and draw to blend a wash. Four ideas carried the work. Primal mimesis, the reticle as pseudo haptics, the mapping of kinematics to visual form, and the machine as a respondent rather than a generator.
The synthesis
The final poster gathers the pivot, the system, and the early findings onto one sheet. It is the dark artifact this lighter page unpacks, reproduced as it was presented.
Outputs and what comes next
- D1
Kinematic Heatmap
Instrument built - D2
Spatial Lexicon
Data collection pending - D3
Kinematic to visual mappings
Prototype implemented - D4
Universal reach guidelines
Analysis pending