Scientists Built A Way To Assess Physical Cost of Smartphone Use

Every Tap Takes a Toll. A New Digital Body Can Finally Show How Much.

Every tap, swipe, and scroll leaves a digital footprint. Tech companies collect terabytes of this data daily, tracking which buttons get pressed and when. But those logs say nothing about what the body is actually doing: the finger stretching across a too-wide screen, the muscles straining to hit a tiny button, the physical fatigue quietly building up over hours of use.

A team of researchers has now built a system to change that. Called Log2Motion, it takes bare touchscreen interaction data and infers the physically plausible body movements that could have produced those logs, including estimated muscle activity across the arm and hand.

App designers have long been able to see that users tapped a particular button but had no way of knowing whether reaching it required an uncomfortable stretch or excessive muscle strain. Log2Motion offers estimates of physical effort and potential strain that could reveal those hidden costs, using nothing more than the interaction data apps already collect. Published in the Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, the work opens a new path for evaluating smartphone ergonomics before a single user study is run.

How Log2Motion Connects a Virtual Body to a Real Phone

Getting a digital body to interact with a real phone app sounds straightforward. In practice, it had never been done. Physics-based simulation environments, where realistic body models live, had no connection to the software environments where apps actually run. Previous attempts at body simulation in tech design amounted to crude arm movements in isolated virtual spaces, with no real app on the other end.

Log2Motion bridges that gap through a component the researchers call the “Screen Mirror.” Whatever a virtual Android phone is displaying gets projected onto a screen inside the physics simulation, and when the simulated finger taps that screen, the signal travels back to the emulator just as a real touch would. For the first time, a digital body could actually use a real app.

At the heart of the system is a body model called myoArm, which includes 63 muscle-tendon units controlling arm and finger movements, roughly the same muscles at work when a real person reaches across a table to tap a button. Researchers settled on the index finger as the primary input, citing studies showing it offers the greatest comfort and lowest error rate for touchscreen interactions.

Training a Digital Body to Move Like a Human

A body model alone does not move naturally on its own. To get there, the team used reinforcement learning, a type of AI training where an agent learns by doing: land the tap, earn a reward; miss the target, move too jerkily, or burn too much muscle effort, and take a penalty. Over many thousands of attempts, the system learned to move in ways that look and function like real human finger movements.

One detail that makes a real difference: the researchers built in motor noise, the slight natural imprecision present in every human nervous system. No real person taps the exact same spot twice, and neither does this digital one. To handle longer sequences of actions, movements were broken into building blocks called “motor operators,” essentially a vocabulary of basic touchscreen gestures, taps and swipes, that chain together into fluid multi-step interactions.

Testing Smartphone Ergonomics Against Real Human Movement

To verify the system actually produces human-like behavior, the team ran several tests. One checked whether the simulated movements follow Fitts’ Law, a well-established principle in motor science stating that smaller or more distant targets take longer to hit. Log2Motion passed across three trained movement styles: a precise “accurate” mode, a quicker “fast” mode that accepted more errors, and a balanced “normal” mode. Error rates fell within ranges reported in human performance studies.

A motion capture study pushed the comparison further. Six participants performed the same pointing tasks while cameras tracked their 3D fingertip paths in real time. Compared to those recordings, the synthesized movements fell within the range of variability seen across the real participants, with trajectory differences between the simulation and any given human comparable to the differences between two different people.

Smartphone Ergonomics at Scale

To show the system works beyond controlled lab conditions, the team ran it against the Android in the Wild dataset, a collection of 715,000 episodes of real device interactions. Log2Motion generated predictions for error rates, task duration, and physical effort across a wide range of real app tasks, pulling body-level insight from the kind of large, messy real-world data that already exists in abundance.

Effort estimates come from a validated model that calculates perceived physical exertion based on how hard each of the 63 muscles works relative to its maximum capacity. Designed to approximate perceived physical effort, the model has been shown to correlate with human exertion ratings.

For app designers, that adds up to something genuinely new: a way to learn, before a single person sits down to test a prototype, that a given button placement forces an uncomfortable reach or that a swipe demands more physical effort than it should. As the researchers note, the approach could aid both interface design and accessibility evaluation, giving designers a body-aware lens on decisions that, until now, have been made almost entirely blind to the physical side of phone use.

---

Disclaimer: This article is based on a peer-reviewed study presented at the 2026 CHI Conference on Human Factors in Computing Systems. Log2Motion is a research prototype and has not been commercially deployed. Effort and ergonomic estimates described in the study are modeled predictions, not direct physiological measurements, and findings may not generalize across all body types, hand sizes, or physical abilities.

---