Study: How Accurate Is The Calorie Counter On Your Wearable Device?

Researchers at Japan’s National Institute of Health and Nutrition recently ran an impressively detailed study to compare 12 different wearable devices, including the Fitbit Flex, Jawbone UP24, Misfit Shine, Garmin Vivofit, and Withings Pulse O2, to two widely accepted methods of estimating calorie burn. The results, published in the May issue of JAMA Internal Medicine, offer some interesting insights.

The 19 volunteers in the study wore all 12 wearables at the same time. There were six wrist bands, four waist-mounted devices, and two pocket devices.

One part of the experiment involved spending 24 hours in a metabolic chamber, following a standardized protocol including “3 meals, deskwork, watching TV, housework, treadmill walking, and sleeping.” During this portion of the experiment, the caloric expenditure of each participant was measured by carefully monitoring the chamber’s temperature, gas composition, and so on. Based on the results, there were consistent differences in the devices. For example, the Jawbone and Garmin devices underestimated total energy expenditure (TEE), or calories burned, by a couple hundred calories on average, while the Fitbit and Misfit both overestimated the TEE.

For the second part of the experiment, participants drank “doubly labeled water,” which contains rare isotopes of both hydrogen and water. TEE can be estimated by tracking how long it takes for those isotope to appear in your urine. Participants then spent the next 15 days living their normal lives, while wearing all 12 devices at all times, and collecting their urine for later analysis. They were allowed to take the devices off “when bathing, special activities in which wearing the devices would be difficult, or when charging the battery.” The results suggested that all the devices underestimate the TEE, with only the Fitbit Flex and the Omron devices coming with a standard deviation of the TEE calculated by the doubly labeled water. Jawbone and Garmin produces lower values than the Fitbit and Misfit so that seems like a consistent finding. These results may not seem so surprising considering it’s far more likely that over a 15-day period you’d have a greater amount of time with the devices off for charging, sleeping, and those other “special activities.”

The conclusion of the study is that “the findings presented herein suggest that most wearable devices do not produce a valid measure of total energy expenditure.” One could argue that despite being inaccurate relative to the exact amount of TEE, most of the devices were consistently off, which means that they still could be a good measure of whether or not a person is burning more or fewer calories compared to a previous day. Statistical analysis showed that for the most part the devices correctly ranked the individual participants (i.e., if doubly labeled water showed that you were the third-highest calorie burner among the subjects so did most of the wearable devices). For self-tracking, that’s may be good enough.