Research note · Inference
Why 8-bit inference changes real time on entry-level phones
8-bit inference runs a neural network with 8-bit integers instead of 32-bit floating-point numbers. On an entry-level phone this is often what separates real time from a slideshow: on a Huawei FIG-LX1 2017, the public MoveNet Lightning model takes 38.6 ms per frame in floating point and 26.2 ms in 8-bit. This note explains why, and what the technique costs.
Last updated: By the Movelytics team
01Definition
What 8-bit inference is
A trained neural network is a set of numbers, the weights, and a recipe to combine them with an input. By default each number is a 32-bit floating-point value: precise, flexible, and expensive to compute.
8-bit inference, also called quantised or int8 inference, replaces those values with 8-bit integers. Each number can take 256 levels instead of billions. A scale factor stored next to each layer maps the integers back to real quantities.
Two things follow directly. The model file is about four times smaller, since 8 bits replace 32. And the arithmetic becomes integer arithmetic, which is the cheapest thing a processor does.
02Hardware
Why entry-level processors prefer integers
A recent high-end phone has options: a graphics processor, a dedicated neural accelerator, wide and fast processor cores. Floating-point models run well on all of them.
An entry-level phone that is a few years old has none of that in usable form. Its accelerators are absent, or present but inconsistent from one model to the next. What every phone has is a general-purpose processor with several cores. On that processor, three effects add up in favour of 8-bit:
- More numbers per instruction. Processor instructions that handle several values at once fit four times as many 8-bit values as 32-bit ones.
- Less memory traffic. A model four times smaller moves four times less data between memory and processor, and slow memory is the hidden cost on cheap phones.
- Less heat. Less work per frame means the phone takes longer to reach the temperature at which the system slows the processor down.
This is also why we count on the processor rather than on an accelerator for our light model: it is the one path that exists on every phone.
03Evidence
What a public model shows
The effect does not depend on our own model. MoveNet Lightning is published in floating point and in 8-bit: the same network, two files. We ran both on a Huawei FIG-LX1 2017 (Android 9), 4 threads.
Time per frame, model alone
Huawei FIG-LX1 2017 · Android 9 · 4 threads · median over 30 s · preliminary
- PoseTracker Model 8-bit18.4ms
three rounds: 18.4 · 18.4 · 18.4
- MoveNet Lightning 8-bit26.2ms
three rounds: 26.2 · 26.2 · 26.2
- MoveNet Lightning float38.6ms
three rounds: 38.6 · 38.7 · 38.0
Lower is better
Alone, the floating-point file takes 38.6 ms per frame and the 8-bit file 26.2 ms. Nothing changed except the number format.
Frames per second with the camera
Huawei FIG-LX1 2017 · camera at 30 fps · 10 minutes · preliminary
- PoseTracker Model 8-bit28.7fps
slowest 10 s window: 22.1 fps
- MoveNet Lightning 8-bit21.4fps
slowest 10 s window: 18.1 fps
- MoveNet Lightning float14.1fps
slowest 10 s window: 12.5 fps
Higher is better
With the camera, over 10 minutes, the floating-point file processes 14.1 frames per second and the 8-bit file 21.4. The camera delivers 30.
04Frame budget
The frame budget, in one line of arithmetic
A camera at 30 frames per second gives a new image every 33 milliseconds. That is the budget: everything the app does for one frame, the model included, must fit in it, or frames are dropped.
At 38.6 ms, the floating-point file is over budget before the camera, the drawing and the app's own logic have taken their share. At 26.2 ms, the 8-bit file is inside the budget when it runs alone, and no longer when the rest is added: it ends at 21.4 frames per second. At 18.4 ms, there is room left for the rest, and the result is 28.7.
The lesson is not that one figure is good and another bad. It is that on this class of phone the budget is so tight that the number format decides which side of it you are on.
05Cost
What it costs, and how to pay less
Rounding every number to 256 levels loses information. For a pose model the visible symptom is placement: a joint is found, but a little off. How much is lost depends on how the model was made.
- Converted afterwards
- A model trained in floating point and converted at the end can lose a noticeable part of its accuracy. The conversion is a lossy step applied to something that was never prepared for it.
- Designed for it
- A model built and trained with the 8-bit constraint in mind keeps most of its accuracy. The model learns to give good results with the numbers it will really have.
The practical rule is simple and often ignored: measure the file you ship. Accuracy quoted for the floating-point version of a model says little about its 8-bit version, and speed quoted for the 8-bit version says nothing about the accuracy of the floating-point one. On our benchmarks page, the accuracy and the speed of our build are measured on the same 8-bit file.
There are limits to what we show here. One phone, one unit. The runtime settings were chosen for our own 8-bit files, which could disadvantage the public files. And the accuracy cost of 8-bit for each model is not quantified in this note.
06Takeaway
What to ask a vendor
- Is the speed measured on the 8-bit file or on the floating-point one?
- Is the accuracy measured on that same file?
- On which phone, for how long, with the camera on or off?
Our own answers are on the benchmarks page. The wider picture of how we run models on the device is on the technology page. For a developer's comparison of public pose models, see PoseTracker's page on MediaPipe and MoveNet.
Private beta
Measure it on the devices your users hold.
The PoseTracker Model is in private beta.