Reference Models

Timings

The tables in this section contain inference timings for a set of representative models. The quantized models have been imported and compiled offline using SyNAP toolkit. The floating point models are benchmarked for comparison purpose with the corresponding quantized models.

The mobilenet_v1, mobilenet_v2, posenet and inception models are open-source models available in tflite format from TensorFlow Hosted Models page: https://www.tensorflow.org/lite/guide/hosted_models

yolov5 models are available from https://github.com/ultralytics/yolov5, while yolov5_face comes from https://github.com/deepcam-cn/yolov5-face.

Other models come from AI-Benchmark APK: https://ai-benchmark.com/ranking_IoT.html.

Some of the models are Synaptics proprietary, including test models, object detection (mobilenet224), super-resolution and format conversion models.

The model test_64_128x128_5_132_132 has been designed to take maximum advantage of the computational capabilities of the NPU. It has 64 5x5 convolutions with a [1, 128, 128, 132] input and output. Its execution requires 913’519’411’200 operations (0.913 TOPs). Inference time shows that in the right conditions VS640 and SL1640 achieve above 1.6 TOP/s while VS680 and SL1680 able to achieve above 7.9 TOP/s. For 16-bits inference the maximum TOP/s can be achieved with test_64_64x64_5_132_132. With this model we achieve 0.45 TOP/s on VS640/SL1640 and above 1.7 TOP/s on VS680/SL1680. For actual models used in practice it’s very difficut to get close to this level of performance and it’s hard to predict the inference time of a model from the number of operation it contains. The only reliable way is to execute the model on the platform and measure.

Remarks:

  • In the following tables all timing values are expressed in milliseconds

  • The columns Online CPU and Online NPU represent the inference time obtained by running the original tflite model directly on the board (online conversion)

  • Online CPU tests have been done with 4 threads (--num_threads=4) on both vs680 and vs640

  • Online CPU tests of floating point models on vs640 have been done in fp16 mode (--allow_fp16=true)

  • Online NPU tests executed with the timvx delegate (--external_delegate_path=libvx_delegate.so)

  • The Offline Infer column represents the inference time obtained by using a model converted offline using SyNAP toolkit (median time over 10 consecutive inferences)

  • The Online timings represent the minimum time measured (for both init and inference). We took minimim instead of average because this is measure less sensitive to outliers due to the test process being temporarily suspended by the CPU scheduler

  • Online timings, in particular for init and CPU inference, can be influenced by other processes running on the board and the total amount of free memory available. We ran all tests on Android AOSP/64bits with 4GB of memory on VS680 and 2GB on VS640. Running on Android GMS or 32-bits OS or with less memory can result in longer init and inference times

  • Offline tests have been done with non-contiguous memory allocation and no cache flush

  • Models marked with * come precompiled and preinstalled on the platform

Inference Timings On VS680 And SL1680

These tables show the inference timings for a set of models on VS680 and SL1680. All tests have been done on 64-bits OS with 4GB of memory.

Table 2 Synaptics models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

convert_nv12@1920x1080_rgb@1920x1080

17.52

30.81

*

convert_nv12@1920x1080_rgb@224x224

14.25

1.27

*

convert_nv12@1920x1080_rgb@640x360

13.55

5.14

*

sr_fast_y_uv_1280x720_3840x2160

317

32.88

18.09

11.46

*

sr_fast_y_uv_1920x1080_3840x2160

776

50.56

20.40

17.50

*

sr_qdeo_y_uv_1280x720_3840x2160

149

32.49

21.84

20.59

*

sr_qdeo_y_uv_1920x1080_3840x2160

233

38.41

24.11

25.84

*

mobilenet224_full80

66.28

25.17

*

mobilenet224_full1

57.71

14.23

*

test_64_128x128_5_132_132

50.07

119.34

Table 3 Open models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

inception_v4_299_quant

500.54

13502

17.80

100.79

19.59

mobilenet_v1_0.25_224_quant

3.37

166

0.81

2.61

0.77

*

mobilenet_v2_1.0_224_quant

18.60

854

1.85

6.13

1.79

*

posenet_mobilenet_075_float

34.44

61.78

*

posenet_mobilenet_075_quant

28.60

382

6.01

1.84

2.32

yolov8s-pose

14.61

30.79

*

yolov5m-640x480

6606

113.88

54.11

118.82

yolov5s-640x480

2672

72.27

22.17

75.83

yolov5s_face_640x480_onnx_mq

13.00

31.88

*

Table 4 AiBenchmark 4 models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

deeplab_v3_plus_quant

231.76

4090

60.73

7.68

59.81

dped_quant

335.63

1019

8.93

4.74

8.82

inception_v3_float

370.95

436.74

inception_v3_quant

267.95

7210

9.47

59.55

10.22

mobilenet_v2_b4_quant

53.54

875

12.50

11.53

13.63

mobilenet_v2_float

20.84

35.40

mobilenet_v2_quant

18.72

886

2.02

9.27

1.98

mobilenet_v3_quant

51.89

1089

9.76

13.15

10.15

pynet_quant

976.61

3175

18.56

24.45

19.30

srgan_quant

1513.86

3517

54.58

14.72

56.95

unet_quant

265.51

487

9.34

7.73

14.80

vgg_quant

1641.18

2177

29.77

10.74

30.07

Table 5 AiBenchmark 5 models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

crnn_float

240.53

217.89

crnn_quant

113.52

40641

22.54

217.19

23.03

deeplab_v3_plus_float

1181.26

1447.74

deeplab_v3_plus_quant

674.10

2381

97.98

16.68

103.05

dped_float

4043.67

2353.72

dped_instance_float

2236.11

1386.78

dped_instance_quant

3422.17

288

229.41

5.43

953.44

dped_quant

2249.53

3266

196.30

18.70

199.15

efficientnet_b4_float

432.00

579.98

efficientnet_b4_quant

228.86

9649

162.06

54.48

166.33

esrgan_float

1445.08

1522.63

esrgan_quant

770.69

2119

93.78

5.08

101.56

imdn_float

2553.12

2382.78

imdn_quant

1350.97

3215

165.92

8.63

155.46

inception_v3_float

371.39

437.60

inception_v3_quant

221.61

7254

10.26

76.98

11.38

mobilenet_v2_b8_float

152.22

203.15

mobilenet_v2_b8_quant

91.44

889

25.91

14.69

27.18

mobilenet_v2_float

20.98

36.08

mobilenet_v2_quant

12.28

968

2.11

10.04

2.07

mobilenet_v3_b4_float

351.62

461.38

mobilenet_v3_b4_quant

359.39

1497

97.86

20.36

101.09

mobilenet_v3_float

91.33

114.77

mobilenet_v3_quant

97.05

1706

19.82

15.14

20.92

mv3_depth_float

132.65

194.37

mv3_depth_quant

218.06

1513

71.09

15.74

90.96

punet_float

2612.87

1796.33

punet_quant

1660.59

2019

155.60

14.06

149.79

pynet_float

2836.85

1620.06

pynet_quant

2100.18

3441

137.39

15.80

135.94

resnet_float

0.10

2.86

resnet_quant

0.41

132

0.13

3.95

0.12

srgan_float

6192.96

2921.47

srgan_quant

4220.89

12224

200.90

29.85

208.23

unet_float

2909.00

2132.16

unet_quant

1710.41

775

69.08

19.11

95.29

vsr_float

820.35

974.12

vsr_quant

580.30

2124

155.28

20.45

133.86

xlsr_float

518.61

532.46

xlsr_quant

470.63

1700

36.20

3.93

31.38

yolo_v4_tiny_float

187.81

157.75

yolo_v4_tiny_quant

311.65

1406

6.62

4.69

6.03

Inference Timings On VS640 And SL1640

These tables show the inference timings for a set of models on VS640 and SL1640. All tests have been done on 64-bits OS with 2GB of memory.

Table 6 Synaptics models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

convert_nv12@1920x1080_rgb@1920x1080

17.48

34.49

*

convert_nv12@1920x1080_rgb@224x224

15.14

1.25

*

convert_nv12@1920x1080_rgb@640x360

14.70

5.29

*

sr_fast_y_uv_1280x720_3840x2160

274

53.03

17.87

17.01

*

sr_fast_y_uv_1920x1080_3840x2160

524

86.39

20.35

25.90

*

sr_qdeo_y_uv_1280x720_3840x2160

20.33

26.16

*

sr_qdeo_y_uv_1920x1080_3840x2160

22.03

33.56

*

mobilenet224_full80

718.96

52.98

*

mobilenet224_full1

595.52

36.53

*

test_64_128x128_5_132_132

63.95

563.81

Table 7 Open models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

inception_v4_299_quant

255.94

21481

54.07

127.13

53.82

mobilenet_v1_0.25_224_quant

2.80

244

1.00

4.99

0.93

*

mobilenet_v2_1.0_224_quant

12.31

1203

2.40

14.21

2.31

*

posenet_mobilenet_075_float

27.96

90.06

*

posenet_mobilenet_075_quant

18.70

565

9.76

2.48

4.13

yolov8s-pose

20.66

54.59

*

yolov5m-640x480

10657

175.90

60.64

178.00

yolov5s-640x480

4264

101.73

24.90

103.36

yolov5s_face_640x480_onnx_mq

27.31

59.63

*

Table 8 AiBenchmark 4 models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

deeplab_v3_plus_quant

158.81

3679

82.47

8.31

70.85

dped_quant

134.25

1694

25.84

6.71

25.72

inception_v3_float

229.07

706.98

inception_v3_quant

146.25

11130

30.21

80.70

29.82

mobilenet_v2_b4_quant

47.85

1373

18.95

13.60

18.39

mobilenet_v2_float

18.27

52.41

mobilenet_v2_quant

12.44

1282

2.57

13.81

2.44

mobilenet_v3_quant

47.99

1593

12.57

16.38

11.91

pynet_quant

447.61

4803

57.11

31.04

56.30

srgan_quant

829.17

5232

121.92

15.97

121.75

unet_quant

159.01

745

18.58

9.93

24.20

vgg_quant

572.70

3258

103.66

10.65

102.65

Table 9 AiBenchmark 5 models

Model

Online CPU Infer

Online GPU Infer

Online NPU Init

Online NPU Infer

Offline NPU Init

Offline NPU Infer

crnn_float

140.70

352.45

crnn_quant

87.40

70679

33.96

284.46

33.66

deeplab_v3_plus_float

976.50

2465.12

deeplab_v3_plus_quant

580.25

3889

137.92

19.01

133.41

dped_float

2197.13

4492.49

dped_instance_float

1458.77

2556.78

dped_instance_quant

3565.00

438

366.10

dped_quant

1166.57

4025

340.13

14.97

326.37

efficientnet_b4_float

396.99

945.97

efficientnet_b4_quant

266.18

15393

202.61

74.93

200.67

esrgan_float

971.15

2632.86

esrgan_quant

501.45

2394

147.30

5.28

147.75

imdn_float

1596.76

4181.09

imdn_quant

1123.80

5018

281.54

5.57

269.71

inception_v3_float

228.96

724.73

inception_v3_quant

157.53

11300

30.83

96.69

30.12

mobilenet_v2_b8_float

132.81

338.71

mobilenet_v2_b8_quant

88.94

1395

39.21

15.75

38.26

mobilenet_v2_float

18.09

52.04

mobilenet_v2_quant

11.68

1395

2.69

14.11

2.58

mobilenet_v3_b4_float

327.02

765.60

mobilenet_v3_b4_quant

364.18

2358

136.42

22.52

135.36

mobilenet_v3_float

90.62

181.89

mobilenet_v3_quant

100.55

2586

24.74

20.79

23.85

mv3_depth_float

189.75

331.05

mv3_depth_quant

301.72

2307

80.36

18.43

98.45

punet_float

1856.32

3173.80

punet_quant

1572.03

2736

259.45

11.74

249.44

pynet_float

2747.17

2833.27

pynet_quant

2126.74

5113

282.27

17.12

275.28

resnet_float

0.08

2.16

resnet_quant

0.04

205

0.17

5.09

0.23

srgan_float

3295.12

5420.38

srgan_quant

1740.11

17125

423.24

29.30

420.17

unet_float

1726.05

3776.94

unet_quant

1315.86

1089

155.08

14.68

195.52

vsr_float

680.60

1905.83

vsr_quant

621.03

1732

200.22

11.42

156.74

xlsr_float

595.84

987.90

xlsr_quant

570.30

2063

42.01

3.27

41.15

yolo_v4_tiny_float

125.44

254.04

yolo_v4_tiny_quant

321.50

2064

13.68

8.08

12.73

Super Resolution

Synaptics provides two proprietary families of super resolution models: fast and qdeo, the former provides better inference time, the latter better upscaling quality. They can be tested using synap_cli_ip application, see synap_cli_ip.

These models are preinstalled in $MODELS/image_processing/super_resolution .

Table 10 Synaptics SuperResolution Models on Y+UV Channels

Name

Input Image

Ouput Image

Factor

sr_fast_y_uv_960x540_3840x2160

960x540

3840x2160

4

sr_fast_y_uv_1280x720_3840x2160

1280x720

3840x2160

3

sr_fast_y_uv_1920x1080_3840x2160

1920x1080

3840x2160

2

sr_qdeo_y_uv_960x540_3840x2160

960x540

3840x2160

4

sr_qdeo_y_uv_1280x720_3840x2160

1280x720

3840x2160

3

sr_qdeo_y_uv_1920x1080_3840x2160

1920x1080

3840x2160

2

sr_qdeo_y_uv_640x360_1920x1080

640x360

1920x1080

3

Format Conversion

Conversion models can be used to convert an image from NV12 format to RGB. A set of models is provided for the most commonly used resolutions. These models have been generated by taking advantage of the preprocessing feature of the SyNAP toolkit (see Preprocessing) and can be used to convert an image so that it can be fed to a processing model with RGB input.

These models are preinstalled in $MODELS/image_processing/preprocess and can be tested using synap_cli_ic2 application, see synap_cli_ic2.

Table 11 Synaptics Conversion Models NV12 to RGB 224x224

Name

Input Image (NV12)

Ouput Image (RGB)

convert_nv12@426x240_rgb@224x224

426x240

224x224

convert_nv12@640x360_rgb@224x224

640x360

224x224

convert_nv12@854x480_rgb@224x224

854x480

224x224

convert_nv12@1280x720_rgb@224x224

1280x720

224x224

convert_nv12@1920x1080_rgb@224x224

1920x1080

224x224

convert_nv12@2560x1440_rgb@224x224

2560x1440

224x224

convert_nv12@3840x2160_rgb@224x224

3840x2160

224x224

convert_nv12@7680x4320_rgb@224x224

7680x4320

224x224

Table 12 Synaptics Conversion Models NV12 to RGB 640x360

Name

Input Image (NV12)

Ouput Image (RGB)

convert_nv12@426x240_rgb@640x360

426x240

640x360

convert_nv12@640x360_rgb@640x360

640x360

640x360

convert_nv12@854x480_rgb@640x360

854x480

640x360

convert_nv12@1280x720_rgb@640x360

1280x720

640x360

convert_nv12@1920x1080_rgb@640x360

1920x1080

640x360

convert_nv12@2560x1440_rgb@640x360

2560x1440

640x360

convert_nv12@3840x2160_rgb@640x360

3840x2160

640x360

convert_nv12@7680x4320_rgb@640x360

7680x4320

640x360

Table 13 Synaptics Conversion Models NV12 to RGB 1920x1080

Name

Input Image (NV12)

Ouput Image (RGB)

convert_nv12@426x240_rgb@1920x1080

426x240

1920x1080

convert_nv12@640x360_rgb@1920x1080

640x360

1920x1080

convert_nv12@854x480_rgb@1920x1080

854x480

1920x1080

convert_nv12@1280x720_rgb@1920x1080

1280x720

1920x1080

convert_nv12@1920x1080_rgb@1920x1080

1920x1080

1920x1080

convert_nv12@2560x1440_rgb@1920x1080

2560x1440

1920x1080

convert_nv12@3840x2160_rgb@1920x1080

3840x2160

1920x1080

convert_nv12@7680x4320_rgb@1920x1080

7680x4320

1920x1080