Skip to content

efficientdet.md

Latest commit

 

History

History
241 lines (222 loc) · 19.5 KB

efficientdet.md

File metadata and controls

241 lines (222 loc) · 19.5 KB
Raw

EfficientDet TensorFlow Lite Benchmarks

Environment

  • HW
    • RasPi4: Raspberry Pi 4 Model B Rev 1.2 4GB
    • RasPi3 B+: Raspberry Pi 3 Model B+ 2GB
    • RasPi3 V1.2: Raspberry Pi 3 Model B V1.2
    • RasPi2 V1.1: Raspberry Pi 2 Model B V1.1
  • OS
    • Raspberry Pi OS 64bit (raspios_arm64-2021-04-09) Linux raspberrypi 5.10.36-v8+ #1418 SMP PREEMPT Thu May 13 18:19:53 BST 2021 aarch64 GNU/Linux
  • SW
    • TensorFlow Lite 2.5.0
    • OpenCV 4.5.2 (Self build)
    • Python3 3.7.3
    • ARM NN Delegate
      • Arm NN v21.05
      • Arm Compute Library v21.05
    • Coral EdgeTPU Delegate
      • libEdgeTPU Release Frogfish
      • Edge TPU Compiler version 15.0.340273435

Dataset

How to benchmarks

Source

# AP
$ Python object_detection_benchmark_tflite.py --model PATH_TO_MODEL_FILE --images_dir /PATH_TO_COCO2017VAL --annotation_path PATH_TO_instances_val2017.json --threads 4

# Latancy
$ python benchmark_tflite.py --model _PATH_TO_MODEL_FILE --thread=N(num threads) --count 350

Models

Results

COCO2017VAL mAP

  • -: Segmantation fault
  • *: Edge TPU Compiler Internal compiler error. Aborting!
  • **: Transfer on tag 1 failed. Abort. Deadline exceeded: USB transfer error 2 [LibUsbDataOutCallback].
    See details (google-coral/edgetpu#11).
Model Input FP32
XNNPACK delegate		 FP32
ARM NN delegate		 INT8 EdgeTPU
EfficientDet-lite0 320x320 26.03 26.03 25.65 25.54
EfficientDet-lite1 384x384 30.16 30.16 29.61 29.7
EfficientDet-lite2 448x448 33.16 33.16 32.72 28.5
EfficientDet-lite3 512x512 36.32 36.32 36 *
EfficientDet-lite3x 640x640 　　　　　　　38.68 38.68 38.24 **
EfficientDet-lite4 640x640 　　　　　　　39.7 - 39.38 *

Latency mean (ms)

  • FP32: XNNPACK delegate
  • *: Transfer on tag 1 failed. Abort. Deadline exceeded: USB transfer error 2 [LibUsbDataOutCallback].
    See details (google-coral/edgetpu#11).
  • **: bad_alloc error.
  • -: Cannot be measured because it hangs during measurement.
Model Input Kind Threads RasPi2 V1.2 32bit RasPi3 V1.2 32bit RasPi3 V1.2 64bit RasPi3 B+ 32bit RasPi3 B+ 64bit RasPi4 32bit RasPi4 64bit
EfficientDet-lite0 320x320 FP32 1 2467.89 893.71 824.30 781.42 720.83 390.53 354.84
2 1440.69 570.44 548.65 504.12 483.75 255.46 232.89
3 1087.39 476.24 469.08 419.68 414.90 221.25 201.95
4 935.22 429.04 467.31 378.83 342.94 211.91 193.39
INT8 1 2051.47 956.82 727.91 834.33 624.07 415.20 344.03
2 1195.27 575.96 452.35 489.97 388.97 252.07 232.77
3 908.50 444.88 366.23 380.45 317.03 200.77 143.85
4 801.27 381.82 323.70 325.58 277.76 119.99 122.06
EdgeTPU 1 725.72 439.64 525.40 138.78 468.48 134.98 99.77
EfficientDet-lite1 384x384 FP32 1 4937.80 1747.13 1613.29 1532.98 1191.83 736.27 691.68
2 2803.23 1143.85 1064.76 987.37 780.83 455.01 424.00
3 2095.07 910.80 899.22 806.23 650.56 387.35 362.36
4 1784.62 810.20 833.29 718.50 652.47 371.40 348.76
INT8 1 3951.50 1831.74 1399.45 1590.94 1131.02 774.12 546.90
2 2215.11 1065.62 850.91 919.48 675.48 452.86 324.85
3 1662.73 817.57 678.96 703.03 532.03 349.22 251.86
4 1466.31 696.81 593.82 595.76 458.83 293.58 211.73
EdgeTPU 1 1055.67 637.37 774.14 188.16 693.04 170.43 141.15
EfficientDet-lite2 448x448 FP32 1 8091.58 2894.44 2623.55 2487.69 2145.50 1175.96 1098.97
2 4640.80 1818.92 1720.31 1609.35 1272.05 733.35 690.23
3 3448.82 1451.73 1440.32 1297.75 1046.23 631.99 589.15
4 2903.33 1290.80 - 1147.47 963.23 593.51 565.30
INT8 1 6122.66 2871.39 2201.83 2488.91 1785.89 1196.17 857.87
2 3460.94 1657.75 1325.39 1431.96 1057.31 703.44 505.17
3 2612.35 1279.42 1058.44 1093.88 836.27 533.97 394.94
4 2287.92 1071.26 917.29 918.52 711.60 449.62 331.20
EdgeTPU 1 1707.71 369.0 1267.29 299.01 1138.33 285.80 219.76
EfficientDet-lite3 512x512 FP32 1 16927.50 6053.98 5486.44 5417.56 7206.10 2437.04 2308.64
2 9393.49 3633.08 3375.07 3163.71 4272.14 1456.22 1370.47
3 6898.60 2865.23 2733.98 2523.05 3471.59 1226.60 1167.75
4 5689.39 2487.72 - 2181.24 3149.46 1193.19 1139.21
INT8 1 11941.24 5528.07 4203.37 4712.73 3414.30 2270.01 1657.71
2 6501.27 3115.55 2449.24 2644.12 1965.97 1281.57 948.42
3 4871.55 2323.23 1890.65 1978.40 1495.18 968.61 716.57
4 4186.65 1907.66 1614.45 1636.10 1260.19 789.91 601.34
EfficientDet-lite3x 640x640 FP32 1 30619.75 10588.51 9554.02 9362.97 4094.17 4277.40 4089.07
2 17056.55 6310.35 5807.83 5609.27 2497.62 2563.39 2472.28
3 12317.63 4952.69 4752.24 4464.57 2009.69 2139.81 2067.91
4 10172.31 4329.66 - 3889.91 1814.83 2086.10 2015.04
INT8 1 21097.05 9829.84 7476.25 8399.33 2130.3 4034.59 2979.10
2 11673.98 5394.65 4312.28 4654.59 3463.11 2243.64 1685.61
3 8545.67 4045.67 3309.98 3445.96 2622.84 1664.28 1267.22
4 7345.22 3335.55 2798.00 2839.76 2191.00 1377.81 1050.79
EdgeTPU 1 * * * * * * *
EfficientDet-lite4 640x640 FP32 1 43830.09 14935.86 13472.70 13163.81 ** 5943.51 5743.08
2 23943.83 8675.71 8041.79 7790.65 ** 3511.83 3399.70
3 17214.86 6744.65 6535.24 6069.23 ** 2974.06 2835.45
4 14132.20 5898.34 - 5214.19 ** 2878.69 2847.02
INT8 1 28275.65 12926.22 10082.42 11087.51 8242.53 5237.25 4008.46
2 15463.81 7030.26 5710.84 6071.24 4601.04 2911.11 2244.46
3 11184.80 5199.34 4348.49 4446.50 3463.92 2145.94 1659.16
4 9468.00 4245.06 3608.48 3599.83 2841.39 1717.24 1360.11

ARM NN Delegate Latency mean (ms)

  • Delegate Parameter : {'backends': 'CpuAcc', 'logging-severity': 'info', 'number-of-threads': 'N', 'enable-fast-math': 'true'}
    N: Num of Threads
  • -: Segmantation fault
Model Input Kind Threads RasPi3 64bit RasPi4 64bit
EfficientDet-lite0 320x320 FP32 1 1016.98 520.00
2 618.40 320.08
3 497.48 258.25
4 442.76 229.91
INT8 1 2064.85 1187.71
2 1081.59 636.04
3 795.55 464.67
4 628.61 363.14
EfficientDet-lite1 384x384 FP32 1 1760.95 917.73
2 1063.56 568.35
3 846.41 454.73
4 772.97 413.50
INT8 1 3895.82 2300.32
2 2060.10 1215.24
3 1447.93 849.08
4 1158.61 676.47
EfficientDet-lite2 448x448 FP32 1 2712.48 1419.79
2 1627.73 884.95
3 1305.44 706.44
4 1176.66 624.21
INT8 1 5863.40 3524.52
2 3088.07 1850.40
3 2218.84 1315.06
4 1742.30 1016.84
EfficientDet-lite3 512x512 FP32 1 4891.51 2772.21
2 2867.34 1665.98
3 2292.63 1348.81
4 2048.76 1190.78
INT8 1 10307.17 6244.21
2 5374.72 3241.10
3 3824.26 2297.68
4 2954.27 1751.42
EfficientDet-lite3x 640x640 FP32 1 - 4699.25
2 - 2843.76
3 - 2238.66
4 - 2018.51
INT8 1 17802.66 10737.77
2 9204.81 5561.44
3 6509.33 3907.89
4 5128.52 3001.89
EfficientDet-lite4 640x640 FP32 1 - -
2 - -
3 - -
4 - -
INT8 1 22841.22 14074.93
2 11882.22 7238.01
3 8330.73 5070.61
4 6427.59 3883.23

Raspberry Pi 4 vs Raspberry Pi 5 vs N100

  • HW
    • Raspberry Pi 4 4GB
    • Raspberry Pi 5 8GB
    • CHUWI LarkBox X (Intel Alder Lake-N N100 12GB)
  • SW
    • TensorFlow Lite v2.18.0 XNNPACK delegate
Model Input Kind Num Threads Raspberry Pi 4 Raspberry Pi 5 N100
EfficientDet-lite0 320x320 FP32 1 328.65 104.01 76.59
2 210.16 72.88 47.26
3 182.03 64.30 39.54
4 175.85 62.25 34.93
INT8 1 197.75 44.94 54.76
2 116.99 29.30 33.35
3 90.83 24.78 27.72
4 78.01 22.88 23.88
EfficientDet-lite1 384x384 FP32 1 648.05 198.98 152.89
2 399.63 135.65 92.09
3 341.43 120.56 74.42
4 330.95 114.63 65.82
INT8 1 372.26 81.63 103.46
2 213.30 52.58 61.36
3 161.69 42.92 48.39
4 138.30 39.72 41.58
EfficientDet-lite2 448x448 FP32 1 1048.77 328.92 247.20
2 649.24 220.23 144.75
3 552.84 189.06 116.34
4 533.46 180.56 100.15
INT8 1 601.72 123.81 162.45
2 339.00 77.02 94.43
3 252.52 62.48 74.99
4 214.18 55.59 61.97
EfficientDet-lite3 512x512 FP32 1 2209.33 663.83 507.96
2 1341.90 421.97 290.20
3 1130.94 361.03 227.58
4 1128.58 350.48 192.61
INT8 1 1202.47 232.96 308.76
2 654.57 142.04 173.97
3 477.11 113.39 134.32
4 399.45 100.87 108.58
EfficientDet-lite3x 640x640 FP32 1 3942.57 1181.11 941.47
2 2389.41 749.79 529.43
3 2000.87 634.69 413.08
4 2046.68 615.69 347.94
INT8 1 2185.16 417.99 540.98
2 1175.21 246.85 300.16
3 848.33 195.63 228.84
4 702.02 170.98 186.74
EfficientDet-lite4 640x640 FP32 1 5514.79 1623.24 1313.08
2 3337.60 995.35 736.92
3 2742.85 824.13 572.08
4 2833.70 779.46 477.21
INT8 1 2988.07 550.06 733.56
2 1590.05 320.75 405.72
3 1145.18 250.39 304.51
4 952.46 216.47 244.17