benchmark demo error
Hello,
we have an AI core module https://up-board.org/ai-edge/ai-core/ connected through PCIe to main Xeon Processor running Linux Centos.
In reality we can see Movidius MyriadX connected directly via USB to root Hub onboard :
lsusb
Bus 001 Device 002: ID 8087:8000 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 002: ID 03e7:2485 Intel Movidius MyriadX
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 006: ID 04e2:1410 Exar Corp. XR21V1410 USB-UART IC
Bus 002 Device 005: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 002 Device 003: ID 2c42:5114
Bus 002 Device 004: ID 0424:4050 Standard Microsystems Corp.
Bus 002 Device 002: ID 0424:2640 Standard Microsystems Corp. USB 2.0 Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
later on lsusb -t gives :
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
|__ Port 1: Dev 96, If 0, Class=Vendor Specific Class, Driver=, 480M
lsusb -v :
Bus 004 Device 096: ID 03e7:2485 Intel Movidius MyriadX
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x03e7 Intel
idProduct 0x2485 Movidius MyriadX
bcdDevice 0.01
iManufacturer 1 Movidius Ltd.
iProduct 2 Movidius MyriadX
iSerial 3 03e72485
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 32
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
We can usually run the benchmark demo OpenVINO application without problems but we sometimes see benchmark OpenVINO demo crashing with segmentation fault or other occasions where Movidius is not enumerated.
I can now trigger the segmentation fault with 100% success by continuously launching benchmark demo and stopping it with Ctrl-C. If I stop it near the very beginning and immediately restart it, after few attempts I get the Movidius and the benchmark demo completely stuck.
here is one of the log when we get into error :
[root@xxx]# /root/inference_engine_samples_build/intel64/Release/benchmark_app -i /root/inference_engine_samples_build/intel64/Release/car.png -m /root/squeezenet1.1_FP16/squeezenet1.1.xml -d MYRIAD
[ INFO ] InferenceEngine:
API version ............ 1.6
Build .................. custom_releases/2019/R1_c9b66a26e4d65bb986bb740e73f58c6e9e84c7c2
[Step 1/8] Parsing and validation of input args
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] /root/inference_engine_samples_build/intel64/Release/car.png
Progress: [....................] 100.00% done
[Step 2/8] Loading plugin
[ INFO ]
API version ............ 1.6
Build .................. 22443
Description ....... myriadPlugin
Progress: [....................] 100.00% done
[Step 3/8] Read IR network
[ INFO ] Loading network files
[ INFO ] Network batch size: 1, precision: FP32
Progress: [....................] 100.00% done
[Step 4/8] Configure input & output of the model
[ INFO ] Preparing output blobs
Progress: [....................] 100.00% done
[Step 5/8] Loading model to the plugin
I: [ncAPI] [ 280563] ncDeviceOpen:737 ncDeviceOpen() XLinkBootRemote is running for 4.1-ma2480...
W: [ncAPI] [ 285664] ncDeviceOpen:750 ncDeviceOpen() XLinkBootRemote returned error X_LINK_COMMUNICATION_FAIL for 4.1-ma2480
E: [ncAPI] [ 253.882114] benchmark_app[2398]: segfault at 0 ip 00007f0a965fcb20 sp 00007ffe09c07968 error 4[ 300664] ncD in libc-2.17.so[7f0a964bd000+1c3000]
eviceOpen:799 Failed connection to device (4.1-ma2480) with error 5
Segmentation fault
[root@xxxt]#
At this point we have seen at least in one occasion where ProductID and manufacturer appear with "Error" on output to lsusb command regarding Movidius. But other times Movidius still looks being there in the lsusb enumerated peripherals.
Once the system is in this state it is not possible to run again benchmark test correctly even after reboot command, that issues an entire reboot of the main processor operating system.
After reboot command, Movidius is not enumerated as connected peripheral through USB and lsusb command does not list it anymore.
Of course at this point benchmark demo OpenVINO still crashes .
Only a power cycling the entire system allows to recover correct communication and allows to run again the benchmark OpenVINO demo successfully.
Do you have any suggestion or idea what we could try in order to make the system more robust and avoid this problem to occur ?
I updated to OpenVino latest version 3.1 and problem still happens
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] /root/inference_engine_samples_build/intel64/Release/car.png
[ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
API version ............ 2.1
Build .................. custom_releases/2019/R3_ac8584cb714a697a12f1f30b7a3b78a5b9ac5e05
Description ....... API
[ INFO ] Device info:
MYRIAD
myriadPlugin version ......... 2.1
Build ........... 32974
[Step 3/11] Reading the Intermediate Representation network
[ INFO ] Loading network files
[ INFO ] Read network took 9.01 ms
[Step 4/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1, precision: FP16
[Step 5/11] Configuring input of the model
[Step 6/11] Setting device configuration
[Step 7/11] Loading the model to the device
W: [ncAPI] [ 890412] [benchmark_app] ncDeviceOpen:651 Failed to find a device, rc: X_LINK_DEVICE_NOT_FOUND
[ ERROR ] Can not init Myriad device: NC_ERROR
but with this latest version 3.1 I cannot see "segmentation fault" , however it suffers of same problem of not communicating anymore with Movidius correctly if I CTRL-C while the benchmark test is starting up.
and Myriad disappears from lsusb list :
lsusb
Bus 001 Device 002: ID 8087:8000 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 03e7:f63b Intel
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 006: ID 04e2:1410 Exar Corp. XR21V1410 USB-UART IC
Bus 002 Device 005: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 002 Device 003: ID 2c42:5114
Bus 002 Device 004: ID 0424:4050 Standard Microsystems Corp.
Bus 002 Device 002: ID 0424:2640 Standard Microsystems Corp. USB 2.0 Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Do you have any idea what the cause of this could be ?
thank you
Kind Regards