How to recover from I2C bus lockup?

waynepiekarski

Hi,

I've been testing an Up Board with a variety of I2C peripherals and encountering some problems with buggy I2C devices. Occasionally I will run into a situation where I2C read/writes will timeout, and the bus master loses arbitration. i2cdetect gets stuck. If I unplug the I2C devices, and replug them back in, everything usually works again, but this isn't a long term solution that works automatically. I need a system that can recover from faults like this.

I've read various articles such as AN-686 which discusses clocking out the problem.
https://www.analog.com/media/en/technical-documentation/application-notes/54305147357414AN686_0.pdf
I tried this out, and was able to sometimes recover the bus by using the GPIO pins attached to the I2C lines to toggle SCL ten times, and then SDA ten times. The bus would become available again, and i2cdetect would work again. However, I found other situations where it appears like the I2C master itself was stuck. I could unplug the I2C peripherals, and it was still stuck. Rebooting the Up Board did fix the problem, so it appears like it could be correctable in software.

I tried messing with the I2C driver but that did not fix anything.
echo -n "808622C1:00" > /sys/bus/platform/drivers/i2c_designware/unbind
echo -n "808622C1:01" > /sys/bus/platform/drivers/i2c_designware/unbind
echo -n "808622C1:00" > /sys/bus/platform/drivers/i2c_designware/bind
echo -n "808622C1:01" > /sys/bus/platform/drivers/i2c_designware/bind

Rather than doing these hacks that aren't working, is there a better way to recover?
(1) How to reset the onboard I2C master on the UpBoard without a reboot?
(2) How to clock out the bus to unlock any peripherals on the I2C bus?

I'm using Ubuntu 18.04 with the upboard kernel: 4.15.0-37-generic #40~upboard06-Ubuntu

Thanks.

DCleri

This seems to be a bug in the I2C kernel driver, can you please try the newer kernel we have available for UP Series?
https://wiki.up-community.org/Ubuntu_18.04

waynepiekarski

Thanks for that, I will give this kernel update a try soon. However, a driver fix might address the I2C master being stuck (which I've found to be quite rare), but will it help to deal with stuck peripherals as well? (which happens often with buggy circuits that I'm trying to fix).

Thanks!

waynepiekarski

I updated my kernel in the system as instructed, and now uname -a is Linux upboard32 5.0.0-1-generic #2~upboard5-Ubuntu SMP Wed Apr 8 06:17:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I think I have multiple problems here that all look very similar ...

1. I have been testing some flaky I2C peripherals, and sometimes these lock up the bus. I would still like to know how to reset the bus to recover from this problem. In dmesg I see the following:

[ 6287.077062] i2c_designware 808622C1:01: controller timed out

2. I have a special test case I've noticed that causes the I2C bus to lock up almost instantly. I have multiple threads, each writing to the I2C bus, one thread is spinning in a loop writing as fast as possible to some of my peripherals, to try and push everything as hard as possible. This works, but once I start up gstreamer and start reading from some USB H264 cameras I have connected, then the I2C bus fails. In dmesg I see the following:

[ 6466.437220] i2c_designware 808622C1:01: i2c_dw_handle_tx_abort: lost arbitration
[ 6467.468864] i2c_designware 808622C1:01: i2c_dw_handle_tx_abort: lost arbitration
[ 6468.496329] i2c_designware 808622C1:01: controller timed out
[ 6469.520296] i2c_designware 808622C1:01: controller timed out
[ 6470.544411] i2c_designware 808622C1:01: controller timed out
[ 6470.554822] xhci_hcd 0000:00:14.0: ERROR unknown event type 37
[ 6471.568423] i2c_designware 808622C1:01: controller timed out
[ 6472.592577] i2c_designware 808622C1:01: controller timed out
[ 6472.607633] xhci_hcd 0000:00:14.0: ERROR unknown event type 37 

In both cases, if I stop my test app, unplug my I2C peripherals and plug them back in again, the bus now works again, and I can restart the test again.

If you have any suggestions on a software technique to reset the I2C bus, and also how to deal with the USB+I2C issue, would be appreciated. Thanks!

waynepiekarski

Any ideas on this? Thanks

DCleri

There are many issues reported for that I2C drivers and unfortunately we don't have a solution as we are not the kernel maintainers for that driver.

Looking only at many other people having similar problems with that I2C driver, it looks like a lot of changes/fixes have been implemented between kernel 5.3 and 5.4.
At the moment we are testing internally Kernel 5.4, and it is planned to be released to support both Ubuntu 18.04 and 20.04 in the coming weeks.

waynepiekarski

Do you have any suggestions on where the bugs are being tracked for the I2C drivers? It would be good to review the current problems and see if there are any workarounds or discussion about what the cause could be. Thanks.

Also, do you have any suggestions on how to perform an I2C reset, sending out SDA and SCL pulses to reset all the devices on the bus that might be stuck?