LPC bus degradation issue -- symptoms if I don't update BIOS?

hungrycactus

Hi there!

I bought a a few Up2 boards last year, and they have been running solid with Centos running in command line/non-GUI for half a year. However a few months ago, one by one they started having symptoms of locking up randomly, every few days/weeks -- no video signal, ethernet lights blinking but not responding, blue power light on.

I just read that Bios v3.0 fixes the LPC bus degradation issue (https://forum.up-community.org/discussion/3900/intel-bug-for-alla-cpu-intel-cele4ron-n3350-j3355-j3455-e-pentium-n4200). My Up2's are only on BIOS v 2.1 (without the fix). Could this be related? Does the LPC bus degradation issue cause system crash/freeze?

I will surely install the BIOS patch now, but did I already cause permanent damage to the CPU/LPC/system?

Your insight is greatly appreciated.

-Ethan

DCleri

LPC bus is not used on UP Squared, but the new BIOS versions improves stability and comes with updated security patches.

I strongly suggest to update but I don't think the board is damaged in any way

hungrycactus

Thanks @DCleri --

If the LPC bus is not used on the Up2, then the lockups I am seeing must be caused by something else?

I still need to explain to my team lead why the Up2 boards seemed to have been fine for half a year, and out of the blue they started having symptoms of locking up randomly, every few days/weeks. The Centos 7 hasn't been touched for half a year...that's why I am worried there is some hardware degradation whether it is related to LPC bus or not. Do you have any other insight as to why I am seeing the same locking issues across three Up2's?

-Ethan

DCleri

Hi @hungrycactus

I don't believe it is a hardware degradation issue.

I don't know the setup, if indoor or outdoor, location, power supply, plus also the software after long time can expose issues not visible at first.

It is important that you collect logs and more information to help you debug the issue you are facing.

hungrycactus

@DCleri --

We have 4 sets of Up2's:

• Indoor in server rack cooled environment
• Power supply is the UP Squared power supply 5V@6A
• We have 4 USB devices plugged in (2x on the USB 3.0 ports, and 2x on the USB 2.0 breakout pin header cable). These are USB->serial cables, which I doubt use that much power.
• Running Centos 7.7 with kernel 3.10.0_1062

What logs can I start looking at?
-Ethan

DCleri

error logs and system logs that you can record to identify the issue when happening.

Also that Kernel version you are using is really old and Apollo Lake Platforms were not supported with that version, so you might experience unexpected behavior.

hungrycactus

@DCleri -- Thanks for the heads up on the old kernel. It is interesting as I look back when we updated the kernel, things seemingly got increasingly worse:

1) 3.10.0_862 on 10/15/18 - have not re-tested, but I don't recall noticing any crashes.
2) 3.10.0_957 on 8/7/19 - counted a few crashes
3) 3.10.0_1062 on 1/9/20 - many crashes

To note, all the above was on the old BIOS 2.1.

At this point, we still can't pin point the exact problem. Have you seen this behavior and does this make any sense to you? Regarding the kernels, I see 2 options:
1) roll back to the older 862 kernel, only on the basis that it seems to be more stable.
2) update to a newer 4. x or 5.x kernel? Hoping the newer ones will resolve the crashes.

Do you have any insights into why the older kernels seem to be more stable? Is there a kernel the Up2's (on 5.0 BIOS) that will run more stable on? I don't need the latest and greatest, just the most stable set up. What would you recommend?

Much thanks.
-Ethan

DCleri

It is impossible to provide an answer without testing each specific kernel and debugging it properly.

The kernel you mentioned are not supported at all by Apollo Lake as Intel submitted patches only in Kernel newer than 4.4
As it is an x86 CPU, it will still work, but it is not easy to understand or predict the behaviour.

We currenlty directly support kernel 4.15 for Ubuntu and 4.14 for Yocto, and soon moving to 5.0 and 4.19 respectively.
Please make sure you test the system with a compatible kernel so we can support you or you can try to use a mainline kernel which is more recent (> 4.4) and of course keep the BIOS to the latest available version.