UP Squared sporadically booting into the BIOS only

Adnan

Hello,
can you possibly help with the following problem one of our customers experiences with an UP Squared board, P/N: UPS-GWS01P4-A20-0864?

Here is the customer's problem description:

There is a Linux system installed, which did not cause any problems in the beginning. This system is installed in our measuring instruments.

Currently we have the following problem:
We use the device with the Linux system installed and switch it off after the measuring process is finished.
At the next boot the UP-Squared starts directly into the BIOS and not into our Linux system as usually.
However, this does not happen every time, but sporadically.
The measuring device was not moved at the time when the error started to turn up.
To solve the problem, we have to restart from the BIOS and then start "normally" into the Linux system.

Enclosed Photo1 of the "faulty" start directly into the BIOS.

And Photo2 after the restart and afterwards manual procedure into the BIOS.

According to that this sounds like the issue turns up on cold boots only and only sporadically. According to the provided pictures the boot order is automatically changed and the USB boot device is taken out from the first position in the boot order.
Could this be a BIOS problem? The picture shows BIOS 2.18.1263, but strangely the issue turned up only after using the board for a while…

I'm grateful for any hints
Thank you.
Adnan

snuhg

Take this with a grain of salt, but to me your description reminds me of a problem other users have reported on this forum. These reports vary in some of the details but the common symptom is always the same. Namely, at boot-up the system is unable to detect/initialize the eMMC storage device. This results in the system not finding the files needed to complete the boot process and by default the system resorts to opening the BIOS pages.

When looking at your first picture (where the system failed to boot) you can see that there is no additional info for {Hard Disk} in the list of boot device options. That means the system was unable to initialize/detect a hard disk (i.e. the eMMC on the UP2 board). When such a failure occurs there are also error messages during boot-up to the effect off something like "error xyz, unable to initialize eMMC". I am not sure if these messages are included in the system logs (probably not). In contrast, the second picture (where the boot-up was successful) shows a hard disk entry in the list of boot devices which also contains info about the OS (debian) and the type of storage device (M...., which would probably read MMC if there were more space on the screen to show the complete list entry).

Why does your system fail to detect the hard disk and why does it occur randomly? Only AAEON knows the complete answer, but discussions on this forum seem to suggest bad electrical contacts, i.e. poor soldering, as the underlying problem. The resulting effect will vary from board to board, depending on quality control in the factory and how good or bad an individual soldering contact on a particular board is. In the worst case, initialization of the MMC will always fail, so it is not even possible to install an OS in the first place. In less severe cases it is possible to install an OS, but the boot process might randomly fail at later times. I am not an expert on soldering and/or PCBs, but I can imagine that a bad contact might get worse with time and/or show different behavior depending on ambient conditions (temperature, ....) and God knows what else.

Mind you, there are also reports on this forum concerning an issue that may be characterized as "random reboots". That's a different case from yours because these boards were randomly rebooting, i.e. after boot-up they would run for a while and then suddenly reboot. The cause was found to be a software bug, I believe, and I think it was an issue with the original UP-board only and not the UP2.

ApeiChen

Hi

May I know if you update the BIOS to the latest version the problem still occur or not?
Thanks.

Antoine

@ApeiChen said:
Hi

May I know if you update the BIOS to the latest version the problem still occur or not?
Thanks.

Yes it does

Our BIOS info : UP-APL01 R4.2 (UPA1AM42 ) (03/12/2019)

@Adnan
I don't think that your BIOS version is 2.18.1263, I believe this is just the Aptio Setup Utility version.
The BIOS version is showed in the "Main" tab.

Adnan

Hm, ok. Our customer returned only now stating that they could update several systems, but on some machines they still get the error shown here. He seems to try to update the file "UPA1AM40.BIN" if I see this correctly (please see the attached "Index.png").

I'm trying to get hold of the current BIOS versions on the boards that can't be updated and the version he tries to update to. Maybe there is a clue in there...

Adnan

Hello, I have some more details from our customer now:

• Current BIOS version is v2.1
• He is trying to update to version v4.0
• The customer already updated 6 systems from v2.1 to v4.0 successfully without a problem. So generally this should work...
• For the updates always the tool from the package "UPA1AM40.zip" from https://downloads.up-community.org/download/up-squared-uefi-bios-v4-0/ was used.
• The customer already did try to reset the BIOS to Default values without success.

The customer provided the following picture (please see attached) with the comments:

• Here you see a picture of the system after I started go.nsh.
• What about the error message (red arrow)?
• Some seconds after I started go.nsh the system reboots with 2.1 again..

DCleri

Old boards (A10) can have BIOS R2.1 and it is ok to upgrade.
The UPS-GWS01P4-A20-0864 boards come already with a BIOS R4.2 or newer so the customer is actually downgrading it, which it shouldn't be done.

It's important to understand the starting point otherwise it is not possible to debug the issue.

What Linux system you have installed?
Debian has not been tested, so I would suggest to setup and use a supported distribution, e.g. Ubuntu: https://wiki.up-community.org/Ubuntu
We haven't experiencing such issues with Ubuntu (or Yocto), but I cannot confirm that for other distributions as it may depends on the way grub is configured.

ease_Dieter

I am an employee of the company our distributor (Adnan) has written for. Sorry for the late answer but I was sick.
A short summary:
Most of the A20-boards at our customers work without problems. But we have 2 boards which run fine for a long time, but once they boot directly to the Bios when we start them. Look at the first 2 pictures on this site. There you can see, that the FIXED BOOT Order has been changed! (Sorry, but I don’t think that our Debian Linux has changed this order.) After we have resetet the Bios to their Default Settings, everthing was ok.
Then we have asked you: what was the problem? You said: please use the actual Bios version and tell me if the problem persist. Then we have done it already now with 7 boards and we not have had this problem again with the faulty board. I don’t know whether the boot problem has been solved for all the time, because it works without problems before the boot problem for a long time, too.
The actual problem is to update the second A20-board with the boot problem to Bios version 4.0.

To your last answer (thank you for it):
We only have A20 boards. All of our A20-boards have had Bios R2.1 and it was possible to update their Bios to 4.0 until this one A20 board. We have no A20 boards with Bios R4.2.
Please look at the last picture of Oct, 23. What about the error message (the red arrow)? Is it helpful for you?

Yes, we use Debian but independent of the operating system the Bios update should work!

But one warning has been displayed when I start the flashing of the device, please look at the added picture. But this warning has been displayed at each of the 7 updates and every of these updates work fine but not with this device.
What kind of information do you need of the starting point? Please let me know.

eas

Hi @ease_Dieter
Thanks for the update!
We have had the same issue for several of our boards and are now having issues with a system at one of our customers which is really inaccessible and connected via a 4G modem.

Do I understand it correctly if you have updated the BIOS on the two boards that malfunctioned and they have been working since then? That would be around a month since the system upgrade?

I think we have 5 boards and have had problems with 3 of them so far. We've started to upgrade the BIOS but it would be really appreciated if someone from AAEON or INTEL could confirm if BIOS upgrade fix the issue. I see there are quite a few posts on this forum describing similar issues, so I would think that they have looked into the issue? @DCleri @ApeiChen

ease_Dieter

We have updated 7 boards up to now. 2 of them are working for about 2 month I think. 4 boards a shorter time. One board is not updatable.

eas

Thanks for the feedback @ease_Dieter

DCleri

In order to provide support for the specific issue, I would need to have the exact same setup and a procedure to reproduce it consistently

As explained above the issue has not been experienced before with the supported distributions.
Unfortunately we cannot test any variations but we can review and provide feedback for you to try if it fixes the issue.

Upgrading the BIOS to the latest version (current available version is R4.6) might help to solve the issue and it seems it did for most of the boards @ease_Dieter

When you update the BIOS, the system use a UEFI Shell and not the operating system to upgrade, the warning is ok and the upgrade just works.

Let me know if you have any other questions

ease_Dieter

Here the Note for the Bios 4.6 update (https://downloads.up-community.org/download/up-squared-uefi-bios-v4-6/):
Please make sure your UP Squared has BIOS 4.0 currently installed before proceeding.

But it is not possible to update first to Bios 4.0!!

ease_Dieter

Any further help to update the board to BIOS 4.0 and then to 4.6???

DCleri

Hi @ease_Dieter

What do you mean that you cannot update to BIOS 4.0?

What BIOS version do you currently have on your UP Squared?

ease_Dieter

Now I have the faulty board back from our customer in USA. First I have had the same problem to update to 4.0. But then I have seen that I can use fs1: instead of blk1:. blk1: has been displayed as the Removable Harddisk. I have done all the updates of the other boards with blk1:. But only this board have made problems. See above.
With fs1: the update has made no problems. Then I have made directly the update to 5.0, too.

My last questions now:
In the history file of 4.9 you wrote:
Delete non-BIOS created boot option to fix issue that boot option disabled suddenly
Can you give me more informations about this? It sounds like our initial problem, that the Bios forgot were to boot from - look at the first post on this side.

ease_Dieter

Any answer for my last question?

DCleri

there was an issue on the eMMC where sometimes it could't be found from a cold boot.
After that the BIOS was loosing the boot configuration and the system wouldn't boot anymore.

In order to fix it, the BIOS was updated to never disable eMMC boot.

ease_Dieter

That was exactly our problem. I have no further questions. All our problems are solved now. Thank you for your help.