Losing bootable OS, possibly caused by BIOS battery failure

aplummer

For a project I'm working on, I've installed Ubilinux 3.0 on eight UP boards so far. After between about 2 weeks and about 4 months of use, three of them have stopped booting their operating system. Once I get to the EFI shell, I can find and boot manually by running:
fs0: then EFI\ubilinux\grubx64.efi,
and the operating system comes back. Running:
grub-install; update-grub
fixes the booting problem and the operating system boots automatically.

But I need for this to not happen in the first place, as once a unit fails to boot, without physical access to it I can't rescue it. My belief is that the cause is a faulty or intermittent BIOS battery - the CR2032 stuck to the top of the network socket - as a) this is something that could conceivably degrade over time and b) if I unplug it and replug it, I can induce the issue I described above.

My current best guess at a permanent fix for this is to run:
cp /boot/efi/EFI/ubilinux/grubx64.efi /boot/efi/EFI/boot/bootx64.efi
as part of the installation of the operating system - as suggested in https://forum.up-community.org/discussion/935/up-board-without-rtc-battery-doesnt-boot-from-emmc as this then means that the operating system will still boot even if the battery fails.

But - if my guess about the battery being the cause is right - the number of batteries that have failed is far higher than I'd expect for this kind of battery. And given that the battery failing means it will no longer boot, that seems like an big problem, and one that I would like to have seen UP officially address.

To boil this down into some questions:

• is faulty or intermittent BIOS batteries a known issue? And have UP addressed it officially at all?
• are the batteries replaceable? They're wired into their little plastic jackets - is it possible to open and securely close these to use a new battery?
• is copying the bootloader to EFI's default location the right thing to do here? And if so, why isn't it done by ubilinux 3.0? Is it done by ubilinux 4.0?
• is there a way of forcing the UP to boot into the EFI shell? Currently, I have to insert the USB stick I used for flashing the BIOS to get to the shell.

Javier Arteaga

Hi aplummer,

Yes - it's a known issue on the original UP board that EFI boot entries may be lost (in my experience, after power cuts).

@aplummer said:
But I need for this to not happen in the first place, as once a unit fails to boot, without physical access to it I can't rescue it. My belief is that the cause is a faulty or intermittent BIOS battery - the CR2032 stuck to the top of the network socket - as a) this is something that could conceivably degrade over time and b) if I unplug it and replug it, I can induce the issue I described above.

You're right that the issue can be induced in this way, but a faulty battery isn't the only possible cause of issues with EFI variable storage. The underlying issue may be in firmware or software too.

My current best guess at a permanent fix for this is to run:
cp /boot/efi/EFI/ubilinux/grubx64.efi /boot/efi/EFI/boot/bootx64.efi
as part of the installation of the operating system - as suggested in https://forum.up-community.org/discussion/935/up-board-without-rtc-battery-doesnt-boot-from-emmc as this then means that the operating system will still boot even if the battery fails.

Correct. (By the way, GRUB can also take care of this with grub-install --removable.)

• is faulty or intermittent BIOS batteries a known issue? And have UP addressed it officially at all?

Not as far as I know. I'd defer to the hardware team as to whether the root cause is known.

• are the batteries replaceable? They're wired into their little plastic jackets - is it possible to open and securely close these to use a new battery?

If you need to replace batteries (again, not necessarily the issue), I'd look into sourcing a battery with the connector already soldered to save the trouble.

• is copying the bootloader to EFI's default location the right thing to do here? And if so, why isn't it done by ubilinux 3.0? Is it done by ubilinux 4.0?

It's a valid permanent workaround in software, yes. The ubilinux 4.0 installer should do this.

• is there a way of forcing the UP to boot into the EFI shell? Currently, I have to insert the USB stick I used for flashing the BIOS to get to the shell.

After a certain point, UP BIOS stopped bundling a built-in EFI shell to make space for feature ROMs like PXE, so it's just not always available.

Regards