[solved] UPboard Ubilinux Freezing

DavidDavid New Member Posts: 4
edited February 22 in Others

OS: ubilinux 4.9.45-ubilinux+

I have 100s of UPBoards deployed remotely i do not have physical access to them.
1. They do not have screens attached to them
2. they are running 24/7

I have had some issues of the devices freezing.
the only way to fix this issue is to hard reset the device (unplugging and plugging back in again)... this is a major problem for me as i cannot gain access to them myself and they should not be freezing like they do..

On problematic devices i have pulled the sys and kernel logs and there is nothing in between the timespan of the device being down.

The last line logged in the Sys log was:
Aug 9 00:17:01 ubilinux CRON[14157]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
and then a complete system crash...

Rough Hardware Info:
1. I am powering the device over the 5V GPIO pin with a UPS designed specifically for the UPBoard...
2. I also am utilizing 2 USB ports (1 for a custom BLE dongle, 2 for a Wifi dongle)
3. I have a Custom PCB attached getting powered by the UPS
4. The Custom PCB Utilizes the C7 10 pin connector to connect to a 3G mini-PCie Telit Cellular module

That is a rough description of the hardware.. and most of the time everything seems fine..
however we do seem to "randomly" get complete OS freezes and the only fix is to recycle the power..

Is there any logs / things you can think of that i can look at to get a better understanding of what the problem is.. or do you have a possible solution that will fix the issue at its core.

This is urgent, as i say i have 100's nearly 1000's of UPboards and they are "randomly" experiencing this issue..

Regards,
David Hutchinson

Comments

  • Filippo BelliniFilippo Bellini Emutex Posts: 10 admin
    edited August 2018

    Dear David,

    Thank you for your precise description of the issue. It looks similar to the situation experienced by another user. We're investigating the problem. I hope we will be able to get back to you soon.

    Kind regards

  • DavidDavid New Member Posts: 4

    @Filippo Bellini

    Have there been any further advancements with this issue? i would really like to know how to recreate this issue (i have tried multiple things) or best case scenario find a solution. I don't think the Upboard is running out of memory as this is logged in the sys logs. However i do know that i can recreate a similar issue, if i reboot the device multiple times consecutively it will freeze on boot. To recreate i just put reboot after 25 seconds into the crontab.

    I am not fully convinced it is the same issue as this happens on boot up and the other "Freeze" i am experiencing happens while running and after a "random" period of time.

    Regards,
    David Hutchinson

  • DavidDavid New Member Posts: 4

    I managed to get a kernel dump please see image

    This happened after rebooting the device multiple times in a row
    It was saying there was an error at drivers/gpu/drm/i915/intel_runtime_pm.c line 1059 vlv_power_well_enabled

    https://github.com/emutex/ubilinux-kernel/blob/upboard-4.9/drivers/gpu/drm/i915/intel_runtime_pm.c

    i had a look at the specified line of code. but i have limited knowledge of linux kernels and drivers so it didn't make much sense to me.
    Any help would be appreciated.

    Thanks,
    David Hutchinson

  • Jesse KaukonenJesse Kaukonen New Member Posts: 41 ✭✭

    There's an open i915 bug they've reported to Intel: https://bugs.freedesktop.org/show_bug.cgi?id=106721

    You'll have to compile to kernel without i915 to avoid this particular issue, I'm afraid, at least until there's a patched driver. To do this, comment out CONFIG_DRM_I915=m when building the kernel (use these instructions).

    https://github.com/emutex/ubilinux-kernel/blob/upboard-4.9/arch/x86/configs/upboard_defconfig#L2097

  • Dan O'DonovanDan O'Donovan Emutex Posts: 218 mod

    Hi David

    I think there may be 2 or 3 distinct issues described in this thread:

    1) The original post describes a problem with UP boards freezing at random, with apparently no clues of the cause in the kernel or system log files. Unfortunately, I think there is not enough information available to reproduce or further diagnose this issue. I am not aware of this specific issue on other UP boards, and do not have the same configuration of hardware peripherals and power supply that may be a factor in triggering this. My suggestion would be to try to eliminate the power supply and external peripherals as a factor, if possible, by running 2 UP boards side-by-side with the same application where one board has a standard power supply and no external peripherals while the other has the full hardware configuration described in the original issue. Also, it may be worth enabling persistent log storage for journald (if not already enabled) and using ‘journalctl –b-1’ after rebooting a frozen system to see any addition log information can be retrieved when the issue occurs.

    2) The i915 driver issue highlighted by Jesse Kaukonen has been seen to occur occasionally after multiple reboots during the OS start-up sequence. The workaround proposed by Jesse appears to be effective in eliminating this specific issue. To avoid rebuilding the kernel, an alternative approach to implementing this workaround may be to remove the kernel module from the filesystem (and run 'update-initramfs') to ensure it doesn't get loaded.

    3) As mentioned above in an earlier post, the UP board may freeze during reboots but this would not appear to be related to the issue reported in the original post, as it happens only when a reboot is triggered. One possible cause of that can be the i915 driver issue (2). However, we've identified a separate issue where the UP board can freeze at random while executing a reboot sequence, occurring at a very early stage during initial boot before the OS is loaded (and before the BIOS splash screen appears or the num-lock key is lit on the keyboard). This can be reproduced using only a BIOS EFI script to repeatedly trigger a reboot before the OS is loaded. We have reported this issue to AAEON for further investigation at BIOS level.

    Please let us know if you manage to further narrow down issue (1) above. In the meantime, we will let you know if you we hear of any further developments on issues (2) and (3) above.

    Best regards,
    Dan

  • Dan O'DonovanDan O'Donovan Emutex Posts: 218 mod

    Hi all

    In the context of increasing the reliability of reboots on the UP board, we also recommend the following measure:

    Please edit the file /etc/default/grub and ensure the following options are included in "GRUB_CMDLINE_LINUX_DEFAULT"

    GRUB_CMDLINE_LINUX_DEFAULT="reboot=efi,cold fsck.mode=force fsck.repair=yes"
    

    Then run update-grub from the command line to apply the changes.

    This does not address the issue raised in the original post, but we believe that this does help in reducing the probability of issues that may occur on the UP board during reboots.

    AAEON are actively investigating the issue (3) mentioned in my previous post, where the board may freeze immediately following a reboot.

    Best regards,
    -Dan

  • RansuRansu New Member Posts: 4
    edited September 2018

    Hi,

    I'm a new owner of a Up Board, the one that looks like a Raspberry Pi. I've noticed issues with rebooting so I came to this forum searching for similar reports. I have Arch Linux installed to the eMMC and it seems to run alright but more often then not it fails to reboot. When the board enters this state I have to fully remove power to get it to boot again.

    @Dan O'Donovan I'll try the above kernel command line settings to see if it helps any with reboots.

  • RansuRansu New Member Posts: 4

    @Dan O'Donovan I only need to add "reboot=efi,cold" right? I don't think I want to run fsck every single time.

  • Dan O'DonovanDan O'Donovan Emutex Posts: 218 mod

    We used only the 'force' mode option in our tests, but using the following fsck options may be sufficient for your needs:
    fsck.mode=auto fsck.repair=yes

  • RansuRansu New Member Posts: 4

    @Dan O'Donovan I like to report back that I've been using my up-board for some time now as a server and I experience no issues rebooting with the above options in my grub configuration.

  • ccaldeccalde Administrator, Moderator, Emutex Posts: 266 admin

    Hi @Ransu ,

    Thanks for reporting your results with the UP board rebooting.

  • RansuRansu New Member Posts: 4
    edited February 22

    I also added the note about it to the Arch Linux Wiki page for the UP Board https://wiki.archlinux.org/index.php/Up-board#Rebooting

Sign In or Register to comment.