input/output error in /dev/mmcblk0rpmb

m
m New Member Posts: 15
Dear all,

/dev/mmcblk0rpmb seems to contain bad blocks/sectors when checking with badblocks:
sudo badblocks -b 4096 -c 4096 -s /dev/mmcblk0rpmb
I ran this test because input/output errors are thrown when I use vgdisplay:
$ sudo vgdisplay
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 0: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4128768: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4186112: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4096: Input/output error
Any ideas why this happens? (This error occurs on two different up boards. I am not sure whether this is a hardware or software issue.)

Thanks to you!

Comments

  • DCleri
    DCleri Administrator, AAEON Posts: 1,213 admin
    Hi fb,
    Could you provide more info about the hardware and software?

    Board type (RAM, eMMC)
    BIOS version
    OS type and version
    Kernel type and version
  • m
    m New Member Posts: 15
    Hi Emutex,

    yes, of course. The same problem occurs on two boards which two absolut identicial setups (but possibly different BIOS versions).

    Up Board 4GB Ram, 64 GB eMMC
    BIOS: UP-CHT01 R0.S (UPC1BM05) (06/04/2016), Version 2.17.1249
    (the other BIOS version may be older but I cannot check at the moment because I cannot reboot the second board)
    Ubilinux 3.0 (current Version from your website; fresh install; no modifications apart from badblocks and lvm2 installed)
    kernel Linux 4.4.0-ubi3-amd64

    Should be straight forward to reproduce the error ...

    Thank you for your help!
    fb
  • m
    m New Member Posts: 15
    edited October 2016
    Now, my filesystem of /dev/mmcblk0p3 went to read only for one of the two boards. I executed fsck:
    $ fsck
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/mmcblk0p3: recovering journal
    /dev/mmcblk0p3 contains a file system with errors, check forced.
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    Free blocks count wrong (13921515, counted=11829383).
    Fix<y>? yes
    Free inodes count wrong (3720214, counted=3720210).
    Fix<y>? yes
    
    /dev/mmcblk0p3: ***** FILE SYSTEM WAS MODIFIED *****
    /dev/mmcblk0p3: ***** REBOOT LINUX *****
    /dev/mmcblk0p3: 97262/3817472 files (0.2% non-contiguous), 3427449/15256832 blocks
    

    I don't know whether these errors are related. Mh. Any idea why that happens?

    Best regards,
    fb
  • m
    m New Member Posts: 15
    I got a similar problem on the other up board: I have an external hard disk (powered with an external power adapter), connected via USB 3.0. According to kern.log, the USB device was just disconnected:
    [55512.825973] usb 2-1: USB disconnect, device number 2
    [55512.827124] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154140672 size 262144 starting block 30157568)
    [55512.827132] Buffer I/O error on device dm-0, logical block 30157568
    [55512.829975] scsi 0:0:0:0: rejecting I/O to offline device
    [55512.829980] scsi 0:0:0:0: [sda] killing request
    [55512.829987] scsi 0:0:0:0: rejecting I/O to offline device
    [55512.829990] scsi 0:0:0:0: [sda] killing request
    [55512.830010] scsi 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [55512.830015] scsi 0:0:0:0: [sda] CDB: Write(10) 2a 00 01 cc 12 00 00 00 1e 00
    [55512.830018] blk_update_request: I/O error, dev sda, sector 241209344
    [55512.830044] scsi 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [55512.830047] scsi 0:0:0:0: [sda] CDB: Write(10) 2a 00 01 cc 12 1e 00 00 02 00
    [55512.830050] blk_update_request: I/O error, dev sda, sector 241209584
    [55512.830057] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 134086656 size 131072 starting block 30150656)
    [55512.830060] Buffer I/O error on device dm-0, logical block 30150656
    [55512.830066] Buffer I/O error on device dm-0, logical block 30150657
    [55512.830068] Buffer I/O error on device dm-0, logical block 30150658
    [55512.830070] Buffer I/O error on device dm-0, logical block 30150659
    [55512.830073] Buffer I/O error on device dm-0, logical block 30150660
    [55512.830075] Buffer I/O error on device dm-0, logical block 30150661
    [55512.830077] Buffer I/O error on device dm-0, logical block 30150662
    [55512.830078] Buffer I/O error on device dm-0, logical block 30150663
    [55512.830080] Buffer I/O error on device dm-0, logical block 30150664
    [55512.877976] sdhci: Timeout waiting for Buffer Read Ready interrupt during tuning procedure, falling back to fixed sampling clock
    [55512.925686] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154533888 size 262144 starting block 30157664)
    [55512.925863] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154927104 size 262144 starting block 30157760)
    [55512.926443] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 155320320 size 262144 starting block 30157856)
    [55512.926795] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 156106752 size 786432 starting block 30158048)
    [55512.926965] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 157024256 size 131072 starting block 30158272)
    [55512.927204] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 157155328 size 655360 starting block 30158304)
    [55512.927460] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 158072832 size 393216 starting block 30158528)
    [55512.927613] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 158597120 size 262144 starting block 30158656)
    [55512.938104] scsi 0:0:0:0: rejecting I/O to dead device
    [55512.943944] scsi 0:0:0:0: rejecting I/O to dead device
    [55512.949795] scsi 0:0:0:0: rejecting I/O to dead device
    [55512.955604] scsi 0:0:0:0: rejecting I/O to dead device
    
    I had the same problem even around a month ago (with another hard disk) ... Somehow, these errors look really alike ...

    Best regards,
    fb
  • m
    m New Member Posts: 15
    Hi Emutex,
    the issue still exists. I have bought another external harddisk (USB 3.0; externally powered) and only installed ubilinux and a afp server to backup some files on the up board. I installed, formatted the external harddisk, and backed up the data several times now, and every time, the file system gets corrupted after some days. By disabling the automatic-power-off features in ubilinux, the filesystem of the extern harddisk only gets corrupted sometimes, but the file system of the eMMC gets corrupted much more often. When I then execute a "touch x" in my home directory, I get the message "touch: cannot touch ‘x’: Read-only file system".
    This is more than frustating. I will now use my second up board for some tests. It would be great if you could tell me either which tests to do, or how to proceed in general. At least one of the board is useless at this point in time. ;( I hope that corrupting the file system is not a "feature" of the up board or ubilinux *haha* ;)
    Thank you for your much appreciated help,
    fb
  • m
    m New Member Posts: 15
    Hi dcleri!
    Do you have any news on that topic? If not, I would contact the official support, because the problem still exists. Maybe they know.
    Best greetings,
    fb
  • DCleri
    DCleri Administrator, AAEON Posts: 1,213 admin
    Hi fb,

    I am sorry for not replying you earlier.
    We have investigated the issue on linux and we will be releasing an updated Linux Kernel for ubilinux and Ubuntu which will include a fix for the issue.

    In our tests the timeout is not happening anymore and so the eMMC I/O errors.

    We are still investigating if the issue may affects other OS and, eventually if it can be solved at BIOS level.
  • m
    m New Member Posts: 15
    Hi dcleri!
    You are writing, that you have found the bug? That sounds like good news! Thank you very much!
    Are you able to only solve the bug for /dev/mmcblk0rpmb, or also for the other partitions?
    I am really looking forward to be able to use the board!
    Best greeting,
    fb
  • m
    m New Member Posts: 15
    Hi dcleri!

    When do you plan to release the updated linux kernel? Thank you!!

    Best greetings,
    fb
  • Jaap Crezee
    Jaap Crezee New Member Posts: 19
    Hi Fb,


    If you are not interested into the 40 pin IO of the UP board you can use a vanilla kernel; on my system which is running 4.8.5 the problem is not present:

    # badblocks -b 4096 -c 4096 -s /dev/mmcblk0rpmb | tail -n 5
    Checking for bad blocks (read-only test): done
    1019
    1020
    1021
    1022
    1023
    # echo $?
    0
    # uname -r
    4.8.5+
    #


    I probably have TPM disabled in the bios and have not enabled that in the kernel; could be related see: https://lwn.net/Articles/694798/

    regards,


    Jaap
  • m
    m New Member Posts: 15
    Hi Jaap,
    I may somewhen use the GPIOs. I expect the board to reliable run with ubilinux as a os.

    Hi dcleri,
    it would be nice if you could provide some more information on when to expect my up board be running. I have physical access to the board until Saturday, but then have no physical access to the board for at least for some months. The board must soon be running reliable, and at the moment I have to reinstall the os/sync data every two days because one of the file systems gets corrupted. I cannot trust a filesystem that corrupts regularly. It thus would be great if you could provide the fix. (I am running linux-image-4.4.0-ubi3-amd64 which seems to be the newest kernel according to
    apt-cache search linux-image
    
    .)

    Thank you for your understanding,
    fb