input/output error in /dev/mmcblk0rpmb

m

Dear all,

/dev/mmcblk0rpmb seems to contain bad blocks/sectors when checking with badblocks:

sudo badblocks -b 4096 -c 4096 -s /dev/mmcblk0rpmb

I ran this test because input/output errors are thrown when I use vgdisplay:

$ sudo vgdisplay
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 0: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4128768: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4186112: Input/output error
  /dev/mmcblk0rpmb: read failed after 0 of 4096 at 4096: Input/output error

Any ideas why this happens? (This error occurs on two different up boards. I am not sure whether this is a hardware or software issue.)

Thanks to you!

DCleri

Hi fb,
Could you provide more info about the hardware and software?

Board type (RAM, eMMC)
BIOS version
OS type and version
Kernel type and version

m

Hi Emutex,

yes, of course. The same problem occurs on two boards which two absolut identicial setups (but possibly different BIOS versions).

Up Board 4GB Ram, 64 GB eMMC
BIOS: UP-CHT01 R0.S (UPC1BM05) (06/04/2016), Version 2.17.1249
(the other BIOS version may be older but I cannot check at the moment because I cannot reboot the second board)
Ubilinux 3.0 (current Version from your website; fresh install; no modifications apart from badblocks and lvm2 installed)
kernel Linux 4.4.0-ubi3-amd64

Should be straight forward to reproduce the error ...

Thank you for your help!
fb

m

Now, my filesystem of /dev/mmcblk0p3 went to read only for one of the two boards. I executed fsck:

$ fsck
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
/dev/mmcblk0p3: recovering journal
/dev/mmcblk0p3 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (13921515, counted=11829383).
Fix<y>? yes
Free inodes count wrong (3720214, counted=3720210).
Fix<y>? yes

/dev/mmcblk0p3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mmcblk0p3: ***** REBOOT LINUX *****
/dev/mmcblk0p3: 97262/3817472 files (0.2% non-contiguous), 3427449/15256832 blocks

I don't know whether these errors are related. Mh. Any idea why that happens?

Best regards,
fb

m

I got a similar problem on the other up board: I have an external hard disk (powered with an external power adapter), connected via USB 3.0. According to kern.log, the USB device was just disconnected:

[55512.825973] usb 2-1: USB disconnect, device number 2
[55512.827124] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154140672 size 262144 starting block 30157568)
[55512.827132] Buffer I/O error on device dm-0, logical block 30157568
[55512.829975] scsi 0:0:0:0: rejecting I/O to offline device
[55512.829980] scsi 0:0:0:0: [sda] killing request
[55512.829987] scsi 0:0:0:0: rejecting I/O to offline device
[55512.829990] scsi 0:0:0:0: [sda] killing request
[55512.830010] scsi 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[55512.830015] scsi 0:0:0:0: [sda] CDB: Write(10) 2a 00 01 cc 12 00 00 00 1e 00
[55512.830018] blk_update_request: I/O error, dev sda, sector 241209344
[55512.830044] scsi 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[55512.830047] scsi 0:0:0:0: [sda] CDB: Write(10) 2a 00 01 cc 12 1e 00 00 02 00
[55512.830050] blk_update_request: I/O error, dev sda, sector 241209584
[55512.830057] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 134086656 size 131072 starting block 30150656)
[55512.830060] Buffer I/O error on device dm-0, logical block 30150656
[55512.830066] Buffer I/O error on device dm-0, logical block 30150657
[55512.830068] Buffer I/O error on device dm-0, logical block 30150658
[55512.830070] Buffer I/O error on device dm-0, logical block 30150659
[55512.830073] Buffer I/O error on device dm-0, logical block 30150660
[55512.830075] Buffer I/O error on device dm-0, logical block 30150661
[55512.830077] Buffer I/O error on device dm-0, logical block 30150662
[55512.830078] Buffer I/O error on device dm-0, logical block 30150663
[55512.830080] Buffer I/O error on device dm-0, logical block 30150664
[55512.877976] sdhci: Timeout waiting for Buffer Read Ready interrupt during tuning procedure, falling back to fixed sampling clock
[55512.925686] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154533888 size 262144 starting block 30157664)
[55512.925863] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 154927104 size 262144 starting block 30157760)
[55512.926443] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 155320320 size 262144 starting block 30157856)
[55512.926795] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 156106752 size 786432 starting block 30158048)
[55512.926965] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 157024256 size 131072 starting block 30158272)
[55512.927204] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 157155328 size 655360 starting block 30158304)
[55512.927460] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 158072832 size 393216 starting block 30158528)
[55512.927613] EXT4-fs warning (device dm-0): ext4_end_bio:329: I/O error -5 writing to inode 4723963 (offset 158597120 size 262144 starting block 30158656)
[55512.938104] scsi 0:0:0:0: rejecting I/O to dead device
[55512.943944] scsi 0:0:0:0: rejecting I/O to dead device
[55512.949795] scsi 0:0:0:0: rejecting I/O to dead device
[55512.955604] scsi 0:0:0:0: rejecting I/O to dead device

I had the same problem even around a month ago (with another hard disk) ... Somehow, these errors look really alike ...

Best regards,
fb

m

Hi Emutex,
the issue still exists. I have bought another external harddisk (USB 3.0; externally powered) and only installed ubilinux and a afp server to backup some files on the up board. I installed, formatted the external harddisk, and backed up the data several times now, and every time, the file system gets corrupted after some days. By disabling the automatic-power-off features in ubilinux, the filesystem of the extern harddisk only gets corrupted sometimes, but the file system of the eMMC gets corrupted much more often. When I then execute a "touch x" in my home directory, I get the message "touch: cannot touch ‘x’: Read-only file system".
This is more than frustating. I will now use my second up board for some tests. It would be great if you could tell me either which tests to do, or how to proceed in general. At least one of the board is useless at this point in time. ;( I hope that corrupting the file system is not a "feature" of the up board or ubilinux *haha*
Thank you for your much appreciated help,
fb

m

Hi dcleri!
Do you have any news on that topic? If not, I would contact the official support, because the problem still exists. Maybe they know.
Best greetings,
fb

DCleri

Hi fb,

I am sorry for not replying you earlier.
We have investigated the issue on linux and we will be releasing an updated Linux Kernel for ubilinux and Ubuntu which will include a fix for the issue.

In our tests the timeout is not happening anymore and so the eMMC I/O errors.

We are still investigating if the issue may affects other OS and, eventually if it can be solved at BIOS level.

m

Hi dcleri!
You are writing, that you have found the bug? That sounds like good news! Thank you very much!
Are you able to only solve the bug for /dev/mmcblk0rpmb, or also for the other partitions?
I am really looking forward to be able to use the board!
Best greeting,
fb

m

Hi dcleri!

When do you plan to release the updated linux kernel? Thank you!!

Best greetings,
fb

Jaap Crezee

Hi Fb,


If you are not interested into the 40 pin IO of the UP board you can use a vanilla kernel; on my system which is running 4.8.5 the problem is not present:

# badblocks -b 4096 -c 4096 -s /dev/mmcblk0rpmb | tail -n 5
Checking for bad blocks (read-only test): done
1019
1020
1021
1022
1023
# echo $?
0
# uname -r
4.8.5+
#


I probably have TPM disabled in the bios and have not enabled that in the kernel; could be related see: https://lwn.net/Articles/694798/

regards,


Jaap

m

Hi Jaap,
I may somewhen use the GPIOs. I expect the board to reliable run with ubilinux as a os.

Hi dcleri,
it would be nice if you could provide some more information on when to expect my up board be running. I have physical access to the board until Saturday, but then have no physical access to the board for at least for some months. The board must soon be running reliable, and at the moment I have to reinstall the os/sync data every two days because one of the file systems gets corrupted. I cannot trust a filesystem that corrupts regularly. It thus would be great if you could provide the fix. (I am running linux-image-4.4.0-ubi3-amd64 which seems to be the newest kernel according to

apt-cache search linux-image

.)

Thank you for your understanding,
fb