Monday, March 30, 2015

Linkstation returns "Structure needs cleaning" error message to applications when it started after a power failure

Today, my linkstation was not shutdown cleanly due to power failure.
After bringing it up, it showed some weird behavior. Sometimes it hung and the transmission application showed error message "Structure needs cleaning" for some torrents which were already in the download list.

Anyway I was able to log in to the linkstation and investigate the /var/log/message. It showed that the issue was with the XFS files system of the sda6 partition. You can find out the mount point of the sda6 using mount command or cat /proc/mounts command.

root@LS-VL406:~# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,relatime,errors=continue,barrier=0,data=writeback 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
udev /dev tmpfs rw,nosuid,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=4,mode=620 0 0
/dev/ram1 /mnt/ram tmpfs rw,relatime,size=15360k 0 0
/dev/sda1 /boot ext3 rw,relatime,errors=continue,barrier=1,data=writeback 0 0
/dev/disk1_6 /mnt/disk1 xfs rw,noatime,attr2,usrquota,grpquota 0 0
/dev/disk1_6 /opt xfs rw,noatime,attr2,usrquota,grpquota 0 0

root@LS-VL406:~# ls -la /dev/disk1_6
lrwxrwxrwx    1 root     root            4 Mar 31  2015 /dev/disk1_6 -> sda6


I unmounted the /mnt/disk1 and executed xfs_repair as the crash report suggested.

Steps I followed to fix the issue,

1. Check whether the /mnt/disk1 is being used. kill or stop processes which are using the partition.

lsof | grep disk1

2. unmount the partition

umount /mnt/disk1

3. repair the partition

xfs_repair /dev/sda6

Then I rebooted the linkstation and it seems that the issue is resolved now.

Information which was available in the /var/log/messages file about the XFS corruption.

Mar 30 22:09:09 LS-VL406 kernel: Backtrace:                                                                                                                                               
Mar 30 22:09:09 LS-VL406 kernel: [<c001b79c>] (dump_backtrace+0x0/0x114) from [<c001b8e4>] (dump_stack+0x18/0x1c)                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel:  r6:c0481434 r5:c11dc800 r4:c01c870c                                                                                                                                
Mar 30 22:09:09 LS-VL406 kernel: [<c001b8cc>] (dump_stack+0x0/0x1c) from [<c018564c>] (xfs_error_report+0x54/0x64)                                                                                     
Mar 30 22:09:09 LS-VL406 kernel: [<c01855f8>] (xfs_error_report+0x0/0x64) from [<c01856b8>] (xfs_corruption_error+0x5c/0x78)                                                                         
Mar 30 22:09:09 LS-VL406 kernel: [<c018565c>] (xfs_corruption_error+0x0/0x78) from [<c01aed28>] (xfs_bmap_read_extents+0x4f8/0x580)                                                                    
Mar 30 22:09:09 LS-VL406 kernel:  r6:00000000 r5:00000000 r4:c22bf000                                                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel: [<c01ae830>] (xfs_bmap_read_extents+0x0/0x580) from [<c01c870c>] (xfs_iread_extents+0x78/0xe0)                                                                      
Mar 30 22:09:09 LS-VL406 kernel: [<c01c8694>] (xfs_iread_extents+0x0/0xe0) from [<c01a5dbc>] (xfs_bmapi_read+0xec/0x32c)                                                                             
Mar 30 22:09:09 LS-VL406 kernel:  r7:00000000 r6:c62b6240 r5:00000000 r4:c62b6270                                                                                                                    
Mar 30 22:09:09 LS-VL406 kernel: [<c01a5cd0>] (xfs_bmapi_read+0x0/0x32c) from [<c0181098>] (__xfs_get_blocks+0x180/0x524)                                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0180f18>] (__xfs_get_blocks+0x0/0x524) from [<c0181494>] (xfs_get_blocks+0x28/0x30)                                                                              
Mar 30 22:09:09 LS-VL406 kernel: [<c018146c>] (xfs_get_blocks+0x0/0x30) from [<c0109658>] (__block_write_begin+0x1e4/0x4d4)                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0109474>] (__block_write_begin+0x0/0x4d4) from [<c0109af8>] (block_write_begin+0x54/0x88)                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c0109aa4>] (block_write_begin+0x0/0x88) from [<c01816a0>] (xfs_vm_write_begin+0x48/0x7c)                                                                            
Mar 30 22:09:09 LS-VL406 kernel:  r6:18a07ebd r5:00000000 r4:18a07ebd                                                                                                                                
Mar 30 22:09:09 LS-VL406 kernel: [<c0181658>] (xfs_vm_write_begin+0x0/0x7c) from [<c00ac70c>] (generic_perform_write+0x198/0x20c)                                                                    
Mar 30 22:09:09 LS-VL406 kernel:  r8:cbd9e000 r7:00000000 r6:18a07ebd r5:00000143 r4:cbd9fd60                                                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c00ac574>] (generic_perform_write+0x0/0x20c) from [<c00ac7dc>] (generic_file_buffered_write+0x5c/0x90)                                                              
Mar 30 22:09:09 LS-VL406 kernel: [<c00ac780>] (generic_file_buffered_write+0x0/0x90) from [<c0186af0>] (xfs_file_buffered_aio_write+0xc0/0x150)                                                      
Mar 30 22:09:09 LS-VL406 kernel:  r9:cbd9e000 r8:c62b6240 r6:c62b63f8 r5:00000000 r4:18a07ebd                                                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0186a30>] (xfs_file_buffered_aio_write+0x0/0x150) from [<c0186f68>] (xfs_file_aio_write+0x174/0x1ec)                                                             
Mar 30 22:09:09 LS-VL406 kernel: [<c0186df4>] (xfs_file_aio_write+0x0/0x1ec) from [<c00e11a0>] (do_sync_write+0xa8/0xf0)                                                                             
Mar 30 22:09:09 LS-VL406 kernel: [<c00e10f8>] (do_sync_write+0x0/0xf0) from [<c00e1a88>] (vfs_write+0xbc/0x150)                                                                                      
Mar 30 22:09:09 LS-VL406 kernel:  r8:c000bca8 r7:cbd9ffa8 r6:007a81b0 r5:00010000 r4:c5c81f00                                                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c00e19cc>] (vfs_write+0x0/0x150) from [<c00e21dc>] (sys_pwrite64+0x80/0x94)                                                                                       
Mar 30 22:09:09 LS-VL406 kernel:  r8:c000bca8 r7:000000b5 r6:c5c81f00 r5:00010000 r4:007a81b0                                                                                                                      
Mar 30 22:09:09 LS-VL406 kernel: [<c00e215c>] (sys_pwrite64+0x0/0x94) from [<c000bb00>] (ret_fast_syscall+0x0/0x2c)                                                                                  
Mar 30 22:09:09 LS-VL406 kernel:  r6:00000000 r5:00000000 r4:18a07ebd                                                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel: XFS (sda6): Corruption detected. Unmount and run xfs_repair