Disable automatic package metadata update in Fedora

Sometimes my internet speed becomes slow when I use Fedora after a system reboot. Once this happens, I experienced the effect even though all the network related applications are closed. My internet connection is not a super fast, so it annoyed me a lot. I wanted to find out the reason behind this issue.

Step 1: use netstat to identify the PID/applications which have opened TCP connections,

[root@desktop9 ~]# netstat -anp -t
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.0.1:53          0.0.0.0:*               LISTEN      1252/dnsmasq        
tcp       13      0 192.168.1.9:48208       192.168.1.50:445        CLOSE_WAIT  6776/gvfsd-smb      
tcp        0      0 192.168.1.9:44632       137.189.4.14:80         ESTABLISHED 10934/python        
tcp        0      0 192.168.1.9:44635       137.189.4.14:80         CLOSE_WAIT  10934/python        
tcp        0      0 192.168.1.9:44690       137.189.4.14:80         CLOSE_WAIT  10934/python        
tcp       13      0 192.168.1.9:36304       192.168.1.50:139        CLOSE_WAIT  6769/gvfsd-smb-brow

The output shows that the process 10934/python has a established connection with destination 137.189.4.14:80.

Step 2: Use ps command to find out more information of this process

[root@desktop9 ~]# ps -efw | grep 10934
root     10934     1  0 18:56 ?        00:00:00 /usr/bin/python -OO /usr/bin/dnf -v makecache timer
root     11204 11153  0 19:02 pts/0    00:00:00 grep --color=auto 10934

The output shows the culprit. it was "/usr/bin/dnf -v makecache timer"

The man page of dnf describes makecache downloads and caches in binary format metadata for all known repos available in the system.

Man page output:

       dnf [options] makecache
              Downloads and caches in binary format metadata for all known repos. Tries to avoid downloading whenever possible (e.g. when the local  metadata
              hasn't expired yet or when the metadata timestamp hasn't changed).

       dnf [options] makecache timer
              Like  plain  makecache but instructs DNF to be more resource-aware, meaning will not do anything if running on battery power and will terminate
              immediately if it's too soon after the last successful makecache run (see dnf.conf(8), metadata_timer_sync).


It seems that the dnf makecache is scheduled to run after every reboot. I decided to disable this and do manual metadata updates whenever it's required.

Command to disable dnf-makecache.service:

systemctl disable dnf-makecache.service 

Update 1: Disable the dnf-makecache.timer as well. Otherwise the service will be restarted.

systemctl disable dnf-makecache.timer

Update 2: I started using Fedora 29 Server edition and noted packagekit is also involved in this.
Therefore I disabled them as well,

systemctl disable packagekit.service
systemctl disable packagekit-offline-update.service

TestDisk, The best data recovery tool I have ever used

One of my friends had a problem with his portable hard disk (WD10TMVW)  which appeared to be an issue with a corrupted partition table. It was not detected by Windows. Fortunately it was detected by Fedora, but it was not auto mounted. It was appeared only in /dev directory.

I tried many data recovery tools to retrieve the data, but all the tools failed, many of them even failed to recognize the disk. However "TestDisk", in fact photo_rec was able to detect the disk and recover few files. after recovering few files, it went to an endless loop. I believe this  was happened due to some bad sectors.

So I went though their forum and found out that editing a status file (i.e. photorec.ses ) you can start recovery from a given sector. this status file is used by the tool to resume previously stopped recovery session.

The second line of this file keeps the last processed sector.
If you encounter the endless loop, you can change the highlighted value and restart the recovery. You must choose Y when the tool asks to continue previous session.

2nd line of photorec.ses generated by photo_rec:
/dev/sdc partition_none,255,blocksize,512,fileopt,fat,disable,ext,disable,options,paranoid,keep_corrupted_file_no,wholespace,search,status=ext2_off,185497,inter

Whenever you encounter the endless loop, you have to do this modification and restart the recovery.
So, eventually I was able to recover most of the data from the disk, thank to this great tool.

Home page of TestDisk: http://www.cgsecurity.org/wiki/TestDisk

Download video lectures from Coursera

Today I came a cross Coursera website and found some great video tutorials on computer science. After going through some of the available courses, I decided to download all the files belongs to a course. However It was annoying to click on each link to download video, power point presentation, PDF document and etc. So I did a google search and found out that there are plugins for common browsers and download managers which can be used to download all the files in one go. Anyway instead of using the already available options, I decided to write my own script to download the files as I had time to kill. Writing your own software for already available solutions may not make much sense. But it will help you to learn some thing new.

I tested the script "High Performance Scientific Computing" and "The Hardware/Software Interface" pages and worked perfectly.

To download the files using the script, you want two local files.

1. Cookies information after you logged in to the Coursea site. This information can be extracted using a browser plugin. I used a plugin called "cookie.txt export". Save this information to a file called "cookies.txt".
2. Course page (html) where the links you want to get downloaded listed.

You can download the script from here,
Link: https://sites.google.com/site/blogswapoff/coursera_downloader.tar.gz?attredirects=0&d=1

Copy the script where the html page and the cookies.txt are copied. Then execute the script.
You need to provide the file name of the html page as the first argument.

E.g.: nohup ./coursera_downloader.pl Coursera_hpsc.html &

As usual, the script was executed on my Buffalo Linkstation NAS. Use nohup when you execute the script so that you can push the script to background and close the terminal if required.


root@LS-VL406:/mnt/disk1/share/downloads/courseea_files# ls -ltr
-rw-rw-r--    1 root     root         1731 Apr 26 09:37 cookies.txt
-rw-r-----    1 root     root       322767 Apr 26 16:29 Coursera_hpsc.html
-rwxr-xr-x    1 root     root         1785 Apr 26 17:43 coursera_downloader.pl
drwxr-xr-x    2 root     root           97 Apr 26 19:26 High_Performance_Scientific_Computing_Course_Overview
drwxr-xr-x    2 root     root         4096 Apr 26 19:29 Week_1_-_Introduction_to_the_Course_amp_Lecture_1
drwxr-xr-x    2 root     root         4096 Apr 26 19:33 Week_1_-_Lecture_2_Binary_storage_and_version_control
drwxr-xr-x    2 root     root         4096 Apr 26 19:36 Week_1_-_Lecture_3_Demo_of_Python_and_git
drwxr-xr-x    2 root     root         4096 Apr 26 19:40 Week_2_-_Lecture_4_Continued_demo_of_Python_and_git
drwxr-xr-x    2 root     root         4096 Apr 26 19:43 Week_2_-_Lecture_5_Python_language_features
drwxr-xr-x    2 root     root         4096 Apr 26 19:47 Week_2_-_Lecture_6_NumPy_and_timing_Python_code
drwxr-xr-x    2 root     root         4096 Apr 26 19:50 Week_2_-_Cloud_Computing_Amazon_Web_Service_Supplemental_Information
drwxr-xr-x    2 root     root         4096 Apr 26 19:53 Week_3_-_Lecture_7_Python_debugging_and_Fortran_intro
drwxr-xr-x    2 root     root         4096 Apr 26 19:56 Week_3_-_Lecture_8_Fortran_subroutines_functions_and_arrays
drwxr-xr-x    2 root     root         4096 Apr 26 19:59 Week_3_-_Lecture_9_Multi-file_codes_and_Makefiles
drwxr-xr-x    2 root     root         4096 Apr 26 20:02 Week_4_-_Lecture_10_Fortran_modules_and_Newtons_method
drwxr-xr-x    2 root     root         4096 Apr 26 20:06 Week_4_-_Lecture_11_Computer_architecture_and_memory_optimization
drwxr-xr-x    2 root     root         4096 Apr 26 20:09 Week_4_-_Lecture_12
drwxr-xr-x    2 root     root         4096 Apr 26 20:12 Week_5_-_Lecture_13
drwxr-xr-x    2 root     root         4096 Apr 26 20:16 Week_5_-_Lecture_14
drwxr-xr-x    2 root     root         4096 Apr 26 20:17 Week_5_-_Lecture_15
drwxr-xr-x    2 root     root         4096 Apr 26 20:20 Week_6_-_Lecture_16
drwxr-xr-x    2 root     root         4096 Apr 26 20:22 Week_6_-_Lecture_17
drwxr-xr-x    2 root     root         4096 Apr 26 20:26 Week_6_-_Lecture_18
drwxr-xr-x    2 root     root         4096 Apr 26 20:30 Week_7_-_Lecture_19
drwxr-xr-x    2 root     root         4096 Apr 26 20:33 Week_7_-_Lecture_20
drwxr-xr-x    2 root     root         4096 Apr 26 20:35 Week_7_-_Lecture_21
drwxr-xr-x    2 root     root         4096 Apr 26 20:38 Week_8_-_Lecture_22
drwxr-xr-x    2 root     root         4096 Apr 26 20:39 Week_8_-_Lecture_23
-rw-------    1 root     root      3386311 Apr 26 20:39 nohup.out

Linkstation returns "Structure needs cleaning" error message to applications when it started after a power failure

Today, my linkstation was not shutdown cleanly due to power failure.
After bringing it up, it showed some weird behavior. Sometimes it hung and the transmission application showed error message "Structure needs cleaning" for some torrents which were already in the download list.

Anyway I was able to log in to the linkstation and investigate the /var/log/message. It showed that the issue was with the XFS files system of the sda6 partition. You can find out the mount point of the sda6 using mount command or cat /proc/mounts command.

root@LS-VL406:~# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,relatime,errors=continue,barrier=0,data=writeback 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
udev /dev tmpfs rw,nosuid,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=4,mode=620 0 0
/dev/ram1 /mnt/ram tmpfs rw,relatime,size=15360k 0 0
/dev/sda1 /boot ext3 rw,relatime,errors=continue,barrier=1,data=writeback 0 0
/dev/disk1_6 /mnt/disk1 xfs rw,noatime,attr2,usrquota,grpquota 0 0
/dev/disk1_6 /opt xfs rw,noatime,attr2,usrquota,grpquota 0 0

root@LS-VL406:~# ls -la /dev/disk1_6

lrwxrwxrwx    1 root     root            4 Mar 31  2015 /dev/disk1_6 -> sda6

I unmounted the /mnt/disk1 and executed xfs_repair as the crash report suggested.

Steps I followed to fix the issue,

1. Check whether the /mnt/disk1 is being used. kill or stop processes which are using the partition.

lsof | grep disk1

2. unmount the partition

umount /mnt/disk1

3. repair the partition

xfs_repair /dev/sda6

Then I rebooted the linkstation and it seems that the issue is resolved now.

Information which was available in the /var/log/messages file about the XFS corruption.

Mar 30 22:09:09 LS-VL406 kernel: Backtrace:                                                                                                                                               
Mar 30 22:09:09 LS-VL406 kernel: [<c001b79c>] (dump_backtrace+0x0/0x114) from [<c001b8e4>] (dump_stack+0x18/0x1c)                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel:  r6:c0481434 r5:c11dc800 r4:c01c870c                                                                                                                                
Mar 30 22:09:09 LS-VL406 kernel: [<c001b8cc>] (dump_stack+0x0/0x1c) from [<c018564c>] (xfs_error_report+0x54/0x64)                                                                                     
Mar 30 22:09:09 LS-VL406 kernel: [<c01855f8>] (xfs_error_report+0x0/0x64) from [<c01856b8>] (xfs_corruption_error+0x5c/0x78)                                                                         
Mar 30 22:09:09 LS-VL406 kernel: [<c018565c>] (xfs_corruption_error+0x0/0x78) from [<c01aed28>] (xfs_bmap_read_extents+0x4f8/0x580)                                                                    
Mar 30 22:09:09 LS-VL406 kernel:  r6:00000000 r5:00000000 r4:c22bf000                                                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel: [<c01ae830>] (xfs_bmap_read_extents+0x0/0x580) from [<c01c870c>] (xfs_iread_extents+0x78/0xe0)                                                                      
Mar 30 22:09:09 LS-VL406 kernel: [<c01c8694>] (xfs_iread_extents+0x0/0xe0) from [<c01a5dbc>] (xfs_bmapi_read+0xec/0x32c)                                                                             
Mar 30 22:09:09 LS-VL406 kernel:  r7:00000000 r6:c62b6240 r5:00000000 r4:c62b6270                                                                                                                    
Mar 30 22:09:09 LS-VL406 kernel: [<c01a5cd0>] (xfs_bmapi_read+0x0/0x32c) from [<c0181098>] (__xfs_get_blocks+0x180/0x524)                                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0180f18>] (__xfs_get_blocks+0x0/0x524) from [<c0181494>] (xfs_get_blocks+0x28/0x30)                                                                              
Mar 30 22:09:09 LS-VL406 kernel: [<c018146c>] (xfs_get_blocks+0x0/0x30) from [<c0109658>] (__block_write_begin+0x1e4/0x4d4)                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0109474>] (__block_write_begin+0x0/0x4d4) from [<c0109af8>] (block_write_begin+0x54/0x88)                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c0109aa4>] (block_write_begin+0x0/0x88) from [<c01816a0>] (xfs_vm_write_begin+0x48/0x7c)                                                                            
Mar 30 22:09:09 LS-VL406 kernel:  r6:18a07ebd r5:00000000 r4:18a07ebd                                                                                                                                
Mar 30 22:09:09 LS-VL406 kernel: [<c0181658>] (xfs_vm_write_begin+0x0/0x7c) from [<c00ac70c>] (generic_perform_write+0x198/0x20c)                                                                    
Mar 30 22:09:09 LS-VL406 kernel:  r8:cbd9e000 r7:00000000 r6:18a07ebd r5:00000143 r4:cbd9fd60                                                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c00ac574>] (generic_perform_write+0x0/0x20c) from [<c00ac7dc>] (generic_file_buffered_write+0x5c/0x90)                                                              
Mar 30 22:09:09 LS-VL406 kernel: [<c00ac780>] (generic_file_buffered_write+0x0/0x90) from [<c0186af0>] (xfs_file_buffered_aio_write+0xc0/0x150)                                                      
Mar 30 22:09:09 LS-VL406 kernel:  r9:cbd9e000 r8:c62b6240 r6:c62b63f8 r5:00000000 r4:18a07ebd                                                                                                          
Mar 30 22:09:09 LS-VL406 kernel: [<c0186a30>] (xfs_file_buffered_aio_write+0x0/0x150) from [<c0186f68>] (xfs_file_aio_write+0x174/0x1ec)                                                             
Mar 30 22:09:09 LS-VL406 kernel: [<c0186df4>] (xfs_file_aio_write+0x0/0x1ec) from [<c00e11a0>] (do_sync_write+0xa8/0xf0)                                                                             
Mar 30 22:09:09 LS-VL406 kernel: [<c00e10f8>] (do_sync_write+0x0/0xf0) from [<c00e1a88>] (vfs_write+0xbc/0x150)                                                                                      
Mar 30 22:09:09 LS-VL406 kernel:  r8:c000bca8 r7:cbd9ffa8 r6:007a81b0 r5:00010000 r4:c5c81f00                                                                                                        
Mar 30 22:09:09 LS-VL406 kernel: [<c00e19cc>] (vfs_write+0x0/0x150) from [<c00e21dc>] (sys_pwrite64+0x80/0x94)                                                                                       
Mar 30 22:09:09 LS-VL406 kernel:  r8:c000bca8 r7:000000b5 r6:c5c81f00 r5:00010000 r4:007a81b0                                                                                                                      
Mar 30 22:09:09 LS-VL406 kernel: [<c00e215c>] (sys_pwrite64+0x0/0x94) from [<c000bb00>] (ret_fast_syscall+0x0/0x2c)                                                                                  
Mar 30 22:09:09 LS-VL406 kernel:  r6:00000000 r5:00000000 r4:18a07ebd                                                                                                                                  
Mar 30 22:09:09 LS-VL406 kernel: XFS (sda6): Corruption detected. Unmount and run xfs_repair

Fix /opt/etc/init.d scripts not executing issue in LS VL Buffalo linkstation

I noticed that the transmission-daemon start up script (S95transmission.sh) added to /opt/etc/init.d/ directory does not get executed when the linkstation is started up.

Let's look at how startup scripts put in to /opt/etc/init.d/ directory are invoked.

1. /etc/init.d/rcS invokes /etc/init.d/rc.optware script. These are the last few lines in the rcS script.

/usr/local/bin/hdd_raid_syncspeed.sh
            
# Optware setup
[ -x /etc/init.d/rc.optware ] && /etc/init.d/rc.optware start
                                                     
exit 0 

2. Then /etc/init.d/rc.optware script mounts /opt partition and invokes /opt/etc/rc.optware script.

case "$1" in
    start)
        echo "Starting Optware."
        if test -n "${REAL_OPT_DIR}"; then
            if ! grep ' /opt ' /proc/mounts >/dev/null 2>&1 ; then
                mkdir -p /opt
                mount -o bind ${REAL_OPT_DIR} /opt
            fi
        fi
[ -x /opt/etc/rc.optware ] && /opt/etc/rc.optware
    ;;
    reconfig)

3. This /opt/etc/rc.optware script simply executes scripts matches with S??*  located in /opt/etc/init.d/ directory.

#!/bin/sh

# Start all init scripts in /opt/etc/init.d
# executing them in numerical order.
#
for i in /opt/etc/init.d/S??* ;do

# Ignore dangling symlinks (if any).
[ ! -f "$i" ] && continue

case "$i" in
  *.sh)
# Source shell script for speed.
(
trap - INT QUIT TSTP
set start
. $i
)

So after some investigations I have found out that rcS script didn't execute it's last few lines. It seemed it exited just after executing /usr/local/bin/hdd_raid_syncspeed.sh line. The simple solution was to move this line to the bottom of the script so that the optware setup is executed first. Now the startup script added to the /opt/etc/rc.optware is started whenever the linkstation is rebooted. :)

The last few lines of rcS after the modification,

#/usr/local/bin/hdd_raid_syncspeed.sh
            
# Optware setup
[ -x /etc/init.d/rc.optware ] && /etc/init.d/rc.optware start
                                         
/usr/local/bin/hdd_raid_syncspeed.sh
                        

exit 0