How to shutdown after dd completes or alternative imaging for Linux with that feature

Discussion in 'all things UNIX' started by J_L, Aug 19, 2014.

Thread Status:
Not open for further replies.
  1. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    I've decided to use dd, because it's the only one I know that can hot image in Linux. Problem is, I don't know how to schedule a shutdown after it completes. If that's not possible, please provide any alternatives with that feature.

    Although cold imaging is acceptable and Clonezilla has that feature, unfortunately it does not work with my FakeRAID device even after enabling and mounting it with dmraid. At least that's the case in Hiren's Parted Magic. Hiren's Mini XP has no problems with the NTFS FakeRAID, but doesn't seem to have an imaging program with my requested feature.

    Freeware-only please.
     
  2. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Actually, forget about dd hot imaging, it's as slow as molasses. 13.1 MB/s, are you serious?! Macrium was at least 4x faster. Now how to make it shutdown after imaging...
     
  3. jnthn

    jnthn Registered Member

    Joined:
    Sep 22, 2010
    Posts:
    185
    Ctrl c (edit: This is to stop the process. Safe bet is to just let dd finish on its own since you are doing an image. If it's done it'll pop a message like xxx recotds in xxx records out)? I think dd is slow cos it does a sector by sector copy (even of free space).
     
  4. oliverjia

    oliverjia Registered Member

    Joined:
    Jul 21, 2005
    Posts:
    1,517
    J_L, macrium reflect does not support EXT4, only supports up to EXT3 currently. For EXT4, I think only sector-by-sector imaging is possible with macrium.
     
  5. NGRhodes

    NGRhodes Registered Member

    Joined:
    Jun 23, 2003
    Posts:
    2,331
    Location:
    West Yorkshire, UK
    if your doing it from the command line, just join a shutdown command e.g.

    sudo dd if=/dev/sda of=/mnt/somewhere && sudo shutdown -h now
     
  6. NGRhodes

    NGRhodes Registered Member

    Joined:
    Jun 23, 2003
    Posts:
    2,331
    Location:
    West Yorkshire, UK
    Better off using the Linux standard dump/restore if you want to skip free space (runs at a file system level rather than partition level though).
    You can also look at piping to tar or cpio (compression) which may improve performance.
     
  7. amarildojr

    amarildojr Registered Member

    Joined:
    Aug 8, 2013
    Posts:
    1,989
    Location:
    Brasil
    Does this work on Arch or it's a Debian-family only command? On Arch I do:

    dd if=/input of=/output bs=[sector physical size] && sync ; shutdown -P now

    So it ends up:

     
  8. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    I'd rather not risk Ctrl+c. How do I make it faster? 13.1 was with
    conv=sync,noerror bs=64K and gzip after waiting 3+ hours to complete, normally it seems to be 3-4x slower (although I had to use Ctrl+c that time). I'll try fiddling around with the options later.
    * Removing gzip makes it 30 MB/s and conv=sync,noerror shouldn't be necessary. Tried other bs, but this was the fastest.

    Macrium Reflect detects my drive as ext3 and the default options seems to work normally. Even restored it successfully and made an incremental.

    Thanks, I'll try &&, never thought it was that easy (maybe that's why my Google searches were fruitless). What is the Linux standard dump/restore? How reliable is it?
     
    Last edited: Aug 19, 2014
  9. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Just tried &&, but it prompted me for my password again. I guess su instead of sudo would fix that, which I have to enable.

    Is there a fast compression available that won't affect imaging speed? Gzip -9 doesn't still slows it down. Not sure how to do tar.
     
  10. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,461
    Disclaimer: the only official means of hot imaging on Linux are LVM or btrfs snapshots. Snapshots are the only way you can guarantee the integrity of individual files (and they're still not sufficient for things like SQL databases that bypass the filesystem, but that's another matter).

    ... With that aside, what you want is probably 'gzip -1', minimal compression. -9 is the slowest setting. Another possibility might be lzop, if you have a live CD that supports it. lzop is not very good compression, but compresses fast.

    Do keep in mind though that
    - dd usually wastes a lot of space, even with compression, since deleted files are still present on the disk. This will be especially evident with lzop.
    - dd does not do anything to ensure filesystem integrity. If something goes wrong, you may lose the whole backup, not just a few files.

    This sort of thing is really better done with tar or cpio, on a filesystem level. That is also much faster. For a live backup you could do something like this:

    Code:
    # cd /
    # find . -xdev | cpio -o -H newc | gzip > /path/to/backup/volume/backup-blahblah.cpio.gz
    data i
    find with -xdev doesn't descend into other partitions or filesystems, which prevents cpio from trying to snarf up virtual filesystems like /proc. The caveat is that if you have a separate partition for /home, etc. that you want backed up, you have to specify it too in the find command (since find would otherwise not descent into it).

    cpio -o just means create an archive. -H newc makes sure to use a format compatible with the Busybox version, in case you're stuck with that.

    Restore just consists of
    - partitioning and formatting a drive
    - mounting the partitions and cd'ing in
    - unpacking the archive with 'gzip -cd /path/to/backup.cpio.gz | cpio -idu'
    - updating /etc/fstab and reinstalling the bootloader

    (If you want more info on that BTW feel free to ask; this is how I mantain Debian system backups.)[/code]
     
  11. jnthn

    jnthn Registered Member

    Joined:
    Sep 22, 2010
    Posts:
    185
    Not to derail or anything, but Disks util has an option to take an image of a disk. Does it employ dd to do so?
     
  12. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,461
    Which disks utility? There are over a dozen for Linux.
     
  13. jnthn

    jnthn Registered Member

    Joined:
    Sep 22, 2010
    Posts:
    185
    gnome-disk-utility
     
  14. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,461
    Hmm, I'd never noticed that. Might be for btrfs and LVM volumes only, like Snapper? Not sure.
     
  15. NGRhodes

    NGRhodes Registered Member

    Joined:
    Jun 23, 2003
    Posts:
    2,331
    Location:
    West Yorkshire, UK
    You also need a filesystem which supports freezing during LVM snapshots - EXT4 and XFS do (seamlessly, no user invention needed when doing an LVM snapshot).
     
  16. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    It's too late for LVM or brtfs unfortunately. gzip -1 is still too slow. I'll try what Gullible Jones suggested, is this how to make it fit my criteria?:
    Code:
    sudo -i
    cd /
    find . -xdev | cpio -o -H newc | gzip > /path/to/backup/volume/backup-blahblah.cpio.gz && poweroff
    Only issue with that is I'm not sure how to do this:
    "- updating /etc/fstab and reinstalling the bootloader"

    Can anyone recommend a lightweight LiveCD that supports what Mr. Jones and my original post described? I need something faster for my old machine.
     
    Last edited: Aug 19, 2014
  17. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    So let me get this straight, using Lubuntu LiveCd (which handily recognizes my RAID automatically), I open up GParted. Then unless the whole disk is corrupt, I just format the system partition. Then mount and cd it. Then follow your command.

    I'm not sure what updating /etc/fstab means when it's already backed up, and is reinstalling the bootloader even necessary when you haven't created the partition table from scratch? Guess I'll find out.

    *Stuck on third step, permission denied. Can't seem to write to the new partition even with sudo. Strangely, I can delete the automatically created lost+found folder on the same mounted partition.

    *sudo -i to the rescue yet again. I'll do this just in case since I deleted and recreated the partition in my panic to gain permission: sudo grub-install --root-directory=/media/sda1 /dev/sda

    *Damn, gave up waiting for root device. So this is what you mean by updating /etc/fstab. But how do I do that?

    * I get it now, format changes the UUID. Just update /etc/fstab with sudo blkid right?
     
    Last edited: Aug 20, 2014
  18. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    I'm officially stuck. Even after doing all of that again, /proc/cmdline still references the wrong UUID. Since that file is temporary, I cannot edit it. What did I miss?

    *Seems like grub itself is somehow set to the wrong UUID. This time I'll try manually setting the UUID of the partition back to what it was before.
     
    Last edited: Aug 20, 2014
  19. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Success at last! Here's how I restored my drive:
    Code:
    System Tools > GParted > /dev/sda1 > Format to > ext4
    LXTerminal > sudo -i  > tune2fs /dev/sda1 -U c9ce7109-d95a-4681-bbcb-c32b262abb7e
    PCManFM > 80 GB volume > Mount Volume > cd /media/lubuntu/c9ce7109-d95a-4681-bbcb-c32b262abb7e
    gzip -cd /media/lubuntu/09B4E44B3C129DDA/eOS.cpio.gz | cpio -idu && poweroff
    Reinstalling grub isn't needed after all unless the MBR is corrupt. Hoping to make a script out of this one day.

    Only issue was permissions I needed to fix in /media.
     
  20. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,461
    Yeah, reinstalling GRUB isn't necessary if it's already installed. (Sorry for the slow response, I'm on east coast time.)

    Yes, updating fstab => you edit /etc/fstab to use the new UUIDS or device names, or more preferably labels via LABEL=whatever. Your method of modifying the device UUID directly also works.

    For stuff like this I think you always want to use sudo -i.

    Not sure how you got permissions errors on /media, cpio preservers permissions by default.

    Oh, when decompressing the cpio archive you must be root. Otherwise you will get a superficially working system with totally messed up, insecure permissions; because root is the only user on Linux that can set file ownership (as opposed to just rwx permissions).
     
  21. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Thought so, I did locate and remove Flexnet.

    Are you sure updating /etc/fstab is enough? When I did that, grub still points to the original UUID when I checked the boot parameters.

    I could only use sudo -i to even write to the drive. Otherwise permission denied.

    It was a folder mounted within /media that somehow became 750 instead of 755.

    Although I had to be root, going to double-check. So I see if /home/me is owned by J_L and other folders by root?
     
  22. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,461
    Right, on Debian or Ubuntu you'd run update-grub. For Fedora or other distros it would be 'grub2-mkconfig -o /boot/grub/grub.cfg'.

    Not sure why that is, but then I'm not a great fan of sudo.

    Not sure what happened there either...

    Yup, you don't want your user having RW access to the rest of the filesystem. Though if you did run the cpio, etc. commands as root that should not be a problem.
     
  23. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Oh I see. I still prefer restoring the original UUID onto the partition, since I can automate a shutdown.

    "sudo gzip..." wouldn't work either, I needed to become root.

    Hope it doesn't occur next time, messed with my symbolic links and home folder icons.

    OK, will report back if there's such a problem.
     
Loading...
Thread Status:
Not open for further replies.