Encrypted USB drive partition lost

Discussion in 'encryption problems' started by msahb, Jan 25, 2016.

  1. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    I have a Laptop with LinuxMint / Windows 8.1 dual boot.

    I have a USB stick with a truecrypt volume on it. I use Linux almost exclusivley with the Windows system as a backup when I need it.
    The volume was formatted with a Windows file system, although I don't remember if it was NTFS or [v]fat32.

    I am not entirely sure whether the entire stick is encrypted or a partition on it, but I used to mount it under linux with the command
    truecrypt --mount "$devname" "$mntpoint"
    with devname defined as
    devname=`readlink -e /dev/disk/by-id/usb-SanDisk_Cruzer_Fit_4C530009850124110471-0:0-part1`

    Based on that, and also based on TrueCrypt under Windows showing me something like Disk[x]/Partition1 when I mounted the USB drive manually under Windows, I am assuming it must be a partition.

    The other day, the laptop was low on battery and I put it in sleep mode with the volume mounted, assuming it would survive until I could plug it in.
    But when I opened the lid and turned it on, it started booting rather than waking up form sleep mode, so it must have run out of battery after all.
    So I decided to boot into Windows to run a disk check on my truecrypt container.

    When I started Windows and tried to mount the USB stick, it told me that the volume header had a problem and asked me if I wanted to restore it.
    Since I don't have a header beackup, I selected to restore it from the volume-internal header.
    That did not do anything and I tried to restore the volume-internal header again one or two more times.
    I ended up being able to mount it and the data was there. I started a disk scan under Windows and it told me the file system was fine, so I assumed everything was fine now.

    I unmounted everything and booted into Linux. In Linux, I wanted to mount the container as usual, but now all of a sudden, there is no
    /dev/disk/by-id/usb-SanDisk_Cruzer_Fit_4C530009850124110471-0:0-part1
    The only disk by ID is /dev/disk/by-id/usb-SanDisk_Cruzer_Fit_4C530009850124110471-0:0 (without the -part1)

    TrueCrypt in both Windows and Linux no longer show Disk[x]/Partition1 but only "Disk[x]"

    Trying to mount the disk in Linux TrueCrypt results in Truecrypt saying "mount: you must specify the file system type"

    Trying to mount the disk in Windows TrueCrypt results in TrueCrypt mouting the disk on the specified drive letter and Windows comes up with the specified drive letter in the Windows Explorer, but when I try to access it, Windows wants to format it.

    I had tried the restore header from internal backup again on the Stick/Disk (since the partition was no longer listed), but this time, it did not come back.


    What I do not understand is the fact that I had gotten the partition mounted under Windows after doing restoring the internal header backup and was able to see the files and perform a file system scan which reported that nothing was broken and nothing needed to be fixed in Windows, but then after booting back into Linux right away,
    the partition was gone.


    I assume that I somehow ruined the main partition table of the USB stick and thus TrueCrypt can no longer find the encrypted partition.
    I am also there might be a chance to get back into the partition if I would create a new partition in the partition table, but am not sure what settings to use (I guess the starting position would be the most crucial detail)
    Is there a way to somehow fix the partition of the USB stick in order to enable TrueCrypt to "see" it again, so I can try to mount it?
    I don't have volume header or partition table backup, so the easy way won't work...

    I have tried Testdisk (search only, no modifications), but am not sure which disk type to use (Intel, EFI GPT or None).
    Trying type Intel and then Analyze says "Partition sector doesn't have the endmark 0xAA55"

    Trying type EFI GPT and then Analyse says "Bad GPT partition, invalid signature. Trying alternate GPT. Bad GPT partition, invalid signature."

    Quick Search does not show any partitions (I assume once I go into Quick Search it doesn't matter which disk type selection I came through, does it?).

    Thank you in advance...
     
  2. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    I GOT IT FIXED!!!

    It turns out that I had in fact messed up the USB drive's partition table. I still don't understand how, because the last thing windows did was successfully access the data and then back in Linux the partition was gone right away. Maybe what I did in Windows before getting back to my data messed up the partition table and Windows was still working on a cached intact version of the partition table. That's the only way I can explain it.

    Anyway, here is how I fixed it, I am writing this down hoping that it might help someone else at some point who might stumble across my post.
    This is a way to recover the partition without having to mess with the original drive, trying to restore the partition tableor anything. It works with dump files of the drive, so you only need a basic version of Linux, enough disk space and some luck. Here it goes:

    The partition table was messed up (as I had expected), but the encrypted partition itself was OK (as I had been hoping).
    The problem was that truecrypt had no way to find the partition without the partition table.
    So the challenge was to either restore the partition table or find a way to extract just the partition from the stick.

    Just as a proof of concept, using an old USB drive, I had done the same thing I had done creating this USB drive in question: take the drive as it was (i.e. one partition with data as it came from the manufacturer) and use TrueCrypt under Windows to encrypt the partition. Then back in Linux, using dd, I copied the entire flash drive into a dump, using /dev/sdc (not /dev/sdc1!!!) as "if" parameter.
    dd if=/dev/sdc of=sdc.dump
    Then I "shaved off" the part of the dump that represented whatever was on the drive in front of partition1.
    For this drive, I knew from fdisk that partition1 started at 8kB (16 sectors of 512 byte). So shaving off the 16 sectors:
    dd if=sdc.dump ibs=512 of=sdc1.dump skip=16
    I could have also used ibs=8K and skip=1. Either way...

    That left me with sdc1.dump which should be an exact representation of partition1 (plus whatever else might have been on there after partition1 in case partition1 did not fill up the entire drive up to the last sector - not sure about that one).

    Then I was able to mount sdc1.dump as a regular truecrypt container file.

    Now in order to do the same thing for my messed up USB drive, I had to know how much to shave off of the beginning of the dump.
    I had ordered an identical stick on Amazon hoping that the manufacturer had not changed the specs in the meantime. I also tried to find details online through Google.
    Before the new drive was delivered, I found a very technical feedback on Amazon for my exact stick that mentioned a 16k partition offset. THANK YOU THANK YOU THANK YOU!!!!

    I tried it, and got lucky. Shaved off 16k off of the dump of the messed up drive and I was able to mount the remaining dump as regular input file.

    A hint for those who are not as lucky: you might be able to just shave off an increasing amount of data in the beginning until you hit the right amount.
    The good thing is that truecrypt will tell you even if you do not copy the entire remainder, but only a small piece. This will make the process a whole lot faster.
    If you shave off the correct amount in the beginning and just copy a small piece of the partition (e.g. 10MB), you will not get back into your volume, but you will find if you shaved off the correct amount based on the error message:
    Shaving off the wrong amount of data will give you a "Incorrect password or not a TrueCrypt volume." error.
    Shaving off the correct amount of data will give you a "device-mapper: resume ioctl on truecrypt7 failed: Invalid argument." error

    With that, you could write a script that just copies a few MB from the full dump, shaving off X sectors at the beginning, and trying to mount it in truecrypt.
    Have the script loop over an increasing number for X.
    You know that you hit the right number for X when the error message changes.

    So this is how you might be able to find the correct beginning of a lost partition. I am not sure if the end matters or not, or if it will still work if you copy extra data that is following your partition.

    And obviously, this will only work if you have the correct key/password that you used to encrypt the partition and if you really only messed up the partition table and not the partition itself (and especially the header, I am assuming).

    Hope this is clear enough to be helpful...
     
  3. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    Nice work. I love reading posts where someone gets down and dirty with the fundamentals. It also highlights just how many headaches can be saved by creating a copy of the device partition table. So simple. I always preach headers, partition tables, mbr's until I am worn out. Most don't listen but the ones that do eventually come back and thank me for the suggestions early on.

    I wonder how many sets of eyes will read this thread and realize they don't have these critical backups, only to disregard the notion once again? See where this is headed?
     
  4. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    So now a question, though - I changed from using encrypting partition (encrypting /dev/sdc1) to encrypting the whole drive (encrypting /dev/sdc).
    So now there is no partition table or MBR any more, right?
    I just back up the header and I'm done?
    And restoring the header will automatically do the right thing, regardless of disk layout?

    I guess the downside is that any Windows system I will plug this USB drive in to will want to format it right away. I know I can remove the drive letter in the Computer Management (or whatever it was called), but that only works for a specific machine...
     
  5. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    As I mentioned above I actually enjoyed reading this thread. I can tell that you are beyond fundamental in understanding disk geometry. So you are like a "brother" to me. LOL!

    My personal determination for final structure differs from yours. I have found that some systems (especially Windows) can fail to see a fully device encrypted flash stick sometimes. When you bring up the TC control panel there is nothing to select as the device to mount on those quirky systems. Next, as you have learned from going under the hood using TC, what they label as device encryption is actually partition encryption. 99% of users will simply deploy the out of the box protocol, which again is more aptly determined to be partition encryption rather than true device encryption. My feeling is that because I want hidden volumes I don't want to "tip my hand" that I possess expertise beyond normal scope. Therefore; if I am ever required to mount/deploy the outer volume for an adversary the hidden to me seems more disguised in a normal partition structure. My opinion of course! Volume backup headers, MBR's, are dinky little life saving files. The MBR includes the partition table (446-510 of 512 of course) so just the MBR should be fine for your solo encrypted flash stick. Big files, about 131K for the header and 512 bytes for the MBR and you are done. I have blown away (for testing purposes only) the MBR's and restored many times without any issues at all. For off premises purposes you can even restore a backup volume header that doesn't belong on that volume. An adversary won't know and of course no way to decrypt without a valid header - just don't lose the actual one!
     
  6. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    I had come across a post where the author had argued that encrypting the whole drive was safer from a recovery point of view, because it does not require to backup and rebuild a partition table. The volume will always start at the same place.
    I am not sure where TC starts to put the data in that case. Would it start right at the first sector? Then some tool (or Windows) accidentally messing with the partitions trying to restore a partition table would actually do more damage than it would if the volume was located in a partition a few sectors in.

    I am trying a second approach now where I use a "regular" USB drive as it comes from the manufacturer - simply one partition starting at a given offset (in this case 32 sectors in). I'm shortening that partition just a little bit (e.g. 100MB on a drive with 32GB total) and then creating a second partition in the little 100MB space at the end. Set the partition type to something weird (e.g. "Compaq System Partition") so Windows will leave it alone and not suggest formatting it, then encrypt that partition with TC. It's fine for my use case where I only need to store very little data in that partition.
    That way, I will be back to having to keep a backup of the partition table, but I guess the risk of the TC volume to be accidentally messed up is still smaller back there than if it started right in the first few sectors of the drive.
    Still need to verify that Windows will in fact leave it alone and that the small partition will not show up anywhere unless someone is actually looking at the stick with a disk partition tool.

    On that one, I used "sfdisk -d" to back up the partition table. Besides the actual volume header itself, is that all I need? I am a bit confused with MBR vs partition table, especially in the case of a non-bootable USB drive where I wouldn't expect an actual "Master BOOT Record".
     
  7. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    BTW, with this recent experience, I am at a point were I worry a lot more about recoverability in case of accidents than about total deniability or protection/hideaway from professional adversaries...
     
  8. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    Now you have me running some experiments on a small (256 meg) flash stick just for kicks. I want to compare mbr construction using Gparted versus Windows (or windows third party software) just to see if a hunch I have is correct. Busy today but trying to squeeze this in.

    Let me forward my disclaimer. I only use Linux for my sensitive (hobby actually) system disks. Therefore I no longer have to worry about stupid Windows always trying to fix things that aren't broken. Don't miss that either. I do have 10 Pro on a family machine but nothing there is sensitive. I encrypted the system disk but there is nothing on that disk that would do me any harm or even raise concerns versus any adversary. Strictly privacy e.g. banking (like a bathroom door lock) not really security.

    Stay tuned


    another thought:

    Since you are acquainted with Linux this is like comparing DMCrypt to LUKS. Old school DMCrypt doesn't have any headers and doesn't rely upon canonical information to mount. Its address specific, whereas LUKS has those crucial headers just like TC and VeraCrypt.
     
  9. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    OK I am back. I ran that little 256 meg USB into the ground. LOL! Great little testing stick and I actually have several old 128's to use too. Its been awhile since I have used Windows for "day to day" stuff so I wanted to check before posting my conclusions. I used third party software and a Win 7 VM for these beta's. I was quickly reminded of why I personally prefer to keep a partition on my encrypted devices. So, proceeding with actual partition encryption (TC calls it device but we already established that) I created a FDE TC volume for testing. Put a few junk files in there and its working fine. Now I wanted to test whether or not the generic MBR created prior to volume creation was messed with,and specifically if so how can I fix it. I wanted a much easier way than how you did it above.

    Overview: on a TC FDE (true definition would be partition encryption) volume there are 512 bytes in the MBR, which are not encrypted. Every other byte is encrypted (excluding maybe an odd man out at the end of the space). This MBR contains the partition table information located as 4 slots of 16 bytes each (address given above in this thread). The first 445 bytes are where the loader is placed on a system disk MBR, but of course not in this application. If we zero out the MBR or even fracture the partition table, the TC volume cannot mount, which you discovered and started this thread. How can I get my entire mbr and/or partition table restored? Not even an issue really on a ONE partition encrypted device for my testing. Continuing; I always do a forensic clean of hardware I intend to use for archival purposes (not in this test instance). That means for me its a FAT32 partition where I have software tools that really scrub the sectors down. Then I zero the MBR multiple times and rebuild it to conventional msdos virgin. That is my universal starting point for my encrypted devices. It seemed logical to me that if I were to rebuild the MBR back to conventional virgin AFTER intentionally messing it up with an encrypted volume already on it, that the volume should still mount. It did and every single time without fail. My conclusion is that on a one partition device starting as I described the original MBR can always be rebuilt back to the device and its good to go. Simple and sure. Better - I don't have to save any MBR's on one partition devices because software can one click rebuild an MBR and gets it right every time when I try it.

    By staying with a partition I don't have to worry about the device disappearing like you described above (from the control panel), and TC will find the partition smoothly. If my test is accurate than your critical save is the BACKUP VOLUME HEADER and knowing how to rebuild an MBR when needed (under 5 seconds).

    The process becomes only slightly more complicated when the encrypted volume is on the third or fourth partition(s) of a platter. Still easy, but that is another thread.

    It is easy for me to run these tests because I keep small size media around for the task. I suspect that had you rebuilt the MBR on your usb external drive you would have been able to open it up immediately. What do you think? I know its more risky to do this to your e.g. - 2 TB drive as compared to a 256 meg stick. The process is the same though.
     
  10. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    I was looking through some of my old archives and found small zip files I created years ago. I did some house cleaning since several volumes no longer exist. Those zips contained 512 byte mbr backups and their respective volume header backups for TC 6.3a volumes! If you don't feel comfortable with the test results posted above I suspect you are aware of how to copy out mbr files for safe keeping. The mbr contains the partition table. With those two files in my zip and yours if you did the same, you could easily restore that key data if problems surface. I should say when problems surface but I am trying to think positively. LOL!

    You can name the zip files something meaningful to you making it easy to sort and find them when needed.
     
  11. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
  12. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    Oops, sorry, somehow I missed your posts...

    > I suspect that had you rebuilt the MBR on your usb external drive you would have been able to open it up immediately. What do you think?​
    Yes, probably, but it would have been a one shot "get it right or mess it up even more" - so I decided to mess with the dump files instead.

    What I do not understand is rebuilding of the MBR, or to be more exact - how whichever tool I use to rebuild the MBR knows where the first partition originally started. Looking at different USB sticks, which are all in their original state (partition-wise), they all start at a different position.
    So if I restore a backup of the MBR, then sure, no problem - the info is in there.
    But if I rebuild the MBR from scratch, and especially with a TC encrypted volume that has no recognizable pattern that a recovery tool could go by, I'm not sure if that will always work...

    It will probably always work for you since you said you are originally wiping and building everything (including MBR) from scratch BEFORE you create the encrypted partition. That way, the starting position of partition 1 probably depends on the tool you use for partitioning. If you re-do the same thing later after breaking your MBR, then rebuilding should probably get you the same MBR back, at least if you use the same tool (and version) you originally used.

    If you do not create your own original partition layout and just encrypt whatever partition is there out of the box, you will probably be hosed (unless you are lucky and able to guess it using my method above, which is by no means reliable).

    So yes, MBR backups and volume header backups. ALWAYS
     
    Last edited: Feb 9, 2016
  13. msahb

    msahb Registered Member

    Joined:
    Jan 25, 2016
    Posts:
    7
    Another thing, about the test I described further up of hiding a small second 100MB "Compaq System Partition" partition on a big USB drive - this works perfectly using Linux and TC under Linux shows me both partition in the device selection dialog.
    However, booting into Windows, TC displays the drive with the first partition only, no second partition that I could select and mount... Weird...
    I will try changing the partition type for the second partition to something less exotic, maybe type 82 (Linux swap), but to TC, that should not matter. So somehow Windows must be skipping that partition when TC requests a list of all partitions form the OS (assuming that's what TC does)...
     
  14. Palancar

    Palancar Registered Member

    Joined:
    Oct 26, 2011
    Posts:
    1,592
    I am glad you took the time to come back and post. I wanted to make sure you saw that there were times a generic software MBR rebuild did not work. I appreciate the work you went through to restore your partition table but its so much time and tension that is not necessary.

    I am glad you are adding a full MBR backup to the files you keep for recovery of those external drives. As you know a MBR backup is 512 bytes and takes a fraction of a second to generate using a linux terminal command. But what a life saver those 512 bytes can be!
     
Loading...