Solutions to truecrypt's fatal flaw: power outage data loss

Discussion in 'privacy technology' started by Ulysses_, Nov 24, 2013.

Thread Status:
Not open for further replies.
  1. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    With truecrypt, power outage can cause significant data loss. Allegedly, the entire container or partition can easily be lost according to these two people:
    The above are from here and here.

    Any countermeasures to this flaw, like some sort of redundancy or whatever smart scheme to deal with power outage so as much data as possible can be recovered with conventional recovery and repair tools with as little corruption as possible?

    Any alternative encryption software that takes better care of power outage or disk failure or unsafe unplugging or corruption in the header or crash etc?
     
    Last edited: Nov 24, 2013
  2. chiraldude

    chiraldude Registered Member

    Joined:
    Jul 3, 2010
    Posts:
    157
    From what I gather from the posts you linked to and other recent discussions, this issue seems to be particularly common when running from external USB drives. If that is correct and people are running an OS from an external USB drive, there isn't much you can do. USB was never meant for this!
    You could use ESATA for the external drive so it behaves as an internal drive.
     
  3. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    17,039
    How about a good UPS Battery Backup. I run my whole system including external drives on them, and have plenty of time to close everything done.
     
  4. dantz

    dantz Registered Member

    Joined:
    Jan 19, 2007
    Posts:
    991
    Location:
    Hawaii
    I don't believe this is a common issue at all. I've heard of a few cases, but there never seem to be enough useful details included to explain what actually happened.

    I've performed extensive "sudden shutdown" testing of various types of TC volumes without losing anything other than editing changes to open files that hadn't yet been written to disk.

    Of course, the sudden shutdown of a properly-working system is not the same thing as getting Windows into an unstable state that leads to a crash.

    Here's how I see it: During the unstable state immediately preceding a crash, or during the first bootup after a crash, Windows sometimes performs unwanted actions such as writing fresh boot records to encrypted partitions or reinitializing encrypted disks.

    To anthropomorphise a bit, you might say that Windows wakes up groggily after hitting the floor (dead drunk, or knocked out by a 2x4, or whatever), looks around and notices various problems that ought to be corrected. "Damn, that partition doesn't even have a valid boot record. Guess I'd better write a new one! I won't even bother asking the user, he can thank me later. Done! Ugh, my head hurts. What happened? How did I get here? I can't remember anything. So, what else is wrong? OMG, I must have lost the initialization on the external disk! The user will be so angry! I'd better take care of it before he finds out. There, done! Now we're good to go."

    If the partition or the disk had contained TrueCrypt volumes then both of those actions would have damaged their volume headers. However, at that point both volumes could easily be recovered by restoring the headers from a backup.

    Of course, during or after a crash Windows isn't the only one that might not be making clear decisions. Users tend to go into panic mode when they discover that they suddenly can't get into their encrypted volumes, and they will sometimes hurriedly perform inappropriate actions that end up causing further damage (such as formatting the problem disk, running chkdsk, running testdisk, making partitioning changes, etc.) So please bear with me while I run through another made-up conversation:

    User: "Windows, help me! I've lost access to my encrypted data! What should I do?"

    Windows: "Well, uhhhhhhh, I guess we could try formatting that partition. It doesn't seem to be working right anyway. It has a valid boot record (hah, I just put it there a few seconds ago and you never even noticed), but the data seems kind of screwy. I think the file system is broken."

    User: "But what about the external disk that contains all of my life's work? It's not accepting my password!"

    Windows: "Hmmm, well, let me see. Ok, for starters, it needs needs a fresh MBR and partition table. Let's try that first. Then we can always format it afterwards to get it working again."

    "OK! OK!" clicks the user.

    Now the problems have been compounded and the user is in deep trouble. Time to hit the forums and ask for help!

    Every story has a moral, so here you go: Where encrypted data is involved, Windows is not your friend. Windows doesn't have a clue how to handle it. Back up your data and your headers before this sort of thing happens to you.
     
  5. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    Hasn't anyone thought of modifying truecrypt sources so it echoes every write to a partition to a second partition, with flushes so that only one partition is being written to at any time, therefore the other partition is guaranteed valid enough, with recovery prospects as good as if encryption were not used?
     
    Last edited: Nov 26, 2013
  6. chrisretusn

    chrisretusn Registered Member

    Joined:
    Jun 16, 2004
    Posts:
    1,322
    Location:
    Philippines
    My first reaction, solution, as already mentioned by Peter2150; A good UPS. All of my systems are protect by them. My computers are protect by a UPS that provides unattended shutdown capability. Where I live power outages are a normal part of life.
     
  7. PaulyDefran

    PaulyDefran Registered Member

    Joined:
    Dec 1, 2011
    Posts:
    1,163
    I've had BSODs/Hard Resets with containers mounted. Never recall NTFS formatted ones even making a peep. Fat32 ones complain, but a "Fix Filesystem" hasn't resulted in any data loss yet. As far as power outages, just get a dang UPS :D ...or a laptop :D
     
  8. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    Thanks. If ALL encrypted data can be lost in a power failure, surely there must be something very wrong with the design.

    Any better designs that do not fail so drastically and do not depend so heavily on a ups?
     
  9. dantz

    dantz Registered Member

    Joined:
    Jan 19, 2007
    Posts:
    991
    Location:
    Hawaii
    It's not an "all encrypted data is going to be lost" situation. If a power failure causes your header to be damaged then you restore it from a backup.

    But yes, the software could be designed to be more fault-tolerant. I discussed this in another thread. The tradeoff is, it would have to be done at the expense of other features that the users seem to like even more.
     
  10. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    The solution can be as simple as repeating the header 10 times in the partition - no need to bother the user with backup headers, a valid header can be discovered transparently from the user.

    Or even having 100 containers and using something like FUSE to make them look as one big drive - then if one container breaks, all other containers remain intact and the loss is limited to files with sectors in the broken container.
     
    Last edited: Nov 29, 2013
  11. luciddream

    luciddream Registered Member

    Joined:
    Mar 22, 2007
    Posts:
    2,497
    Bingo!... was just about to say the same thing. I once made a poll in here asking how many people used a surge protector w/battery backup for their computers/monitors/etc...

    I said that I think it's ridiculous how people can be so obsessed with malware screwing up their computers that they buy/use all this security software, when a power outage/surge can do more damage than any malware could. Even completely fry your hardware. And for around $50 you can address this vector.

    I have my computer, monitor, speakers, router, modem all plugged into it. Of course if the electricity goes out the modem won't work anyway (phone lines out), but it protects it from a surge anyway. It's an APC ES 550 with 8 outlets, two RJ-45 jacks, and a data port to sync with the "power options" in your control panel. If I'm around when it happens I'll safely shutdown my computer. But if I'm not I have it so that when the battery reaches 25% it'll shut it down safely for me.

    Everyone that owns a computer should own one of these. I have 2 actually, getting another for my TV, PS2, PS3, lamp and other things.
     
    Last edited: Dec 1, 2013
  12. HopelesslyFaithful

    HopelesslyFaithful Registered Member

    Joined:
    Nov 14, 2012
    Posts:
    65
    Location:
    IL


    umm phone lines are not connected to power BTW...Also many cell sites and phone sites have back up gens. I always get my father-in-law to throw up the gen in black outs so the internet keeps working and i can do fun stuff instead of looking at a wall....or talking to the wife or other people ^-^ i kid about the talking to wife part but other people...sigh
     
  13. luciddream

    luciddream Registered Member

    Joined:
    Mar 22, 2007
    Posts:
    2,497
    Whenever the power goes out in my neighborhood the phones are down too. They must be tied into the power at some point, like the transformers on the telephone poles.

    If I plug the cords into the RJ-45 jacks in my APC the power seems on on my LCD screen on the phone but I can't send or receive any calls.

    Maybe they have me backdoored, lol... who knows? But as far as the OP goes I believe this is the answer. Everyone should have one.
     
  14. HopelesslyFaithful

    HopelesslyFaithful Registered Member

    Joined:
    Nov 14, 2012
    Posts:
    65
    Location:
    IL
    agreed but also if phones are out you always have a cellphone for data...at least if you got sprint and a smartphone ^^ PDAnet woot woot. I range from 1-200GB a month lol...though 200GB was only once :)
     
  15. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    I've had power outages and the like a thousand times. Never lost user data. The scanner always found it was temporary files or caches by windows. At worst, user data missed the latest edits. With truecrypt I would have lost ALL data.

    Unless a backup was kept, of the header. Or a data backup. Go figure. Also, a UPS is only for power outage.
     
    Last edited: Dec 2, 2013
  16. dantz

    dantz Registered Member

    Joined:
    Jan 19, 2007
    Posts:
    991
    Location:
    Hawaii
    That's a bit of a stretch. I've performed numerous "sudden power cut" tests to both mounted and unmounted TC volumes and haven't lost anything but data in RAM (e.g. intentionally unsaved edits) that hadn't yet been written to disk.
     
    Last edited: Dec 2, 2013
  17. Enigm

    Enigm Registered Member

    Joined:
    Dec 11, 2008
    Posts:
    188
    No you wouldn't .
    LOTS of users experience power-failures and lose exactly nothing .
    I've used TC since version 4.1 (2005) and have not lost a single bit of data .

    But I don't see any point in discussing this any further with you, you have clearly made up your mind, based on 2 dubious internet-posts and stopped listening .
    IF you ever 'loose' all your TC-data :
    It's your own fault, even when windows does it 'initialize disk'-stunt .
     
  18. luciddream

    luciddream Registered Member

    Joined:
    Mar 22, 2007
    Posts:
    2,497
    The ones that come with battery backups are also to be able to keep your machinery running during an outage, usually to shut it down properly. And IMO this is the only kind worth having as far as what we're discussing here is involved.

    But I lived without one for several years before coming to my senses, and have numerous power outages. Not once did I lose any data, from mounted/unmounted volumes to any type of data. The only thing I ever got was that *spanking*, err... message telling me I didn't shut down properly, as if I had a choice in the matter.
     
  19. S.B.

    S.B. Registered Member

    Joined:
    Jan 20, 2003
    Posts:
    150
    I too have used TrueCrypt for years and years and have never lost one bit of data from a bad shutdown or power loss. I did at one point lose data during a reinstall of Windows -- my own carelessness -- I allowed Windows to partially overwrite a TC disk.

    But this so-called loss of power flaw; it doesn't exist. Period. Paragraph.

    On the other hand, a defective and/or failing hard drive could easily cause Windows to shut down and cause loss of hard drive data including any encrypted data on the hard drive. Similarly an improperly installed hard drive and/or a defective or improperly installed motherboard could cause data loss and shutdown.

    Let's put the whole thing this way; just because two things happen at the same time doesn't mean that one caused the other. It takes lots of tests, including properly designed tests before one can validly establish a cause/effect relationship. That's a fundamental rule of science.

    In the present case we have lots and lots of evidence that power loss doesn't cause TrueCrypt loss with properly operating and installed hard drives. It's obvious that the problem lies elsewhere.

    __
     
  20. S.B.

    S.B. Registered Member

    Joined:
    Jan 20, 2003
    Posts:
    150
    Further debunking the baseless "fatal flaw" assertion of this thread, attention is directed to the following, found in TrueCrypt FAQs:
    What will happen when a part of a TrueCrypt volume becomes corrupted?

    In encrypted data, one corrupted bit usually corrupts the whole ciphertext block in which it occurred. The ciphertext block size used by TrueCrypt is 16 bytes (i.e., 128 bits). The mode of operation used by TrueCrypt ensures that if data corruption occurs within a block, the remaining blocks are not affected. See also the question 'What do I do when the encrypted filesystem on my TrueCrypt volume is corrupted?

    * * *​

    Additionally, it must be understood that data loss and data corruption is a significant problem associated with computers and hard drives, in general. Numerous software programs and specialized data recovery companies attest to the general problem of computer and hard drive data loss and corruption (see, for example, 334 software results found for "data recovery" on SOFTPEDIA). The problem is completely independent of data encryption and/or TrueCrypt.

    Nevertheless, it is to be noted that problems of data recovery following data loss and/or data corruption due to various hardware problems and/or user errors can be significantly exacerbated in the case of encrypted data. Backup of a user's data is recommended in all computer setups and is particularly important in the case of encrypted data, regardless of software or hardware used to encrypt the data.

    __
     
    Last edited: Dec 3, 2013
  21. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    If your experience is so different from other people's using truecrypt, such as those linked to in the OP, there must be a reason. Here's one possible explanation: you have been using containers, they have been using FDE. Is that so?
     
  22. S.B.

    S.B. Registered Member

    Joined:
    Jan 20, 2003
    Posts:
    150
    My experience is not different from typical TrueCrypt users. (See for example other replies in this same thread.) It's you who is having unusual problems. The problem you claim to have simply does not occur with typical users. The fact that you have found two reports of other users (out of millions of TC users) who seem to have similar problems suggests that the problem you claim to have is very, very, very, rare; and most likely stems from something other than TC.

    As noted there are literally millions of TC users. If the problem you describe were common, both Google and the TC forums would be flooded with reports of the problem. But such a flood reporting this problem simply does not exist anywhere.

    __
     
    Last edited: Dec 3, 2013
  23. Reality

    Reality Registered Member

    Joined:
    Aug 25, 2013
    Posts:
    687
    Not so. Depends on what type you get. My APC UPS also cleans "dirty power" like transients, spikes surges dips and brownouts. All in all an essential piece of equipment for the conscientious. Like others have said, before I got mine, I had outages and lost no data, BUT its not so much the hard shut down, but the cycling on of the power grid when the power comes up again when most of the damage is done (at least in my country) . You also have possible anomalies such as compromised circuitry which doesn't fail straight away, but its life can be shortened and when it does fail you don't remember about your unprotected equipment in things like an electrical storm.
     
  24. S.B.

    S.B. Registered Member

    Joined:
    Jan 20, 2003
    Posts:
    150
    Spent a few minutes looking at the threads cited by OP. Turns out that one of the comments the OP cites (first one) says absolutely nothing about power loss or bad shutdown causing data loss. It simply says that the header was corrupted by Windows:

    "I lost 300GB of data due to my volume header becoming corrupted by Windows. I had my entire system encrypted (boot partition and drives) and attempted to use the various repair features and decryption to no avail."

    So the OP has actually cited only a single instance of another user claiming that power loss or bad shutdown is a problem with TC.

    Meanwhile long time Wilders' participant, dantz, reports in this same thread of extensive testing of TC for data loss caused by bad shutdowns; but reports finding no data loss whatsoever. The extensive, and technically competent TC data recovery assistance generously given by dantz to numerous Wilders' users and visitors gives a high level of credibility to dantz's TC testing IMO.

    Other users in this thread report use of TC over many years without data loss caused by bad shutdown or power loss.

    Even in the first thread cited by OP, multiple users report extensive TC use without data loss:

    "I've used Truecrypt extensively for the past year or so and have yet to have any data loss...."

    "I think TrueCrypt is very robust itself, I've been using it for over 4 years and haven't had a problem..."

    Note that the above users recommend regular backup of encrypted data, as is also recommended by most here.

    __
     
    Last edited: Dec 4, 2013
  25. Ulysses_

    Ulysses_ Registered Member

    Joined:
    Jun 27, 2010
    Posts:
    207
    S.B. your blatant partiality and dishonest refusing to acknowledge any issue with FDE as implemented by truecrypt (and possibly container encryption too), you are not serving your employer Truecrypt Inc. Because anyone can google 'truecrypt' and 'corruption' or something similar, and dig up far more instances of unnecessarily excessive data loss.

    Cases where the loss would not have been so excessive if encryption were not used at all, or if per-file encryption was used. Not that I condone per-file encryption, just saying there must be more resilient designs: for example something in the middle between per-file encryption and full-disk encryption where files are grouped and arbitrarily padded, or the equivalent of a fuse file system joining together lots of tiny containers as if they were sectors or blocks. But this is a programmer thinking loud, perhaps this is the wrong forum for that.

    If people in more google results came here and asked why they lost so much data, you couldn't say "this is rare, you were the unlucky one, not for us to worry". Imagine an operating system being sold under the assumption that people could lose all their data but that's ok because it only happens seldom!

    There must be designs where the data loss is almost as small as non-encryption. At the expense of space used? Or performance?
     
    Last edited: Dec 5, 2013
Loading...
Thread Status:
Not open for further replies.