Encrypting virtual memory as a possible anti-exploit strategy

Discussion in 'all things UNIX' started by Gullible Jones, Feb 3, 2015.

  1. Gullible Jones

    Gullible Jones Registered Member

    May 16, 2013
    Maybe a stupid idea:
    - On boot, the kernel generates a random key, and stores it somewhere in kernel-reserved memory
    - Kernel space is (obviously) left untouched
    - The rest of the virtual memory space (both physical and swap) is scrambled with a fast symmetric algorithm like Blowfish or AES, preferably with a block size larger than the page size
    - When pages are retrieved from memory or swap, they're first decrypted by the kernel
    - When pages are put back in memory or swap, they are encrypted

    My thinking on this is that a LOT of exploits rely on the physical layout of stuff in main memory; so transparently encrypting it, with a symmetric key stored somewhere in protected kernel space, would make things very very difficult for an attacker.

    My questions...

    1. Is this even possible? My understanding of CPU and memory architecture may be inadequate here.

    2. Would it actually have any significant impact on the efficacy of memory exploits? Which classes of exploits would it help with? I would expect stack smashing to be made difficult for instance, but exploits against e.g. null pointer dereferences to be unaffected.

    3. Finally, if yes to both of the above... Is it practical? Even "fast" cyphers like AES can be pretty slow.

    Am I barking entirely up the wrong tree here?
  2. safeguy

    safeguy Registered Member

    Jun 14, 2010
    My understanding of the subject is limited to what I've read so forgive me if I'm wrong here.

    Encrypting virtual memory might serve a purpose in protecting against information leak /unauthorized access or retrieval of sensitive data. Therefore, it is applicable for those who value privacy.

    I can't see how encrypting virtual memory would help against memory exploits though. When an attacker attempts to predict target addresses or jump to an exploited function in memory, it would have been in decrypted state anyway. What you want to do here is already done by randomizing the location of the address space...both in userspace and kernel.

    Even then, ASLR and KASLR has its limitations. Current mainstream OS kernels are either monolithic or hybrid - the subsystem components run in the same address space as the kernel.
  3. NGRhodes

    NGRhodes Registered Member

    Jun 23, 2003
    West Yorkshire, UK
    What exploits are you thinking about ? - ASLR (if used correctly/fully) will prevent exploits that are coded to specific memory address offsets

    One thing you have not thought about: the code would have to be decoded and stored somewhere before it can be executed, still exist at some point decrypted and therefore still be exploitable.

    Cheers, Nick
  4. deBoetie

    deBoetie Registered Member

    Aug 7, 2013
    Agree, that's what would concern me about the scheme, and you could be introducing additional vulnerabilities. It's similar to the situation where you're using SecureString in .Net, it's hard to fully protect against exploits or writing unprotected to swap or whatever. And once the key leaked, it'd be game over presumably.