How system calls are hooked on Win32

Discussion in 'other security issues & news' started by Gullible Jones, Dec 23, 2013.

Thread Status:
Not open for further replies.
  1. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,466
    [Edit: actually this is only one method, there are others.]

    An interesting find, and probably good for HIPS users to know:

    http://uninformed.org/index.cgi?v=8&a=2&p=5#prologue

    In summary, here are two ways of hooking a system call on 32-bit Windows.

    The old method
    1. Find the address of the system call you want to hook
    2. Copy the first few instruction somewhere
    3. Overwrite the original first instruction with a jump to your own function
    4. Splice a jump to the next position onto the copied instruction

    So if it's a HIPS doing the hooking, for instance, then when execution reaches the system call it will jump to one of the HIPS' functions. Depending on user input or whatever, the function will then either jump to the stored instruction, which then jumps back to the rest of the original function; or go somewhere else entirely, denying the system call.

    The new method
    1. Find the the address of the system call
    2. Since the instructions at the beginning of the call are no-ops, just overwrite them with your jump instruction

    Works as before, only this time allowing the system call just jumps back to the call's address (plus however many bytes you overwrote).
     
  2. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Yes, the new method is far superior. Unconditional jmps are fine for *compatibility* but if I have control flow I can simply go straight to the call, not fit for security.

    Callbacks are better, IAT hooking, etc. But hooking in general is *not* good for security, very difficult for some people to grasp how to do it right.
     
  3. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,466
    By jumping directly to the address plus whatever? Couldn't that be done with the new method too?

    Come to think of it, wouldn't the new method be somewhat less secure? With the old one you're copying the first few instructions who-knows-where, then overwriting them. With the new one, the address where the function starts should be entirely predictable - the first five instructions should be no-ops, possibly overwritten, and can be ignored.

    IOW, wouldn't the first method require an attacker to at least have some familiarity with the security software in question?

    I could see why IAT hooking is better I guess. From the attacker's perspective, the address of the original API function is nowhere to be found.

    Edit: Umm wait. When you talk of a security framework without hooking, you mean some standard form of MAC built directly into the kernel, right? With the API functions being forced to obey certain rules themselves, instead of being overridden or whatever?

    That sounds like a great idea, but good luck getting any Linux developers to agree on such a thing. :)
     
    Last edited: Dec 26, 2013
  4. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Not really. If I have control flow over Firefox I can jump past the jmp in Firefox. But the IAT exists outside of Firefox, I can not manipulate it.

    The address is never static, attackers always calculate it - security software won't have an effect on the address in any way that would have impact.

    Seccomp uses no hooks. Integrity levels use no hooks.
     
  5. Gullible Jones

    Gullible Jones Registered Member

    Joined:
    May 16, 2013
    Posts:
    1,466
    Wait, I thought the methods I linked above overwrote stuff in kernel space?

    Ah, okay.

    I was thinking more of LSM.
     
  6. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Yes, I dislike LSM. Though from a 'hooking' point of view there's nothing wrong with its implementation, because it's built into the kernel and you kinda 'hook' LSM, LSM doesn't hook you... sort of.

    I don't believe the above is talking specifically about the kernel, just system calls to the kernel. The hooking would still happen in userspace. I have only skimmed it.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.