My Mamutu Tests

Discussion in 'other anti-malware software' started by Hungry Man, Jul 15, 2011.

Thread Status:
Not open for further replies.
  1. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    I figured I would post this here. It's not an especially formal test but it at least gives some indication of Mamutu's performance. First I tested it with Comodo and then I tested it alone. I'll post the "alone" tests in my next post so that everything isn't super cluttered.

    I chose 15 RANDOM pieces of malware and run each one of them.

    The system is Windows 7 64bit, fully updated - Default UAC, Windows Defender off. Mamutu fully updated. Comodo fully updated. Settings for both at the bottom of this post.



    The number of times Comodo outright blocked an application - 2.
    The number of times default UAC outright blocked an application - 1.
    The number of times Comodo sandboxed and then cleaned malware - 13.
    The number of times Mamutu detected and blocked a program - 9.

    No successful infections. 3 out of the 15 malicious files were blocked before they could successfully start. The 13 files that managed to run were cleaned by Comodo. 9 out of those 13 files were also picked up by Comodo and blocked.


    Now to test Mamutu alone. I think Comodo was getting to everything/ blocking things before Mamutu kicked in. By sandboxing and applying security restrictions on them I think the malicious files weren't able to implement some of the things Mamutu looks for. Still, 9/13 is not bad.

    New test will be in a separate post.



    Mamutu Behavioral Blocker
    Beta updates
    Allow program if 92% of community members allowed it.
    Deny program if 88% of community members allowed it.

    Comodo Internet Security (Firewall and Defense+, no AV) (Password Protected)

    Comodo Firewall: Safe Mode, Alert Settings Low
    -- Ports Stealthed
    -- Create Rules for safe applications
    -- Enable IPv6 filtering
    -- Protect ARP Cache, Block Gratuitous ARP Frames
    -- Block Fragmented IP datagrams
    -- No protocol analysis, no monitoring NDIS protocols other than TCP/IP

    Comodo Defense+: Safe Mode
    -- Autosandbox as Limited
     
  2. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Mamutu stand alone results:

    Successful infections: 2
    UAC Blocked: 2
    Mamutu Blocked: 11

    So mamutu blocked 11/13 malicious files that managed to run without admin.

    It would have been nice to see something break free from Comodo only to be stopped by Mamutu, but Comodo managed to break every piece of malware just fine on its own.

    Mamutu (between its two processes) is using just under 6MB of RAM.
     
  3. emsisoft

    emsisoft Security Expert

    Joined:
    Mar 12, 2004
    Posts:
    328
    Location:
    Nelson, New Zealand
    Thanks for testing!

    Could you please provide us the missed samples for a detail analysis?

    info at emsisoft.com

    Zipped with password please.
     
  4. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Nope! Haha, sorry I really should have thought of that. I was using a VM and after my tests were over I basically scanned to double check what had made it through and then I went ahead and reset my VM's snapshot.

    If I do any more testing I'll keep that in mind next time. Thanks for the great product, it's very light and (from my tests at leasts) very effect at what it's meant to do.

    edit: I may be able to get you one sample actually. I'll see what I can do.
    edit2: I may actually be able to get you more than one... I'll let you know.
    edit3: nope, although if I remember correctly one of them installed a malicious conhost.exe to my windows folder... that's just by memory, I can't recall exactly.
     
    Last edited: Jul 15, 2011
  5. Cudni

    Cudni Global Moderator

    Joined:
    May 24, 2009
    Posts:
    6,963
    Location:
    Somethingshire
    so, as far end users are concerned no benefit at all.
     
  6. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    What do you mean?
     
  7. Kernelwars

    Kernelwars Registered Member

    Joined:
    Aug 12, 2010
    Posts:
    2,155
    Location:
    TX
    i think he meant if you could've sent the missed samples to emsi for analysis..that way if they fix the problem and push a update the community would've benefited from it
     
  8. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Oh, well yes. That would have been nice. The purpose of this test was not to help beef up Mamutu or show that it's vulnerable. It was to see how it performed alone and how it paired with Comodo.

    edit: To clarify, I made this because I personally wanted to see how useful the product was and I wanted to get a feel for what kind of protection it could give me with my other security product and as a stand alone product. The "benefit to the end community" is that they can now see it in a somewhat "real world" environment, however informal the test actually is.
     
  9. Brummelchen

    Brummelchen Registered Member

    Joined:
    Jan 3, 2009
    Posts:
    5,871
    Apples vs Oranges without any further details - pointless
     
  10. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Nice test, on a par with most professional tests.
    Means - just empty numbers without value. :p

    Btw something I learned today - a VM is a somewhat "real world" environment. :blink:

    Cheers
     
  11. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Brummelchen, what further details would you like? I took 15 random malware samples and tested them.

    Like I said, it's an informal test.

    You don't think so? Honestly, I doubt anything I was running was "vm-aware" and I don't see how I could make it any more "real-world" without testing it on my own system, which would defeat the purpose of the test (my security setup is not the one I want tested.)

    The test was for me to see how Comodo and Mamutu work together and to see how Mamutu does alone in the same tests. If you'd like to try to improve the test (to work towards those specific goals) I'm open to it. I still have a VM set up after all it's not super hard for me to do.

    As for "empty numbers without value" that's a matter of opinion. If I have a piece of software block 10% of the malware I download and a piece of software block 50% of the malware I download... guess which one I'm going to be using?

    edit: If you have suggestions for a new test, really, feel free to make them. I can use a new malware index and choose another 15 (or a different number?) malicious files to run. I honestly don't see how I can make it more "real world" than this.
     
    Last edited: Jul 15, 2011
  12. icr

    icr Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    1,589
    Location:
    UK
    There are many programs that are virtual machine aware, I mean when executed in vm it won't cause any activity rather than give an error while if same program is run in actual host machine then it would be different case all together.

    Primarily these are normally rootkits or sometimes even rogue applications that are coded in such a way that they can detect the target machine:ouch: where it is getting executed.

    BTW nice test but rather being informal as you said for yourself, but do try to make some points out of all the suggestion that everyone made.

    My suggestion : always make a backup copy of anything that you are doing be it a bad file, it will help you in pointing out whether its actually a bad file or not;)
     
    Last edited: Jul 15, 2011
  13. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Working outside of a VM isn't an option (I have an XP computer I could do this with but cleanup would be such a pain since it's old.) All of the malware tested either ran and installed or was blocked, if it were vm-aware it wouldn't have run at all (I don't think.) That's why I doubt any of it was.

    I'm not really sure what suggestions anyone has made so far... outside of Emsisoft's original idea to submit the files afterwards, which I'll happily do next time I get the chance to test. As for making the test viable/ more realistic I haven't seen any suggestions for improvement.
     
  14. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Yes, either it's a test on a real machine or a VM test.
    And the term "real world environment" refers to more than just the use of a real machine...

    The bold text is important - if it's just for you, there is no need to publish your results.

    If you like to publish the results, your test should be reproducible, with exact informations about your testing environment and testing procedure, with MD5 hashes of the malware, samples at hand for vendors, etc. etc.
    It should simply meet the minimum requirements for a serious test.

    Do you analyze this "malware" to make sure it is real malware or do you just guess it's malware because there is a link to download this "malware"?

    Cheers
     
  15. Hungry Man

    Hungry Man Registered Member

    Joined:
    May 11, 2011
    Posts:
    9,146
    Do go on...

    I'm not publishing my results as though they're golden. This isn't some av-comparatives type deal where I'm saying to purchase a product on it. And I was actually doing this for another user on notebookreview.com as well as myself as he has a similar setup to mine.

    So reproduce it? No, I dont need MD5 hashes of the malware, what a silly requirement. Perhaps you didn't not see me say "informal."

    Me personally? No. But Comodo analyzed all of it in my first test and called it malware and the site that I got it from is a malware repository... and the scanners after picked up the files as malware... hmmmmm, I wonder...
     
  16. justenough

    justenough Registered Member

    Joined:
    May 13, 2010
    Posts:
    1,549
    Guess that'll teach you, Hungry Man. Doing a post about your informal test within a VM and not saving the files. It's an outrage.

    Really people, is thank you too hard to say? Of course it's a flawed test, but I'm guessing you'd be picking apart professional tests as well. Since there are not that many tests of Mamutu out there, I am glad to see what Hungry Man came up with, even though it's one guy's informal test. It fits with what I already know about Mamutu, a program I've relied on for some time now. My suggestion to the nit-pickers would be to take everything with a grain of salt, try to keep things in perspective, and over time the over-all picture comes together. At least that's how it works for me.
     
    Last edited: Jul 15, 2011
  17. icr

    icr Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    1,589
    Location:
    UK
    Everyone made round about same suggestion(including Emsisoft.....) so everyone was a right word I choose;)

    BTW guess this thread is going nowhere and I don't think it will stay live for long:isay:
     
  18. saakeman

    saakeman Registered Member

    Joined:
    Jul 15, 2011
    Posts:
    89
    Comodo is too good, beats anything, I tested comodo un-updated and it still sand-boxed malware left and right :D :thumb: nothing got through
     
  19. shadek

    shadek Registered Member

    Joined:
    Feb 26, 2008
    Posts:
    2,538
    Location:
    Sweden
    Comodo's sandbox sucks. It auto-sandbox all games... and no matter how much you report this to Comodo they won't fix it in the cloud. You'll have to manually de-activate the sandbox for the processes... and guess if it gets tiresome when you have over 230 games installed on the computer...?
     
  20. LowWaterMark

    LowWaterMark Administrator

    Joined:
    Aug 10, 2002
    Posts:
    18,278
    Location:
    New England
    As stated on the forum previously, and mentioned in this sticky topic, we do not encourage people to do home grown malware testing. Some other issues have also been mentioned in a few of the posts above. So, I'm going to close this thread.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.