I want a test... (split from "Comodo vs AVC")

Re: Comodo vs AVC

I want a test where someone is pulled off of a street and put on a windows computer. They have 100.exe's, 50 malicious and 50 safe - they have no idea which is which. They need to install the safe applications without infecting the computer based only on the product and their own wits.

Let's see how well a HIPS does in that kind of test.

 

Re: Comodo vs AVC

@ Hungry Man

Well it sounds like you've just set yourself a challenge Go to Malware Domains etc & grab as many links with Malware as you can. Download & run them & see for yourself how your comp copes, or not Then post back, if it's still working that is & show us the results.

Yes, seriously why not you ?

A lot of us on here, including myself, have done so Many times over the years, with All sorts of nasties. Most of us are still here with no harm done. It'll be Very interesting to see how you do

 

Re: Comodo vs AVC

Well... that's not exactly my test. The knowledge that each of those links is malicious will change how I treat them, of course. The idea is that the user believes there to be an equal chance that the program is malicious or suspicious. They want to install 50 legit apps but there are also 50 malicious apps mixed in. That's the point - they don't know which is which. Me choosing them from a list of 100% malware defeats the purpose =p

And I'm currently running almost no security software (only EMET.) I wouldn't consider my setup very secure, certainly not against socially engineered malware.

But I'd love to see how a default deny setup like your own works when you don't know which is which.

 

Re: Comodo vs AVC

If i run anything new from some unknown or not so well known source, as often happens with for eg; ARK's etc & other Tools/Apps, or even known sources, i'll run them with ShadowDefender activated. That's for two reasons, 1st is just in case they "might" be dodgy, 2nd in case they are crap or don't work properly etc.

But my HIPS = ProcessGuard & Zemana will alert me to various things they want to do, such as install a Kernel Hook, Driver, Reg startup entry etc etc.

The thing you'll be most interested in, i presume is, If something needs a restart to install properly & therefore needs to be done Without SD, what then ?

I'll upload the file to VT etc & see what they say. If showing clean it's then a matter of going with my intuition. Up till now i'm unscathed from the Hundreds + such decisions i've made over the years. Apart from one about 6 years ago on 98SE when i didn't upload it & i got some Adware installed as well No big deal to get rid of, & yes it could have been worse, but lesson learned Even then i had an AntiExe etc.

 

Re: Comodo vs AVC

The idea is to up the ante a bit. 50% of the files we encounter aren't going to be malicious. The idea is to take a user and put them in a position where they have no clue what the origin of the files is but they know that half are 100% legit and half are 100% malicious.

It might be more fun to say "Install them all without infection" and not tell them how many are legit so as to not sway with statistical analysis.

I'd just be curious to see how certain setups would work in this situation. With an average user I'd say quite a few programs would flat out fail. With a wilders user who knows to use VT... maybe not.

 

Re: Comodo vs AVC

Non Wilders etc peeps, i expect would be more than likely screwed. Especially with the nastier stuff that gets released these days You only have to look at cleanup www's to see how MANY people even with fully patched Vista.W7 get owned

 

Re: Comodo vs AVC

I would think so.

 

I would do it if i still had my VM AND if my PC was more powerful
I stopped suing VM to play around with malware because i didn't have enough ram, although i added 2GB more like a month ago totaling 4GB as of now which would be somewhat enough.

Also since now i'm working + studying i dont have enough time xD

 

Re: Comodo vs AVC

With all due respect, I think the test would be meaningless.
A bare HIPS isn't suitable for a 'pulled-of-the-street' guy/gal.
How would average Joe know whether a 'dll abc requests access to xyz.whatever' message, means mayhem or not?
The HIPS would score as good as the guy/gal gambles, unless it also offers cloud reputation/community rating etc.

Heck, who wants to handle a bare HIPS install if you only have a neutrally named exe and no idea what kind of program it is?
A HIPS isn't, imo, something to ascertain whether something is malware or not.
If I'd get the message 'notepad.exe want to access the interwebs', I would have failed before.

 

If you're going to go this far, then why not go the distance and get some poor soul to use a clean pc and ask them to install the programs, both clean and infected, without trying to change the settings of the different security suites being tested.
It's true that many of the Wilder's members have more knowledge and skill when it comes to using security products. But we're preaching to the choir most of the time.
Let's see how really effective the security products are out of the box.
Hugger

 

It wouldn't be a purely HIPS test. It would be a test that any product can go into and see how a regular user would use their product.

I mean... if you have to be a security researcher or hwhatever for a setup to work it's probably not a great product, right? Or at least a very niche one...

 

Anybody with teenagers pulls this test daily.

 

In order for such a test to be useful you would need to consider the following:

1. The subjects shouldn't know what they are testing as that could skew the results and prevent them from responding naturally.

2. Other factors such as built in browser security should be considered.

3. Testing only exe's will only simulate an install-type scenario, so the subjects would need to browse through a series of controlled sites with exploits, etc.(or at least browse through a pre-determined set of bookmarks or favorites that contain real malware).

4. You would need several people testing each product in order to gather useful data.

I'm sure there are other factors to consider also. This could be a beneficial test for vendors if done correctly.

 

The hard part is choosing an "average" user and then having them go through hundreds of files with different products.

I would only test executable files and not exploits since:
1) Most exploits lead to a dropper file anyway
2) It's easy to tell an exploit page from a regular website
3) It's not so much about protecting against exploits as protecting against socially engineered malware - a situation where you don't know which files can be trusted.

 

In order for this test to work you would need to gather some avg. everyday people and have them browse the web through a specially designed web browser. This web browser would randomly have a "file would like to download and launch" alert which would be offering them either a good file or bad file. The files would be randomly named according to the site they are on. Such as if they are on Facebook a file Facebook.exe could be offered which could be a good program such as Skype or a bad program such as zbot. Then observe as the user answers the alerts and see what happens. A wide range of good programs would need to be offered to make sure not all of them are on the white list.

 

Adding the extra step of browsing wouldn't change anything though.

I mean if I'm going to go into details this is how I would do it:

1) Gather a large group of people, maybe filter out the people who have no idea how to open a file or people who are lead programmers or something.

2) Select 100 .exe's. Rename them to test 1, test 2, etc. Around half will be malicious, around half will be safe. Not necessarily a 50/50 split.

3) Have the product already installed with default settings. Maybe have it be a week old in terms of definitions.

4) Instruct the user that they can do anything to the software that they like except uninstall it (such as update it or change settings.) They can't install any other software except for what's in the test folder. The user can only run each .exe once.

5) Record the average infection rate. Record average number of valid programs installed.

A successful installation would mean that the program is able to open/ work properly. If it's installed but is broken it would not count.

 

Couldn't agree more. Let a few teens have access to a computer and see how these tests hold up.

 

Good to see this thread spilt off Quite why it's in here though ?

However, i doubt they would give truly realistic results, as they are not representative of All the other regular users. I can imagine what the score might be already with RU's = Not good

Making the OS etc a pain to use like Vista is one way, but even then it still gets blasted every day I don't see Any 100% solution to prevent RU's from getting infected, apart from user education. But MOST people i know, have known, hear/read about etc, can't be bothered etc to learn, sometimes even basics. So it's mainly their fault, & will continue to be.

 

Re: Comodo vs AVC

I strongly second that !

 

Re: Comodo vs AVC

Kind of a pointless test since more major vendors that have even half a brain are constantly crawling or receive feeds from malwaredomains, etc. so you are likely to see everyone detect 100% or close to it.

You want to really test malware, load up a P2P client like eMule etc., search for common search terms like serial, crack, xxx, video etc., download those exes and run them. Watch products like Trend and others that heavily depend on URLs, fall flat on their face.

 

Your idea,you do the test. I've played with malware for years,it's no fun anymore (boring). YOU wanna play the game,pony up and do it,otherwise hide behind the curtain.

 

I don't have the motivation, time, or resources to do these tests.

And maybe I'm just misreading but you seem a tidbit aggressive for no apparent reason.

 

You might think/hope so ! Unfortunately that isn't the case, at least not initially as they need to analyise them, & then provide DEF's for them. This can take days or even weeks

And what's more, you might be surprised to know that there are Thousands + of older Malware that still go undetected by Lots of vendors, even after being out there for years

I've tested hundreds of Malware from MDL etc etc, fresh & not so. Most AV's are "Late to the Plate"

 

Found some links that went undetected even after 8 years Yup,I lost faith in AV's a loooong time ago.

 

I have had the same experience. Many videos reviewing AM's will show similar results.