Should we do a security competition?

[Hungry Man]

I see a lot of tests get scrutinized here and I see a lot of people putting their own security setups to the test.

I've never seen a proper test of combined antimalware. I think it would be interesting to look at the entire "suites" that people have put together.

Here are my ideas on how it would be:

The way I figure it it would be:
Each person creates and submits their own setup

They then film (with some screen capture software) their setup in action with predetermined malware. They also film setting up the computer and include relevant info.

They then submit it and it's judged by some group of non-participants.

Everyone uses the same malware... but they'd have 24 hours (after the malware list is released) to create their videos.

We also have a clean computer that we run the malware on as a control... for XP 32bit, Vista/732bit and 64bit under the same conditions.

The malware would have to be looked at first to make sure it also fits in with criteria.

Winner gets bragging rights =p

I personally think Windows, Linux, and OSX should be in separate categories OR we should simply have this as a Windows competition.

Any ideas? I figure with the cumulative knowledge on here we can come up with a fairly good test.

We would also need some impartial judges with moderate/ very high computer knowledge.

[Hungry Man]

I think we would be judging in categories, not just by "who is protected the most."

1. Protection against malware
2. Ease of use/ maintenance
3. Ease/ Time to set up
4. System effect/ "weight" on your resources

Can't think of anything else to judge it on. But each setup would get points in the other categories and be the winner of the category and then one setup would be the overall winner.

[SweX]

And all the videos should be uploaded to the same site, like YouTube or similar?

I'm not going to participate I am just wondering

[Hungry Man]

Well it wouldn't really matter. But youtube would probably be the easiest. If we did youtube we could have a public vote on it.

[SweX]

Originally Posted by Hungry Man

Well it wouldn't really matter. But youtube would probably be the easiest. If we did youtube we could have a public vote on it.

That's what I meant, that it would be much easier for everyone who would like to watch them to have all videos uploaded in one place instead of several sites.

[Hungry Man]

Well if people on here wanted to have the competition there would be a topic in which we got it all set up/ the vids would be posted.

But youtube would probably make thigns easier.

[Page42]

Hungry Man,

You've hit upon a very interesting idea here, imo.
From my perspective, I would be willing to put my setup to the test... I would welcome the results... not due to arrogance, but to curiosity.

Problem is for me, twofold (at least).
One, the time involved and
two, the expertise.
By expertise I mean setting up my machine and whatever else it takes to pull it off.

I humbly offer myself as a judge, however, based upon my qualifications on the lower end of your prescribed scale...

We would also need some impartial judges with moderate/ very high computer knowledge.

[Hungry Man]

I guess judges should be nominated/ elected by other users. I really don't know lol not that far ahead. I just want to see if people are interested in the general idea first.

[Page42]

I know. I was just going for the laugh.
I hope your idea can get some legs, though.
It would be neat.

[Hungry Man]

=p I think it would be a fun way to see which security setups are best for which reasons.

[noone_particular]

With a default-deny security policy in place, there wouldn't be much to see. Just an "access denied" message.

[Hungry Man]

Default-deny would give you a great score in terms of protection but not necessarily for ease of use.

[noone_particular]

For "ease of use", normal usage will need to be defined. In daily usage, it doesn't behave any different than another setup. It's only when you try to do something that changes the system that the policy becomes inconvenient. Mine would lose out under "ease/time to set up". The initial setup was time consuming, but after that, it's done until you change or update something, which I seldom do.

[Hungry Man]

Ease of use would have to be "benchmarked" in a certain way. Like running some random portable application, which is unlikely to be whitelisted. Installing a well known piece of software. Simple opening of browser/ word document.

At this point I'm just curious to see whether or not people actually want to do this

[Konata Izumi]

I'd suggest there are two category:

one for FREE setup
one for PAID setup

[chris1341]

Interesting, lots here test their set-ups against malware every day so I'm sure more than a few would upload results but to what end? The endless debates here would more likely be inflamed than resolved I would imagine.

The 'judges' would have to have criteria. 'What is an infection' (what constitutes malware even!) could cause more debate than the tests are worth. For example is a machine running an active malware sample under Defensewall protection, in the Comodo/Avast/KIS sandbox or otherwise restricted actually infected? Different views on that I'd imagine.

Similarly, what about traces left behind, are they infections? Who has the expertise/authority to establish what is/is not malicious detritus from malware? Who decides a) what the malware is designed to do and b) whether it achieved that?

It is fairly straightforward to run some samples and if they are blocked give a pass and if not a fail but simply allowing malware to run or the presence of excutables or leftovers on your system does not automatically mean infection.

If things like that are resolved it could work. Interesting what the mods would think though.

Cheers

[CogitoTesting]

@ Hungryman

Sorry for my late reply, I've been busy. If life allows me some time I could upload my own results on youtube for everybody to view soon.

Thanks.

[Hungry Man]

Originally Posted by Konata Izumi

I'd suggest there are two category:

one for FREE setup
one for PAID setup

I agree.

Originally Posted by chris1341

Interesting, lots here test their set-ups against malware every day so I'm sure more than a few would upload results but to what end? The endless debates here would more likely be inflamed than resolved I would imagine.

The 'judges' would have to have criteria. 'What is an infection' (what constitutes malware even!) could cause more debate than the tests are worth. For example is a machine running an active malware sample under Defensewall protection, in the Comodo/Avast/KIS sandbox or otherwise restricted actually infected? Different views on that I'd imagine.

Similarly, what about traces left behind, are they infections? Who has the expertise/authority to establish what is/is not malicious detritus from malware? Who decides a) what the malware is designed to do and b) whether it achieved that?

It is fairly straightforward to run some samples and if they are blocked give a pass and if not a fail but simply allowing malware to run or the presence of excutables or leftovers on your system does not automatically mean infection.

If things like that are resolved it could work. Interesting what the mods would think though.

Cheers

These are exactly the questions I want to put to Wilders members. We need clear cut rules and definitions for tests.

But all of that can come after we actually decide on whether this is happening or not, and then we can find judges who everyone agrees are competent and well established users.

Originally Posted by CogitoTesting

@ Hungryman

Sorry for my late reply, I've been busy. If life allows me some time I could upload my own results on youtube for everybody to view soon.

Thanks.

Thanks for the support.

I know I personally test my setup. I believe others do as well. All this would be is testing your setup but with predetermined malware and while running a screen recorder.

[Dark Shadow]

I think it's a great idea,but to be fair I think the test should be seperated.Example, how could one compete using standalone AV vs another using hips or sandboxing. just some food for thought.

[Hungry Man]

Well the idea is to compare whole setups. If someone is only using a single antivirus... that's all there is to it, their scores will be determined by how well that single AV works.

[tipo]

so you do your test with your av today on 10 viruses and i will do mine 10 days from now (`cause i don`t have the time to do it right now). your av will catch, let`s say 7 viruses, and my av will catch them all, `cause in this 10 days that passed the virus database of my av will grow and this viruses will be in the definitions....is this a fair test?? o.O
EDIT: the viruses must be the same, right?

[Hungry Man]

All tests have to be recorded and submitted within 24 hours of the released malware list, which will be PM'd to all listed participants. This will negate, at least partially, the fact that an AV will catch more malware a week later than it did a week prior.

Yes, everyone uses the same malware/ exploits.

[CogitoTesting]

@ Hungryman

If I have to do my test let say this coming Saurday evening or early Sunday morning when should I expect my own MDL list since these malware lists are being constantly updated?

Thanks.

[Dark Shadow]

Originally Posted by Hungry Man

Well the idea is to compare whole setups. If someone is only using a single antivirus... that's all there is to it, their scores will be determined by how well that single AV works.

got it,thats cool.who going first.

[Hungry Man]

No idea yet. I still want to make sure the mods/ admins are ok with it so we don't get shut down halfway through and I'd like to see more users supporting the idea.