Eight Sandboxes Reviewed

[CogitoErgoSum]

I have posted a link to Tech Support Alerts "Eight Security Sandboxes Reviewed and Rated" article below. My only disappointment is that DefenseWall was not tested. If it had been tested it would have done well.

http://www.techsupportalert.com/security_virtualization.htm


Peace & Love,

CogitoErgoSum

[aigle]

I think u can put the DefenceWall near about GesWall. Both look similar.
Review did not take into account the point that some of these like GesWall don,t isolate the file system, rather just the registry, to maintain functionality while others like sandboxie isolate every file/ reg enteries. U can,t compare the two in the same way.

[Franklin]

Thanks for the link CogitoErgoSum.:)

Will admit I am a Sandboxie fanboy and it's great it did so well considering it's a 260 kb download and 900 kb installed.

Have been playing with SB for a while now and have set it's top level folder on another partition so C doesn't fragment as much.No slowdowns at all.

Just an observation I tried once:
Another quirk is that if you set the recycle bin on another partition as the top level folder it can't be seen by windows even though the properties show it is there.

You can still access the folder through Sandboxie.

Icesword can also see the folder.

[CogitoErgoSum]

Franklin, you are very welcome.


Peace & Love,

CogitoErgoSum

[Ilya Rabinovich]

I've red carefully all this "review".

1. Malware isolation. If there is no file system virtualization it doesn't mean that malware is not isolated from the trusted system. If you create new file- does it isolated?

2. VELite and Altris are virtualization tools, there is no sandbox-based restrictions. They doesn't have to be there.

3. "I suspect that malware was penetrating the sandbox by getting access to raw memory through buffer overflow". :wacko: No comments!

4. What does mean "raw memory access"? \Device\PhysicalMemory? Direct disk access?

5. If you terminate main GUI process of defense it doesn't mean defense is broken- good defense must be total driver-based (as my DefenseWall, for instance).

6. ShadowSurfer can be bypassed if malware loads driver, it doesn't protect from unauthorised driver installation.

7. What about other type of attacks? There are a lot of them!

My conclution- this review is an unprofessional :thumbd: .

[ShinyThings]

The link for the testing methodologies does not exist.

Altiris, VELite, and Virtual Sandbox would certainly pass the malware isolation test. That should have tipped him that something was not right with his methodologies for either testing or measuring.

How did GreenBorder do so well with its user mode only sandboxing. If you download any 16 bit DOS program or a Windows program that runs in NTVDM it will perform all operations outside of the GreenBorder sandbox.

Again, where is the testing methodology?

This test seems not very useful or meaningful.

[Franklin]

-{ Quote: "The link for the testing methodologies does not exist.

Again, where is the testing methodology?

This test seems not very useful or meaningful." }-

Geez,is there a testing where nobody complains.:)

Feathers get ruffled whenever anyone does objective testing.

That's because many forum users work on the basis that if they use a product it must be good and if anyone says otherwise they must be wrong.

Quote from the link:

"To answer these questions I used a number of different technical test procedures. Several of these were based upon the methodology devised by Michel Aparicio at his blog site: http://kareldjag.over-blog.com/10-category-69553.html Full details of the technical tests can be found below."

In this test the sandboxed PC was infected using a number of different methods.

The first (and perhaps most testing) infection method was to browse while sandboxed to a hostile "drive-by" web site.

The site I used, a Russian cracked software site, uses flaws in Windows and Internet Explorer to download malware without any user action or knowledge. Typical exploits include the well known iFrame and WMF exploits though the sites will repeatedly try a sequence of exploits if not initially successful. If finally successful, the sites download multiple malware products, often running into tens of megabytes.

[nadirah]

I can never ever trust these so-called "reviews" of any type of security software. The results are not accurate in any way at all.

[Devil's Advocate]

-{ Quote: "I've red carefully all this "review".

1. Malware isolation. If there is no file system virtualization it doesn't mean that malware is not isolated from the trusted system. If you create new file- does it isolated?
" }-

Good question. I was wondering that too. I don't think he was just looking at left over files, but whether they were still running after you cleared it. Otherwise I think GESwall would fail by default.



-{ Quote: "
2. VELite and Altris are virtualization tools, there is no sandbox-based restrictions. They doesn't have to be there.
" }-

he admited that for Altris already he still tested it. :)

-{ Quote: "
3. "I suspect that malware was penetrating the sandbox by getting access to raw memory through buffer overflow". :wacko: No comments!
" }-

LOL.

-{ Quote: "
5. If you terminate main GUI process of defense it doesn't mean defense is broken- good defense must be total driver-based (as my DefenseWall, for instance).
" }-

For what's it worth I think he recognise this. Sandboxie failed because the sandboxed programs could access the system, so it means the driver was defeated.

-{ Quote: "
6. ShadowSurfer can be bypassed if malware loads driver, it doesn't protect from unauthorised driver installation.
" }-

True, but again I don't know what Shadowsurfer is doing in this test. Heck he even admits it. It's a totally different product, even though we tend to put it in the same category.

-{ Quote: "
My conclution- this review is an unprofessional :thumbd: ." }-

Well I don't think he is a professional or claims to be. He's at about the level of the average-higher level wilders member I think (does he read here? It sure looks like it!). Somewhat knowledgable but no expert.

[Ilya Rabinovich]

-{ Quote: "
True, but again I don't know what Shadowsurfer is doing in this test. Heck he even admits it. It's a totally different product, even though we tend to put it in the same category." }-

In fact, SS is an advanced snapshot utility, not a sandbox.

-{ Quote: "
Well I don't think he is a professional or claims to be." }-

Sorry, you are wrong. http://www.techsupportalert.com/contact.htm. "As a computer professional"....

[Franklin]

Definately no expert here and from my laymans point of view I would just like to thank the author of those tests for taking the the time to do as such.:)

Whether nooby,amateur or professional hopefully we all have our objectives.

Keeping safe,learning and teaching!

[pykko]

nice link...my Sandboxie is excellent. :thumb: :-*

[stewieg]

I agree that these tests seem a bit suspicious ( whether intentional or not ). I've been using the Virtual Sandbox product for a few months now, and have not had any problems.

-{ Quote: "Franklin - The site I used, a Russian cracked software site, uses flaws in Windows and Internet Explorer to download malware without any user action or knowledge. Typical exploits include the well known iFrame and WMF exploits though the sites will repeatedly try a sequence of exploits if not initially successful. If finally successful, the sites download multiple malware products, often running into tens of megabytes." }-

I noticed that VS failed this first test, do you have the link to this Russion website? I ran similar tests before I started using VS in VMWare by going to similar sites (such as astalavista.com and going to serials.ws or whatever search engine and allow anything to be installed (activeX objects)). I would watch as the malware was being ran as well as downloading more malware to launch. When I cleared the sandbox, everything was removed. Any information would be greatly appreciated, thanks!

[aigle]

-{ Quote: "Good question. I was wondering that too. I don't think he was just looking at left over files, but whether they were still running after you cleared it. Otherwise I think GESwall would fail by default.
" }-

GesWall clears the registry but not the files. However the files remain isolated in any case I think.

[Perman]

Hi, folks: Does this mean that DeepFreeze is an advanced snapshot utility, in the same catagory as ShadowUser and ShadowSurfer? rather than a sandbox app?

[nicM]

-{ Quote: "Hi, folks: Does this mean that DeepFreeze is an advanced snapshot utility, in the same catagory as ShadowUser and ShadowSurfer? rather than a sandbox app?" }-

Hi,

Snapshots, I don't think, but for sure this is more something as recovery software : There's no distinguishing in sandboxing (eg. all is sandboxed), whereas sandbox software are meant to isolate some datas from the rest of the system.

nicM

[bellgamin]

Thanks for posting this excellent link, CES! The test info therein led me to initiate a trial of Sandboxie -- its good results & small footprint are 2 things I admire.

So far Sandboxie is cruising along splendidly. My only objection is that its system tray icon made me hungry because it rather resembles a slice of pepperoni pizza.;)

Are the tests perfect? Of course not. But I think that they are a lot better than basing my selection on nothing, or upon merely the subjective opinions of other folks.

[CogitoErgoSum]

bellgamin, thanks for the kudos. Just doing my part to bring attention to application sandboxes which I have generally found to be unobtrusive, effective and a complimentary part of a layered defense strategy.


Peace & Love,

CogitoErgoSum

[aigle]

-{ Quote: " application sandboxes which I have generally found to be unobtrusive, effective and a complimentary part of a layered defense strategy. Peace & Love,
CogitoErgoSum" }-

I totally agree.

[SirMalware]

Has anyone here ever had any experience with malware being able to leak outside the sandbox or be able to terminate Sandboxie itself? I suppose it's fair to say....it's coming.

[ErikAlbert]

-{ Quote: "Has anyone here ever had any experience with malware being able to leak outside the sandbox or be able to terminate Sandboxie itself? I suppose it's fair to say....it's coming." }-
In that case it will be fixed by the author of Sandboxie, just like Mozilla fixes Firefox constantly. It's a neverending story and is common for ALL softwares. :)

[trjam]

`````````````Greenborder is also nice. Ewido finds nothing because there isnt anything leftover to find.

[zopzop]

-{ Quote: "GesWall clears the registry but not the files. However the files remain isolated in any case I think." }-

yup the files remain but are "sterile" and can be deleted whenever you want. i've thrown lot's of horrible virri, trojans, and malware at geswall and it hasnt' let me down yet.

[aigle]

I like this approach as it gives a good balance of security and usability. I think both GeSwall and DefenceWall use this approach.

[BrianW]

Franklin, could you please clarify on the “first infection method”. You wrote:
-{ Quote: "The site I used, a Russian cracked software site, uses flaws in Windows and Internet Explorer to download malware without any user action or knowledge. Typical exploits include the well known iFrame and WMF exploits though the sites will repeatedly try a sequence of exploits if not initially successful. If finally successful, the sites download multiple malware products, often running into tens of megabytes." }-

So, your criterion is "the sites download multiple malware products", I assume it means you see some new files downloaded on your system and consider that as a successful attack, right? Please correct me if I’m wrong.

[aigle]

Hi Brian! Welcome to wilders.
What I should understand from ur nick? Am I guessing right?

Thanks.

[Seishin]

Yes this test proves one thing: That freebies are as good as commercial products. Pity that Sandboxie's last version didn't work well in my machine.

And usually the ones complaining are those who have shortcomings in their products, blaming the testers rather than themselves.

Cheers.

[TerryWood]

I have always had a sneaking respect for the TechSupportAlert Concept and newsletter. It has given me a starting point for selecting software.

I was therefor dismayed at the review that emerged for GesWall. I am a user of it but am not blinded by it. What irks me about the review is that the author chose a method which is not now reproducible.

So GesWall is panned, possibly with good reason, possibly not. Either way there has been a fair amount of desertion, from the program on the basis of this one report.

I don't hear anything from TechSupportAlert offering to redo the tests on a more reproducible basis so that the developers can modify their products?

Tests of this sort look dangerously amateur, do no favours to those who rely on them, in fact they don't help anyone including the author.

A CHALLENGE TO TECHSUPPORTALERT. What about a retest, with a bit more reproducibilty rather quick flash statistics for short term gain.

Terry

ps I am no inveterate supporter of GesWall if their product falls short, but I do like fair play

[trjam]

Go with Greenborder as it is the only one to pass all tests.

[TerryWood]

Why should I choose GreenBorder on the basis of these tests?

As I inferred lack of reproducibility invites lack of credibility!

Terry

[Devil's Advocate]

-{ Quote: "In fact, SS is an advanced snapshot utility, not a sandbox.
" }-

Did I say otherwise?

-{ Quote: "
Sorry, you are wrong. http://www.techsupportalert.com/contact.htm. "As a computer professional"...." }-

Well you can be a computer professional without being a computer security professional....

[TerryWood]

Wasn't contradicting you. Just having my ten pennyworth, because I feel so strongly about it.

Terry

(which I am entitled to do)

[trjam]

-{ Quote: "Why should I choose GreenBorder on the basis of these tests?

As I inferred lack of reproducibility invites lack of credibility!

Terry" }-
No, I just meant to say I have used it for awhile and really like it. It seems to have worked for me, but that is just one vote and I realize that.

[Kees1958]

Devil's Advogate

I was very polite to you when I added a few remarks to your 'professional' categorisation of HIPS.
http://www.wilderssecurity.com/showthread.php?t=152694

I think you think you are a more professional security expert than Brian (of GeSwall) or Ilya (of DefenseWall).

Keep on dreaming

:gack:

[JerryM]

It looks as if using a sandbox is complicated, and would slow down your surfing. Is that not true?

Jerry

[software-tester]

-{ Quote: "I have posted a link to Tech Support Alerts "Eight Security Sandboxes Reviewed and Rated" article below. My only disappointment is that DefenseWall was not tested. If it had been tested it would have done well.

http://www.techsupportalert.com/security_virtualization.htm


Peace & Love,

CogitoErgoSum" }-

Thanks for the link CogitoErgoSum.
I will download the apps I've not used before & compare. Thanks.

[bellgamin]

TSA's test of sandboxes is flawed from the standpoint that it discloses little if anything about test bed & testing methodology.

GesWall did splendidly in recent tests by the AV-Comparatives professional testing site. In case you've never visited AV-C...

+Main AV-C website (http://www.av-comparatives.org/)

+Click "Comparatives" button in left-hand column

+Scroll four-fifths of the way down the page until you see line entitled "Comparative of various protection tools October 2006"

+On that line, click "Report (PDF)"

>>>As you read AV-C's report, take note that test bed & methodology are fully disclosed.

[EASTER.2010]

Call me old hat or fashion but SHADOWSURFER does all the sandboxing needed for my researches. Also have SANDBOXIE in the cabinet if or when i decide to run inside it sometime again, but for now SHADOWSURFER pens down malicious files within it's security ring adequate enough as-is for now.

[tayres]

-{ Quote: "It looks as if using a sandbox is complicated, and would slow down your surfing. Is that not true?

Jerry" }-

Sandboxie, for example, slows down web surfing very little (if at all). It only requires 3MB memory, 5MB when browsing, and compared to much of the other security software discussed on these forums, you'll probably find it simple to use.

In terms of security, especially for those who aren't always careful about what email attachments they open or what web sites they visit, using sandbox software in a limited user account seems to me inherently safer than layering AV, AT, HIPS, etc. software in a normal administrator account.

[zopzop]

-{ Quote: "Call me old hat or fashion but SHADOWSURFER does all the sandboxing needed for my researches. Also have SANDBOXIE in the cabinet if or when i decide to run inside it sometime again, but for now SHADOWSURFER pens down malicious files within it's security ring adequate enough as-is for now." }-

easter have you tried shadowsurfer against killdisk (or other low level disk access virii)? a friend of mine had deepfreeze and it didn't work out too well, he wound up emailing pharonics with his results.

[Devil's Advocate]

-{ Quote: "It looks as if using a sandbox is complicated, and would slow down your surfing. Is that not true?

Jerry" }-

I don't notice any slow down while surfing, though I think (though I might be imagining it) a slight delay when opening browsers in sandboxes.

[aigle]

My exeperience--
GesWall- no significant slow down
DefenceWall -- no sign slow down( used very little0
Sandboxie_- no significant slow down
BufferZone-- significant slow down in browser launch and slight slow down in browsing
ShadowUser -- ?slight slow down

All is subjective though.

[JerryM]

Thanks for the replies. I think I will stick with what I have. I'm so safe that many would be bored following me.;D

Regards,
Jerry

[Kees1958]

-{ Quote: "TSA's test of sandboxes is flawed from the standpoint that it discloses little if anything about test bed & testing methodology.

GesWall did splendidly in recent tests by the AV-Comparatives professional testing site. In case you've never visited AV-C...

+Main AV-C website (http://www.av-comparatives.org/)

+Click "Comparatives" button in left-hand column

+Scroll four-fifths of the way down the page until you see line entitled "Comparative of various protection tools October 2006"

+On that line, click "Report (PDF)"

>>>As you read AV-C's report, take note that test bed & methodology are fully disclosed." }-

Bellgamin,

I have visited the site, but only for the AV-comparatives (how does Antivir stand out). My compliments for sharing this info

[Kees1958]

-{ Quote: "My exeperience--
GesWall- no significant slow down
DefenceWall -- no sign slow down( used very little0
Sandboxie_- no significant slow down
BufferZone-- significant slow down in browser launch and slight slow down in browsing
ShadowUser -- ?slight slow down

All is subjective though." }-

I did some script timing:
- fastest GeSWall (IE 7 beta)
- fraction slower DefenseWall (0.2 sec in IE7 RC startup)
- slower BufferZone (1.7 secs in IE7 RC startup)
- slowest Sandboxie (2.3 secs in IE7 beta startup)

Although the timing is slightly blurred by using different IE versions and due to difference in time, the state of the hard disk the first three observations are in line with the 'feel' experience of others.

Sandboxie stands out with all other contributors to this post as no significant slow down. I must have set up something wrong to get completely different results. According to others (Aigle, Devils's Advogate) I have to review my opinion of Sandboxie (regarding speed).

[aigle]

Nice work,I never measured so. Did u check the first launch of browser after booting or susequent launch,as first launch is always slower than sunsequent launch.

[Kees1958]

First launch only, subsequent launches were about 2 to 3 secs faster

[aigle]

So u mean for all sandboxes you rebooted ur PC each time?

[Kees1958]

Yes, and your question. . .

just made me realise why Sandboxie performed worse in my test.

At home we have three PC's:
Wife (has an MBA, but is a computer novice): Antivir + CB + DefenseWall she uses IE (Dep enabled in XP),
Son (a script kiddy, builds websites at school, gamer): Antivir + SSM + BufferZone 4 FireFox only when surfing tacky sites in search of scripts to use (+DEP)
Own (Antivir + SSM + GeSWall (+ DEP).

I tried some virtu/sandbox programs on my own PC for my wife. Because she downloads a lot of paid music (from internet) and some free with LimeWire, therefore sandboxie really was not option.

So I had fiddled a lot with GeSWall, DefenseWall and BufferZone. Then I gave Sandboxie also a try. What I did not think of is, that you have to boot a few times to get Sandboxie in the windows prefetch. This might be the reason why sandboxie performed worse on my PC.

Apologies SandBoxie fans:gack:

Regards Kees

[aigle]

To me I will first run any browser few times in one specific sandbox and then will note the timimg.
For calculating first browser launch timr u have to reboot ur PC for every sandbox one by one and it will take too much time.
I just rely on my subjective feeling and indeed I am very sensitive in this regard.

[aigle]

-{ Quote: "I did some script timing:
- fastest GeSWall (IE 7 beta)
- fraction slower DefenseWall (0.2 sec in IE7 RC startup)
- slower BufferZone (1.7 secs in IE7 RC startup)
- slowest Sandboxie (2.3 secs in IE7 beta startup)

Although the timing is slightly blurred by using different IE versions and due to difference in time, the state of the hard disk the first three observations are in line with the 'feel' experience of others.

Sandboxie stands out with all other contributors to this post as no significant slow down. I must have set up something wrong to get completely different results. According to others (Aigle, Devils's Advogate) I have to review my opinion of Sandboxie (regarding speed)." }-

Ok, I have done some testing as well. I compared the launch of browsers without any sandbox and within sandbox. In each case I launched browsers ouside and within the sandbox multiple times before counting the time to let windows make prefetch data. I recoreded the timings with a stopwatch.
And took an estimated mean value of few readings. Here are my findings9 on same system XP Home SP2).

Time taken by browsers to launch without any sandbox[U]
IE 6 ------------------------- 0.8 sec
Opera 9 --------------------- 1.4 sec
FF 8.6( with no extensions) --- 1.1 sec

With GesWall 2.5.0 beta
Almost same timings or just a difference of 0.1 sec

With Bufferzone
IE ---------1.6 Sec
Opera -----3.6 Sec
FF------------ 2.5 Sec

With Sandboxie current version
IE ----------1.2
FF ----------1.7
Opera( could not run- freezed, as always) :(

[Kees1958]

Bufferzone 1.6 secs slower on IE6 than GeSwall, that pretty much is the same as my recording with IEF (1.7 secs slower).

This proofs what I allready said: I did not allow Sandboxie to be prefetched.

Good test:thumb:

[ccsito]

-{ Quote: "Ok, I have done some testing as well. I compared the launch of browsers without any sandbox and within sandbox. In each case I launched browsers ouside and within the sandbox multiple times before counting the time to let windows make prefetch data. I recoreded the timings with a stopwatch.
And took an estimated mean value of few readings. Here are my findings9 on same system XP Home SP2).

Time taken by browsers to launch without any sandbox[U]
IE 6 -------------------------8 sec
Opera 9 ---------------------1.4 sec
FF 8.6( with no extensions) --- 1.1 sec

With GesWall 2.5.0 beta
Almost same timings or just a difference of 0.1 sec

With Bufferzone
IE ---------1.6 Sec
Opera -----3.6 Sec
FF------------ 2.5 Sec

With Sandboxie current version
IE ----------1.2
FF ----------1.7
Opera( could not run- freezed, as always) :(" }-

8 seconds to launch IE 6 without a sandbox application? Is this right? My system was not that slow in opening the IE browser window. That is almost an eternity. ;D

[aigle]

-{ Quote: "8 seconds to launch IE 6 without a sandbox application? Is this right? My system was not that slow in opening the IE browser window. That is almost an eternity. ;D" }-
Ah, sorry it is 0.8 second. I will correct it.
Thanks for correction.

[nadirah]

-{ Quote: "I've red carefully all this "review".

1. Malware isolation. If there is no file system virtualization it doesn't mean that malware is not isolated from the trusted system. If you create new file- does it isolated?

2. VELite and Altris are virtualization tools, there is no sandbox-based restrictions. They doesn't have to be there.

3. "I suspect that malware was penetrating the sandbox by getting access to raw memory through buffer overflow". :wacko: No comments!

4. What does mean "raw memory access"? \Device\PhysicalMemory? Direct disk access?

5. If you terminate main GUI process of defense it doesn't mean defense is broken- good defense must be total driver-based (as my DefenseWall, for instance).

6. ShadowSurfer can be bypassed if malware loads driver, it doesn't protect from unauthorised driver installation.

7. What about other type of attacks? There are a lot of them!

My conclution- this review is an unprofessional :thumbd: ." }-

I would agree! I never trust reviews. ;)