Are sites being blacklisted when using JavaScript?

If you can visit your own site, then it can be visited from anywhere in the world, right?

Not if it is blacklisted somewhere. That may happen even to an innocent site. Here are three recent cases.

It started when a visitor from a different continent mailed me and said my site could no longer be viewed from there (Oops! Google Chrome cannot connect, and so on). I thought the site was alive and well - but it was on a blacklist in Germany.

A month later, my webhost suspended my account because of another blacklisting. My site, which uses JavaScript, was down for three days.

Both of these two cases were resolved after some effort.

The third case is not resolved. I find it ominous:

A download site, where people get a well-known JavaScript graphics library, got blacklisted at the same time as my site.

The blacklisting service identifies the library's reference manual as the offending part.

The author of that download site died three years ago. His family and friends removed contact address and donation links, and announced that the site would from then on be static. The site has not changed since that time, as far as I can see.

There is no malware in the page with the reference manual or in the JavaScript source text. I know, because I have used the library for years and know the details fairly well. Everything is hand-coded. There is no place where a hacker could hide anything. (I have inspected it carefully. My own site is clean, too; it is simple, and easy to check for unauthorized changes. My site uses the graphics library.)

Three false positives during the past few months. That's why I have to ask these questions:

Have other users of this forum experienced similar blacklisting recently?

Has anybody noticed signs of some poorly written anti-malware scanner crawling around, reporting false positives for a large number of JavaScript-based sites?

Can anyone suggest a method to avoid being blacklisted and still use JavaScript on one's site?

If not, my feeling is that we can expect more such cases - perhaps many.


EDIT:
Forum users who would like to know if their site's IP address is blacklisted somewhere in the world may use tools such as
www.blacklistalert.org
www.mxtoolbox.com
and several others.

 

a large majority of sites use java scripts.
so java scripts themselves are not the problem.

the site could have been reported as malicious by some users.
who knows...

 

no, but I've noticed signs of some poorly written websites around, getting owned by a large number of hackers.

I suggest you seek out facts. Go to the National Vunerability Database website, type in the name of the javascript graphics library that is the subject of these three incidents (you didn't give the name otherwise I would have done this for you), and find out if a vulnerability was discovered in it. If there's a vulnerability, post back and maybe we can figure out whether these three sites were compromised using the vulnerability, or if it was a false positive like you said.

Of course that's assuming that it's the one javascript library that's the issue. Perhaps these websites were compromised by some means other than this library, but through web app vulnerabilities. I would encourage you to focus your efforts on your own website & the security of your server. A simple way to keep your code from sucking is to look at the OWASP top ten web app cheat sheets. If you're hosting your own web server then you need to look for security configuration guides written for whatever server you have installed.

 

Such a "Hostile Visitor" hypothesis is indeed consistent with the facts.

The same user should then have blacklisted both the download site and my site (which contains links to the download site) simultaneously or almost simultaneously. An owner of some competing graphics library could have felt motivated to do so.

- Possible in principle. In practice, I think it sounds farfetched.

This sceptical response is useful, because I always wanted an insightful person to scrutinize my internet security precautions.

A moderator once advised against publishing too many details about that subject in a public forum. That's one reason why I did not reveal the name of the graphics library. The other reason is that it would be appropriate for me to contact the library's owner first.

But here are some details. (Visitors who find them boring may skip to the next post.)

First of all, the National Vulnerability site says that both the download site and my own site are clean. And so says safeweb.norton.com.

The OWASP Top-Ten Cheat Sheet (which is actually a top-sixteen - it took a while to examine them all) recommends a password at least eight characters long. The password for my site's control panel is much longer than that. It is (like the password I use for homebanking) never entered into any computer that is used for surfing.

Many of the Top-Ten tell about server-side scripting (php, etc.). My site has no server-side scripting and no .htaccess file.

All HTML and JavaScript on my site is hand-coded by myself. Any unauthorized change in any file would be conspicuous. It would actually disappear when the file is changed, for I manage my site in an "upload-only, never download" manner. The site is a copy of a master folder on my computer, not vice-versa. No folder on the site contains any files that I did not put there myself. No file has had its content changed except by myself.

Adobe Flash Player, Adobe Reader, Java, and Libre Office on my computers are updated immediately when my ISP tells to do so in their security bulletin, or when I get aware of the need by other means.

Adobe Flash Player is updated manually (separately in IE and Firefox - I'm sure many people do not know this has to be done!). Adobe's automatic update invariably fails, because I don't use admin rights except when needed.

My two computers use Windows XP SP3, because they are old and do not have enough memory for newer versions. They are not connected in a network. Files are occasionally moved from one computer to the other on a USB key or an external disk. No suspicious files have ever been seen on any of these devices. The computers are switched off when not used, and so is the "TDC Homebox" wireless router that my ISP gave me.

That router is hard to penetrate from the outside - or so my ISP says. That's why I use Windows Firewall. My antivirus program is Avast Free. Boot scans are clean.

Sure there is security software around with higher detection rates for historical vira, and better heuristics. But I always think of a hacker workshop as a place with several computers side by side, one for each antivirus program, and a team of hackers fine-tuning their malware until it is invisible to them all. So I believe in keeping programs updated so that vulnerabilities can get fixed immediately.

The JavaScript eval function is used for few "edit + execute" demos on my site. These will reject any text that does not pass a highly intolerant client-side anti-XSS filter. OWASP says input validation is usually based on regular expressions. So is my filter - but I do not feel really competent about how to implement safe eval calls.

I use a free hosting service. They are hosting more than a million sites. Ads are not mandatory, but my site has some.

I believe my site has not been compromised, and none of my computers are infected.


Do you see too many weak points? Or have I missed something? I would like to hear!

 

It sounds like you searched for the name of the websites in the vulnerability database site. What you should search is the name of the javascript graphics library. (if you did that- nevermind then)

The weak point IMO is exactly what you stated- that you're not sure how to implement safe eval calls. I'm not a developer so I can't help you there. But it's my understanding that the OWASP cheat sheets will give you some guidance on that. Each individual cheat sheet has real-world code examples to guide you. Hope that helps.

 

I just searched for the name of the graphics library, with and without postfix and prefix. The Vulnerability Database says there are no matching records.

A search for the name of the library's author gives one matching record. It says an unspecified vulnerability was present in an obsolete version of a so-called tooltip library that was used in an obsolete version of something called eGroupWare. Our download site contains the most recent version of that tooltip. The text says an old version is available too, but the link gives a "no longer supported" message and leads back to the most recent version. There might be a remote possibility that they could have forgotten to remove the old zip file from the server. I do not know how security software would react to a zip file that exists on a server without being linked to.

The graphics library contains a few JavaScript statements of this kind:

Code:

function _mkDiv(x, y, w, h) {
	this.htm += '<div style="position:absolute;'+
		'left:' + x + 'px;'+
		'top:' + y + 'px;'+
		'width:' + w + 'px;'+
		'height:' + h + 'px;'+
		'clip:rect(0,'+w+'px,'+h+'px,0);'+
		'background-color:' + this.color +
		(!jg_moz? ';overflow:hidden' : '')+
		';"><\/div>';
}

It resembles one of the "Dangerous HTML Methods" listed in the Top Ten's DOM based XSS Prevention examples.
Could that cause blacklisting? I will study those OWASP documents a little more.

As to eval, I recall having seen these real-world XSS examples or similar examples before. They tend to contain ampersands or percentage signs. My filter gives an error message, and does not forward the text to eval, if the text contains ampersands or percent signs.

Or a lot of other things. My filter contains about 40 regular expressions. I am aware of a couple that ought to be added. The filter is an modified version of one I found at exploit.blogosfere.it. I just tried one of the XSS Prevention examples in one of the eval textareas at my site. It was rejected, as it should.

 

I wish I knew javascript enough to answer that, sorry. I'm sure there are javascript forums that could answer it.

 

We have enough facts to summarise a little.

Hundreds af people have visited this thread by now. Many more have seen its title without clicking on it. Not a single visitor has reported JavaScript-based sites being blacklisted.

So, my hypothesis about a poorly written security scanner crawling around and blacklisting in East and West can be ruled out.

My usage of the JavaScript eval function cannot explain why the download site for the graphics library got blacklisted at the same time as my site.

The "Hostile Visitor" hypothesis, suggested by moontan, can. But only if such a visitor exists.

And so can the "Vulnerable Library" hypothesis suggested by BrandiCandi, as both of the two sites use the library.

But the graphics library does not contain malware and cannot be vulnerable. It consists of a relatively straightforward JavaScript source text. Only the function declaration shown in the Code above, and a limited number of similar statements, go beyond an elementary level.

The function declaration in the Code contains a string literal with HTML and markup. OWASP says sites should not accept a string of that kind uncritically if a customer enters it into a form. This is - if I understand things correctly - because strings with HTML and markup are sometimes used in cross site scripting (XSS) attacks.

Are security software developers simply unaware that such strings are also used in JavaScript graphics libraries? The string in the Code is used merely to create a div element as part of some graphics.

Could a security software developer, if such a person happens to see this thread, please tell us:

Can a string such as the one in the above Code produce a false positive?
How should the function be rewritten in order not to produce a false positive?

I would like to know!

 

that would sound logical...

 

Found something:

Deep in the graphics library's source text, there is a string expression with two percent signs in it.

The string is later edited with a regular expression that fills HTML code into it. One HTML code if the user prefers quick graphics, another for printable graphics.

Testoutput has shown that the edited string looks pretty much like the one in the Code example in a previous post.

The edited HTML string is inserted into the document when the user calls a paint() method. A batch of graphics then appears on the screen. Elegant solution, devised at a time in the past when the concept of DOM was not available or did not work very well!

And no problem at all for security software. - Or what?

If the relevant JavaScript statements are extracted from various places in the library and put together (Code example below), it begins to look like a case study in the art of writing security software.

This is not malware...

Code:

function _mkDivIe(x, y, w, h) {
    this.htm += '%%'+this.color+';'+x+';'+y+';'+w+';'+h+';';
}

var _regex =  /%%([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);/g;

function _htmRpc() {
    return this.htm.replace( _regex,
        '<div style="overflow:hidden;position:absolute;background-color:'+
          '$1;left:$2px;top:$3px;width:$4px;height:$5px"></div>\n');
}

function _htmPrtRpc() {
    return this.htm.replace( _regex,
        '<div style="overflow:hidden;position:absolute;background-color:'+
          '$1;left:$2px;top:$3px;width:$4px;height:$5px;'+
            'border-left:$4px solid $1"></div>\n');
}

this.setPrintable = function(arg) {
    this.printable = arg;
    if(jg_fast) {
        this._mkDiv = _mkDivIe;
        this._htmRpc = arg? _htmPrtRpc : _htmRpc;
    }
    else this._mkDiv = arg? _mkDivPrt : _mkDiv;
};

function _pntCnvDom() {
    var x = this.wnd.document.createRange();
    x.setStartBefore(this.cnv);
    x = x.createContextualFragment(jg_fast? this._htmRpc() : this.htm);
    if(this.cnv) this.cnv.appendChild(x); // (for access to divs afterwards)
    this.htm = "";
}

function _pntCnvIe() { // (painting on canvas; jg_fast is true in IE)
    if(this.cnv) this.cnv.insertAdjacentHTML("BeforeEnd",
      jg_fast? this._htmRpc() : this.htm);
    this.htm = "";
}

    this.paint = jg_dom? _pntCnvDom : _pntCnvIe;

Security software should be able to tell reliably whether or not this code is an attempt to obfuscate malicious material.

Writing such software should be easy for competent developers. Or is it too difficult?

Is it possible that modern security software could produce a false positive in the above Code?

If the library is a problem, why is it that its download site was blacklisted only three weeks ago and not three years ago?

I hope someone can tell!

 

To have my site taken down again is not what I need, so I decided to take action.

All JavaScript strings with HTML tags were removed from my site the other day. They were also removed from the copy of the graphics library that resides on my site. That means the site no longer uses the official version of the library.

Methods and properties of the DOM are now used instead, in a straightforward manner.

As the one single exception, the modified library still uses the empty string as a quick way to clear the graphics:

Code:

this.cnv.innerHTML = "";

I would not expect such a statement to be confused with obfuscated malware, even in poorly written security software.

No visitor has confirmed the hypothesis that it must have been the usage of HTML code in JavaScript strings that caused my site to be blacklisted and taken down. No one has responded to my question with a post such as

"Yes, the software we sell here at Type-1 Errors, Inc., produces false positives in your code examples."

But the absence of such replies can be explained by an impression I got while browsing through some academic-looking papers from the security software industry: Developers tend to boast of their false positives when the rate is low, but not when the rate is high.

The download site from where I got the library was blacklisted on the same date and at the sime time as my site. Some crawling scanner must have blacklisted my site, then followed my link to the download site, found the same JavaScript code there, and blacklisted that site too.

Incidentally, Google found a log saying that VirusTotal scanned my site at the time my webhost took my site down. The log says 42 out of 42 different malware detecting services found my site clean. I had not yet discovered that log at the time when BrandiCandi commented on my internet security policy.

The "Hostile Visitor" hypothesis suggested by Moontan is still a remote possibility.

It would be good to know if JavaScript strings with HTML tags have suddenly become problematic if they are inserted into the document. That question is probably best asked in a new thread with a more specific title.

But responses here are still welcome.

And to the nearly 500 visitors to this thread so far: Thanks for your patience!

 

Just to be thorough, are you absolutely certain that the reason you're blacklisted is because of a javascript? Or are you inferring it based on other evidence? You know what they say about the word "assume"...

Perhaps the best way to resolve this is to contact the developer/ administrator of the site that blacklisted you. Find out EXACTLY what got you blacklisted. Maybe they would even have potential solutions for you. I'm sure there's some kind of mechanism for blacklisted sites to get un-blacklisted. I just don't have any experience with them.

 

No, not absolutely certain, and that's the reason why I would like to know more about how security scanners react to JavaScript strings with HTML tags in them.

Both were contacted immediately. The first blacklisting service (which listed my own site only) removed my site's IP address from their list automatically via a request-removal link. The second blacklisting, which appeared a month later (both the download site and my own site), was removed after I contacted both the blacklisting service and my webhost by mail.

The first blacklisting service said my site's IP address had been "reported by a user as a spam source".

The second blacklisting service did not provide details. But they identified the "offending" part of my site as a page that describes the graphics library, tells visitors where to get it, and lets visitors try two "edit + execute" demos.

One of those demos contained so-called scripted redirection,

Code:

location.href = newURL;

I thought at the time that this line of code was the reason for the blacklisting, because I had been told that an old version of Internet Explorer refused to execute such statements for security reasons, and because academic literature has it that scripted redirection, which is used in botnet requiters, poses a hard problem for security software. So I removed that demo before I mailed the unlist requests, and explained in the mails that I had done so.

But scripted redirection is not used in the graphics library's user reference manual at the download site - so there must be another source of false positives, and it must have been present both at the download site and at my own site.

My "offending" page contains a srs to a copy of the library's source text on the server.

And so does the download site's page with the user reference manual. That is the only thing the two sites have in common.

The library's JavaScript code looks elementary, except for some strings with HTML-tags in them. My two Code examples are the two worst-looking cases.

No evidence other than simultaneous blacklisting of two sites, with the "offending" parts being identified as two pages, both of which use the same JavaScript library in the same way. The two sites were blacklisted by the same blacklisting service. That service appears to have my webhost as a customer.
Re "assume": The true cause might be something entirely different. I just can't figure out what it could be!

My two mails in response to the second blacklisting were brief but detailed, and held in a polite tone. The suspension was lifted, but they did not reply to my mails. I'm not sure it would be wise to contact them again - they, too, have their reasons for not revealing details, and they might take me for a hacker who wants tips and tricks. (It is not clear if the second blacklisting service scanned on its own initiative or received reports from a third part.)


And now Moontan's "Hostile Visitor" hypothesis re-visited. Here is my own version:

The graphics library moved to its present download site a year or two ago. Before then, people downloaded the library from a different site.

That old site became useless some time after the library's author died, because the domain name got a new owner suddenly and without explanation. There exists a very simple site at the old domain name. It contains some ads. Nothing can be downloaded from it. The new owner is easily identified. He owns many domain names, and obviously hopes to sell them at a profit.

I have advised people to visit the library's new official download site, and not the site he has created.

He may have felt I hurt his business, and then reported both the download site and my site as offensive.

Sounds as pure speculation, doesn't it? I think a professional domain hijacker would spend his time identifying new targets, and not bother about the past.

But the idea cannot be ruled out on basis of the facts as I know them until now. That's a third reason why I prefer not to feed too many details into the search engines.

So - "absolutely certain?" Make that "almost certain".