ERAS http update service failing

We have recently rolled out ERAS 3.1.11 at my company. I have ERAS set to be an update mirror for our NOD32v4 clients, and client updates are happening via the http update service on the ERAS.

During our initial testing phase, things worked great. Our 10 test clients were able to connect to the ERAS just fine, and get hourly updates without a hitch. Last week we started rolling out NOD32 to our entire company (about 400 machines), and we soon started having problems with the http update service on the server becoming unresponsive. Some clients are able to get updates, while others fail.

Using a netstat command on the ERA server, I see that it is indeed listening on port 2221 (the update port), and I also see many connections to that port from various clients. However, most of those connections are in the CLOSE_WAIT state, meaning the client has closed the connection, but the server never closed the socket. Also, each client connection that's stuck in that state has *three* connections in that state. Any clients with three stuck connections cannot connect any more.

I tested this by using a telnet command on various clients to test connectivity to the server. Clients with no stuck connections can connect the server on port 2221 just fine, while clients with those three stuck connections cannot connect again. This suggests that the ERAS http service only allows 3 connections per IP.

The real question, though, is *why* are those connections not being fully closed? It's really wreaking havoc in our network as many clients aren't getting timely updates, not to mention the errors it creates.

I did some digging around the internet and this forum already and found a thread started in late 2007 discussing this exact same problem, but that was regarding ERAS 2.x. There was never a fix to the problem in that thread, although one user did suggest a workaround (which I'm currently doing). I'm very disappointed to see that this problem still exists even in the latest version of ERAS (3.1.11).

Has anyone seen this, and has anyone found a fix for it? The workaround I'm doing is simply using an IIS web server to host the updates rather than the ERAS http service.
-joe

 

I had a different issue with the internal HTTP host going nutty and leaving connections open, chewing up resources. Frankly, I just don't think it is designed for high-load situations and IIS does the job just fine. My initial deployment was around 400 workstations and that seemed to work, but after around the 500 marks the weird things set in and I was forced to move.

 

We started hitting the issue with only about 100 clients. As we installed more and more, the problem just got worse and worse.

But you're right, the IIS workaround is working great, so for now I'm inclined just to stick with it. Plus, by using a real web server for this purpose, I have increased contol over things like logging and connectivity.
-joe

 

Its possible to tell us how use IIS http server with ERAs ?

Thanks

 

This KB steps you through it: http://kb.eset.com/esetkb/index?pag...earch&viewlocale=en_US&searchid=1254315916839

 

Thanks for the reply.

Do you think it's possible do to the same thing with a 2008 Server ?

 

I haven't played around much with IIS7, but the settings and steps should be fairly easy translate in to the new version. Its not like the mirror is doing anything more complex than responding to GET requests and authentication validation if you have that set up.

 

Ok Thanks.

Now I have 230 clients Nod32 and more in the future. I prefer to anticipate because I think the problem will also occur. Bad point for ESET...

 

Yes, I'm running it on a 2008 server with IIS7. Works perfectly. The ESET guide linked by Smacky has all the steps you need. The options and settings will just be in slightly different places on IIS7 since it has a new, redesigned interface, while the ESET guide references the old IIS6 interface (from Windows 2003). Same idea, either way. You might just have to poke around a bit in the IIS7 interface to find what you need, but it's not hard once you get the hang of it.

Also, you could you use *any* web server, such as Apache, etc. The basic idea is that you set up a website that points at the mirror folder created by ESET's update mirror feature. That's really it. Like Smacky suggested up above, the whole duty of the web server is very simple. Clients connect to the web service, ask for the 'update.ver' file, and that file tells the clients what the current virus definition versions are and what files are available. If the client is out-of-date, it downloads the latest file listed in that update.ver list. If it's up-to-date, it just disconnects. Nothing special happens, and as far as the web server is concerned; it's just serving up simple, static content.

This is why I'm so confused as to why the ESET http service can't handle it. I had a trouble ticket opened with ESET on this issue, and after going back and forth a bit they finally said I need to run the http service on a separate machine, as the RAS plus http service together are too much traffic for one machine. This doesn't appear to be the case, as I'm now running the RAS plus IIS just fine with no problems. Ultimately, it seems to come down to the ESET http service not closing connections when the clients disconnect. If it would just close them rather than holding them open for so long, things would probably be fine.

But if you are having problems, or plan on rolling out NOD32 to maybe one hundred machines or more, definitely consider the use of a third party web server. Don't bother with ESET's http service. We started seeing the problems with as little as 100 clients connecting. If you already have 230 clients rolled out, I'd be surprised if you weren't already seeing it. How often do you have your clients set to get updates?

 

Thanks for the reply.

Indeed , we have about 230 clients but this figure will increase in coming months. Yet, I have no problem with connections not closed by ERAS. Customers are updated once per hour.

So that I understand better, could you give me a diagram of your architecture with IIS and ERAS? There are two servers separate?

 

I'm actually just using 1 server. It hosts both the ERAS and IIS, *and* it runs SQL Server Express 2005 which hosts the ERAS database. Additionally, this is a virtual machine on a VMware ESX cluster. (My company is a heavy user of virtualization.) Runs fine with about 400 clients, and load is minimal.

The nice thing is how modular it is, so I can break those components out to different servers down the road if load becomes too high on the one server. For now, though, things are fine. My only hesitance with any of this was using SQL Server Express rather than the full SQL Server, since Express has a 4 gig database size limit. Thankfully it's pretty easy to upgrade that in the future if I start reaching that limit.

 

You will be hard-pressed to ever come close to the 4gig database cap. I cap the event logging at 25 events for each client, 20 scan logs, threats are left unlimited, and logs are purged in a 6 month rotation. With 580 clients that have been up for the full 6 month log window, my database is only 14mb. Most of the content is the database is a reference to the datastore located in the storage folder.

 

Thanks for that bit of info. My SQL database is indeed very small. Had I know it really wouldn't get big, I might not have gone through the effort of installing SQL Server and rather just gone with the built in "Access" method. And sure enough, my 'storage' folder is already up to 800 megs. I'll keep an eye on that one.

Smacky, out of curiosity, mind if I ask how big your storage folder is? Just wondering what kind of growth I'll be looking at. You have about the same number of clients as us, so that would be useful info for me for planning purposes.

 

Datawise it is 525mb, but because it is flat text files I turned on NTFS compression for the directory and am hitting a 3:1 compression ratio with no noticeable performance impact. I would recommend doing that.