Is the ERA HTTP Server Broken?

I couldn't seem to find a topic directly related to this, but having deployed v3 in our office here, the clients seem to work fine in most cases (no slow downs, etc.), but they will not update reliably.

I have configured the ERA server to be a mirror as well for the clients. However, after a set period of time, the clients start barking "Download interrupted." errors. Deciding to investigate further, I changed from default the option in ERA named "Disconnect from server after update." This seemed to improve the situation, but I am still seeing the intermittent errors -- I dug deeper.

By monitoring the packets coming into and out of a client and the server at the same time, I noticed an oddity that I don't think should be: The server responded to an HTTP request that was from much earlier (going by port number chosen by the client OS). If this connection has been closed by the client, the server should not be responding to it. The behavior I have noticed from monitoring the connections is that the clients make an HTTP request of the server and it keeps them waiting, failing to respond in a timely manner. The client seems to tear the connection down after the timeout period, but the server does not complete the teardown correctly, keeping the connections in the CLOSE_WAIT state. What's worse is that it actually responds after the connection has been terminated.

Is this a known bug in the software? I sent in a support case over a week ago and received no more than an acknowledgment from ESET.

 

Make sure that the installation of NOD32 Business Edition on the same computer as ERAS is not also configured to be a mirror server.

If so, then the NOD32 BE mirror and the ERAS mirror will fight over the same TCP/IP port (2221 by default).

Either ERAS should be the mirror OR NOD32 BE.

 

NOD32 BE has the option to provide an update mirror disabled. Still experiencing the same issues very frequent "Download Interrupted" errors on the clients. Clients do seem to update over extended periods of time, but during the week this seems to take longer than the period between updates some days.

 

Sounds like port contention.

Try using 8088 for http transfer of signatures.

 

I took your suggestion and changed the port to 8088 for the mirror in ERA server, pushed the configuration change to the clients and tried connecting with the client on my machine for an update. The update succeeded (and quickly). I then went to the console and forced all clients to update. I then tried to update with my client again, and got the usual response from NOD32 -- the file update.ver gets to 100% and then the update agent sits there, waiting for a the server to close connection so that it can complete. It does not, and the client reverts to the same "Download interrupted." error. This still indicates the same problem -- the ERA HTTP server seems to stop handling connections correctly over time seemingly by the amount of connections and most especially if they all come at once.

 

How many clients update from the mirror via http? This method is not suitable for networks with more than 100-200 workstations.

 

We should be relatively safe then... only 38 at the moment.

 

I personally have had plenty different problems on clients' computers with the BE and the v3 mirror and we decided we should dump it temporary (I mean I advise corporate clients who purcahse BE to use 2.7 until its support is ended)

Try to use NOD32 2.7 and RA v1 in your network , may be you will not have such problems (I personally never had any serious problem with it) . Hope you too

 

Rolling back to BE 2.7/RA v1 is certainly an option worth consideration, but it doesn't really address these problems with BE 3/RA v2 and they have to be addressed before they can be resolved. It's fine with me if I have to restart the service and force definition updates via RA for a while a few times a day if it means we can correct and improve the software -- it'd just go a lot smoother if ESET would respond to support requests or even admit there's a problem, etc.

 

If you create a mirror using ESS/EAV, does it work reliably and the problem is only with ERA?

 

No, it doesn't work any differently with EAV running the mirror. Same behavior in fact -- fine for an arbitrary amount of connections, then the rest of them connect and then fail to ever disconnect correctly. The server doesn't seem to handle them correctly -- it's extremely slow to respond to clients, and responds after they time out (and close the connection). The server doesn't acknowledge closing the connection, and instead gets mired in trying to re-transmit packets for previously closed connections. This can be captured both on the server and at the clients. When I say arbitrary, I mean it as well. Sometimes only five clients can update before it stops working, while yesterday, all of our clients updated within 1-2 minutes and it stopped working after that. Both ERA and EAV seem to have no problem accepting new connections after the problems arise, but they will not respond until all previous connections are answered, regardless of the fact the TCP connections have already been closed (or at least an attempt was made to close them).

I don't know if this problem is an issue with EAV Mirror/ERA mirror or an issue with Windows itself, but logic would follow that if it's a problem Windows, the other features of ERA running on the same server would have similar if not the exact same issues, and that simply isn't the case. Of all the services running on the server (and it's under barely any usage), EAV Mirror and ERA mirror seems to be the only one having an issue.

I've also tried setting EAV's protection to disabled, and I've checked the system logs and the firewall log -- nothing seems to be amiss.

 

Could you try editing the default update task and set groups of computers that will update at different times during the day to see if it makes a difference? I gather that updating via Windows shares works like a charm, the only problem is with updating via http. It's weird as no one else has ever reported this problem and we have large clients who update from mirror via http fine.

 

I could try doing that but I don't believe I'll see any resolution overall. If we leave the clients to update at their normal intervals of 60 minutes, most of them do update -- eventually. That's to say, they manage to update after they've popped up countless "Download interrupted" errors. And then there's the fact that it's most of them, not all. It seems a few of them (never the same ones) get stuck in a perpetual state of inability to ever get a successful connection to the server. It's always around 10 of them that never seem to update when the system is left to its own devices over a few days.

As to configuring the clients in groups to update at different times of the day, I don't see how that helps us. Even if it does avoid the errors, our clients still aren't getting timely definition updates (which is already the primary problem). Is it possible that this issue is something very specific to our hardware and configuration? Sure -- I understand that, it's why I'm trying to work with you to resolve the problem. If there's some type of debugging software you need me to run, or a log to enable, I will be happy to do so and send any results to you.

I just don't buy into working around the problem, I'd rather get it fixed.

 

We have exactly the same problem, but more annoying since we have at the moment about 120 clients (out of 350 purchased licenses) and the ERA HTTP SERVER is crashing about 3-5 times per workday.
Usually the first thing I do when I arrive at the office is to restart the ERA_HTTP_SERVER service instead of taking care of the first coffee ritual.

I'm even thinking to make a batch file with the 2 following commands ("net stop ERA_HTTP_SERVER" and "net start ERA_HTTP_SERVER") and schedule it to run every 2 hours, but tbh it feels like a very cheap workaround.


PS. I noticed in the era_http_server.xml file an option looking like that:
<OPTION OPTNAME="ThreadNumber" VALUE="10" />

I wonder what's happening if that value gets modified 'by mistake' to 100

 

I sent a packet capture of the TCP connection issues to ESET. I received a response to my ticket open with them on Sunday indicating that it's a bug in the software. They claim to be waiting on a response from the developers.

I had never noticed the thread limit, but it would suggest that it is indeed a problem with the threads not closing the connections correctly. If the system is left on its own for extended periods of time, the clients begin to connect at pseudo-random intervals, allowing the limited amount of threads (presumably 10) to time out naturally. If I haven't heard from ESET I may change the setting on our server.

I think the big question is whether this affects every installation of ERA/EAV mirror or if it's specific to certain platforms? We have it running on a 2003 R2 Standard 64-bit server.

 

Our ERA Server is on a 2003 x86 Standard R2 SP2 server.
The server is an Intel SR1500AL, a dual core Xeon and 1GB RAM, SATA Drives

 

Eventually I gave up and reconfigured all clients to get updates from a network share instead of http.

 

Did anyone figure this out? I recently had to setup a mirror as certain systems have no real "web" access, so they HTTP to our server via HOST entries to get updates.

We have 40-ish clients and since I set it up yesterday I've got several clients with the "Downlaod Interrupted" warning comming and going.

 

I spoke with Eset supporting the phone, and they got me to sent them event logs from the server and a couple of the clients.

The amount of failing clients continued to grow as time went by on the phone (a hour or so total, they DID answer the phone after two rings. ).

I rebooted and things have stayed clean for a couple hours now. We'll see how long that lasts.

I'm thinking I'm going to see if I can rig up IIS to be the server, that might be the best bet if I can do it with at least Basic authentication.

 

Ok So a new update was downloaded by the server, and clients started trying to get it. I started getting the "Download Interrupted" alerts again, as well as a couple "blank" alerts.

Here's the latest email I sent Eset:

Anyone been this far into it with support? Is there any steps they got you to do that I can try before they ask?

Anyone know the difference between the “Eset RA HTTP Server” and the “Eset HTTP Server” services?

 

Hello,

ESET NOD32 Antivirus 3.0 Business edition has the same ability of creating & providing mirror for updates as ERA does. The service "ESET HTTP Server" belongs to ENA BE and shouldn't be running when ESET RA HTTP Server is on.

 

I haven't heard anything other than it being an issue they're aware of -- that was several weeks ago and I've yet to see any resolution. Rebooting the server takes care of the issue briefly because it restarts the service. Also, the occasional restart of the service (every fourth or fifth try, approximately) will yield a running service which is somewhat more reliable.

As I've sent to ESET via packet captures, it seems that the HTTP server embedded in ERA is not handling the TCP connections correctly, which leads to the 'Download Interrupted' chaos.

 

Ahhh, OK, well it's not "running", it's just there in "Manual" mode. I do see this service on the clients as well, so that explains that. Not sure why the service is installed when there's no mirror, but hey.

There are no other Mirrors set up other than the one setup in the RAS itself.

 

Well I just called them up again to check a status on my case, and they're going to review the new info I've sent very shorty and are supposed to contact me in a couple hours, so we'll see what's what.

Perhaps you should do the same to prod them on a bit, if we keep on them apparently things get looked at faster, so if there's multiple of us calling daily, then perhaps they'll figure it out faster. I find waiting for them to get back to you can cause it to be weeks to hear back, whereas so far calling seems to keep things moving.

I find that one restart of the service is enough to get it up and running, at least until the next update becomes available, which is annoying, cause I'm not up at 3AM and such.

I guess I'll have to start testing that IIS idea later tonight and see if I can pull it off if they don't have an answer.

This seems like a pretty major flaw for business users and it SHOULD be fixed quickly.

 

Ok so I got an email message asking that I send them a Belarc report, so I do.

I then got an email asking me to compare my settings to Page 13 of the manual. I do, and surprise it's the exact same page I read when I started setting the thing up.

Here's what I sent them this time, we'll see what happens next:

Ever since I replaced the internal Eset HTTP server with IIS6 I havn't had a single update error, which isn't too surprising.

If anyone wants to know how to setup IIS6/2003 to be the mirror, let me know and I'll post some info, there was a couple minor snags (ie: MIME), but nothing too bad.

I'd still rather use the internal HTTP since I'm kind of 'forced' to have it installed with the RA anyway.