ESET Mail Security for Exchange 4.2 Hanging Exchange & Outlook

Since this morning have been struggling with our Exchange 2003 SP2 server, running on Windows Server 2003 R2 32-bit.

It started with two users unable to access a particularly public folder, it hung their outlooks, this escalated into everyone's Outlook hanging "Waitnig to update this folder" and the Exchange Queues to begin to fill and not move (Local delivery, pending submission).

It has v4.2.10019.0 which has been on there for a couple of days with 10016.0 on for a week before that, no issues.

I tried turning everything off, doesn't help. I uninstalled and reinstalled a fresh 19.0 instead of 19.0 update ontop of 16.0, while no ESET was installed, it was fine again.

I was about to uninstall a second time, but ran a repair, although this stopped and said a file was in use and I cancelled it, this appeared to sort it for a while.

It's now begun happening again, I've tried the pre-release updates and that didn't help either.

Sometimes the Exchange System Manager MMC hangs, and ending task on this has also allowed a few emails to squeeze through.

I'm going to have to do seomthing to get mail flow back... I may have to go back to 2.7 or installed again and leave old definitions on.

PS. I noticed the issue while we had 5345, and it is still happening with 5346.

ESET Mail Security
4.2.10019.0

Virus signature database: 5346 (20100806)
Update module: 1031 (20091029)
Antivirus and antispyware scanner module: 1284 (20100729)
Advanced heuristics module: 1109 (20100519)
Archive support module: 1118 (20100729)
Cleaner module: 1048 (20091123)
Anti-Stealth support module: 1019 (20100525)
Antispam module: 1014 (20100212)
SysInspector module: 1216 (20100517)
Self-defense support module : 1016 (20100404)
Mail server protection module: 1005 (20100112)

 

Ok... currently have everything disabled, and first I unticked Transport agent and VSAPI under the Antivirus and antispyware->Mail server protection tree, then the first Mail server protection tree in settings.

Email is working again, but aside from document protection all other modules are off.

Update: I've slowly re-enabled the other modules, so that only Mail server protection for Antivirus and antispyware protection and antispam protection are disabled, ok so far.

 

I've got a ticket open with ESET UK who have been in touch, very helpful.

Narrowed it down to just having VSAPI disabled, everything else including antivirus and antispam protection by transport agent is working.

However, soon as I enable VSAPI, emails start queuing up, Outlooks start hanging (when browsing public folders).

SysInspector log generated and it's all been sent off to developers, waiting to hear back...

 

Nearly a 100 views of the thread, guessing that's people interested in ESMX 4, if anyone has a similiar problem, do say.

I like to think I'm pretty capable, I've solved many problems with other software over the years even I'm stumped as to what's suddenly caused this.

Even after a fresh reinstall of Build 19, no updates, it does it... which I sure was fine before, it had been running for at least a week. It had been days since any Windows Update was installed.. it did have HP Updates applied (firmware updates for iLO and the like) but that was the previous night and it was fine for the start of the morning.

Logically then, since it seems to happen in public folders, I can only think an email has gone in there it doesn't like or can't process possibly.

 

I am also having troubles with ESMX. Everything seemed to work fine until yesterday, my complete SBS2008 server hang. Black screen, nothing to do about it, except a reset.

Today my server hang again. When it happens it produces this error message:

Column Name Value
Client Name Server
Computer Name Server
MAC Address d8d3855e9bee
Primary Server Server
Domain domain.local
IP 192.168.0.xxx
Product Name ESET Mail Security Microsoft Exchange Server
Product Version 4.2.10019
Policy Name Default Primary Clients Policy
Last Connected 2010-08-12 17:39:53
Protection Status Text
Virus Signature DB 5361 (20100812)
Last Threat Alert
Last Firewall Alert
"Last Event Warning Reading of Exchange configuration from Active Directory failed with result 0 and code 8007203A" <<<<<<<<<<-------- !!!
Last Files Scanned
Last Files Infected
Last Files Cleaned
Last Scan Date
Restart Request
Restart Request Date
Product Last Started 2010-08-12 14:09:44
Product Install Date 2010-07-25 14:10:25
Roaming User
New Client Yes
OS Name Windows (R) Small Business Server 2008 6.0.6002 Service Pack 2 x64 Edition
OS Platform Microsoft Windows
HW Platform 64-bit
Configuration Ready (71 minutes ago)
Protection Status Ready (2 hours ago)
Protection Features Ready (2 weeks ago)
System Information Ready (108 seconds ago)
SysInspector No Data
Custom Info
Comment

 

EMSX hung our mailserver today also. We have fought with EMSX/NOD32 for months now (shoot, ever since we switched from Symantec over a 1.5 years ago we have been fighting with this) with it hanging our servers. Went through ESET support and all the exclusions etc. It seemed to just "fix" itself after we installed some firmware updates for our HP DL380. Seemed odd that some firmware updates for iLO would be causing a problem, but been fine for the last month until today.

Hung the server exactly the same way that NOD32 v4 hangs Windows Server 2008 (numerous other threads on that). The server was up and could respond to ping, but could not RDP in. Outlook could not connect to Exchange 2010. Going to the console we could not login and had to hard reboot the server. No Mini-Dump created as the server never fully crashed.

 

I also found out that a signature update (5358, see this link) could cause an instable system. My server did freeze the first time after update 5358 was installed. It did freeze the second time with 5359 installed.

@rockshox: Thanks for the tip! I also have a HP Proliant Server (ML350 G6) so I will install the newest drivers/firmwares with the newest PSP.

I will let you all know .

 

And it happened again. My SBS2008 hangs and it is not possible to reach the server anymore.

Can someone of ESET give me some support on this problem?

 

You should open a support ticket with ESET on this issue. The more people we can get opening Tickets will hopefully trigger someone seeing that this is actually a major problem, it's one thing to have a few issues with a workstation, it's a major thing to have servers lockup. We currently have a ticket open with them and have been going back/forth on this issue.

Once you open a Support Ticket they are going to ask for a copy of your Configuration, a SysInspector log, and most likely a Memory Dump.

 

Completely unsolicited debugging tip of the day:

You can get crashdumps on instances that refuse to BSOD when they probably should by enabling the EMS service and attaching a (virtual) serial terminal to access the SAC serial console. I have this enabled on all my critical Windows instances (most of which are virtualized) so I can get a crashdump even if the instance won't die of its own volition.

Enable EMS on 2k3 with "bootcfg" like this:

bootcfg /ems ON /port COM1 /baud 115200 /id 1​

Or on 2k8 with "bcdedit" like this:

bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200
bcdedit /ems on​

If you want to watch windows 2008 booting in a verbose way all the time and log what it's doing:

bcdedit /set sos yes
bcdedit /set bootlog yes
bcdedit /set quietboot yes​

Once you're done enabling EMS, attach your serial terminal and start the box back up. When you get to the point when you want to cause Windows to bugcheck artificially, type "crashdump" at the SAC> prompt.

 

@ twichert

Thanks for the info, we'll keep that in mind if our current plan doesn't work out. We have a keyboard plugged in and the registry key in place to allows us to force a memory dump (http://msdn.microsoft.com/en-us/library/ff545499.aspx). The biggest issue is getting to the server before ASR reboots the server due to the lockup. We also can't turn off ASR in case the server locks up in the middle of the night, ASR will reboot it for us and hopefully save a trip into work in the middle of the night.

Were now waiting for the next lockup. It could be 3 days or 3 weeks, always random.

 

You are not alone as I am experiencing the same issues and symptoms you guys are. I have been in touch with HP, Microsoft, and now ESET. Originally thought it was a Server issue (HP DL380G6) so I installed every update known to man, and I would get my hopes up and it would lockup again, randomly. Was up three weeks this time and I tought I had it resolved, then locked up again, so know I am here hoping it is a Mail Security issue. Getting ready to try the latest build of Mail Security.

Also note I found this hotfix that I am going to try: http://support.microsoft.com/kb/2265716

 

Very interesting KB article. The symptoms definitely fit with issues we have seen, however Microsoft is stating it's a Windows 2008 R2 issue while I've had both of my Windows 2008 (non R2) File Servers lockup with NOD32 4.x with the exact same symptoms. Our mail server is a Windows 2008 R2 machine. I wonder if the same problem exists in both versions? I wonder if it is a conflict with HP? We are also running HP DL380's....

Let us know what you find out.

 

That is what I thought initially refering to HP article http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01955503 and in fact have replaced the motherboard as well. problem persists. Just to post I am running Exchange 2010 on top of Windows 2008 R2 Enterprise. I am at a loss at this point.

 

We are running a DL380 G5, so that would rule out the motherboard issue. You've replaced it and it hasn't fixed it anyways. We are also running Exchange 2010 on top of Windows 2008 R2 Enterprise.

We removed ESMX from the server for a few weeks several months ago, and did not receive a single lockup during that time.

 

Sounds like it is pointing to ESET being the issue. Wish I could uninstall it for a while to verify, but then I would not have a SPAM solution running. I sent my logs to ESET support but they did not come back with anything except the usual exceptions and such which I already went through. They do not seem to have a solid grasp on this issue.

 

Yes, we have an open support request with them also. They wanted us to be sure we had proper exclusions (which we did). We already have turned off Anti-Stealth and Self-Defense. We provided them with SysInspector logs, Configuration file and a Word document of errors that the Windows Error Reporting seemed to pick up about the time of the lockup.

Last we heard form them they sent some e-mail that we needed to not scan all extensions in our Scan Profiles. So we changed the In-Depth, Context Menu and Smart scan to only scan the list of provided extensions. However I can't figure out how in the world this would affect anything as we don't have any schelduled scans. Were now "waiting" for it to lockup again which can be anywhere from a few days to six weeks........

 

I'd suggest checking the temp folder and deleting all obsolete files.

 

It is interesting to see that all our servers are HP servers, we have a HP Proliant ML350 G6. Could it be a conflict between an HP driver/firmware and ESET?

When I check out my ILO2 log files, I see the following when the server crashes:

Informational - iLO 2 - 08/25/2010 04:11 - Server power restored.
Informational - iLO 2 - 08/25/2010 04:11 - Server power removed.
Caution - iLO 2 - 08/25/2010 04:11 - Server reset.
Informational - iLO 2 - 08/25/2010 04:11 - BMC IPMI Watchdog Timer Timeout: Action=System Power Reset.

 

That looks like HP's ASR (Automatic Server Reboot) kicking in and rebooting the server when it detects that the server is hung. If you have the HP Drivers installed, the ILO2 talks to the driver running on the system. If the ILO2 can't communicate with the system drivers for 10 minutes (default time), it then will power the server off/on to restart it.

You can adjust the ASR time, or turn it off via the HP Management Homepage. However to do that you have to switch the HP Management page to use the SNMP provider instead of the WBEM providers.

 

Anyone that is following this thread, or has ever had their Windows Server 2008 servers lockup with NOD32 or EMSX, can you please post what hardware you are using.

There is definitely many people here with HP servers that have had this error. We might be on to something.

 

I checked C:\Windows\Temp on our Exchange 2010/EMSX mail server. There are a few files in there that we can probably delete out.

Can I ask why you are suggesting to delete the temp folder? I hoping there is actually a reason for this suggestion, and not just something that is "good to do".

 

HP DL 380 G6 here, Windows 2008 R2, Exchange 2010 running ESMX.

Currently waiting for the system to lockup again so I can generate a memory dump to send off to Microsoft.

First time I ever intentionally WANTED to see a blue screen, now I am waiting to force one!

 

Just to list mine:

HP DL380 G5 Windows Server 2008 R2 Enterprise w/ Exchange 2010 and EMSX.

2x HP DL380 G5 Windows Server 2008 Standard w/ NOD32 3.0.695 (4.0.x locked up these two file servers multiple times so rolled back to version 3. Not a single lockup in over a year on version 3.)

 

We are running Windows Server 2008 R2 with Exchange 2010 and the latest ESMX under vSphere 4.0u2 and are experiencing the very same issues. Not sure if that puts a hardware issue to rest or not. We are holding off on 4.1 as I am conservative when it comes to updates of that sort.

The host platform is a Dell R710 with dual quad core Xeon E5520's, 48GB RAM, and is connected to it's SAN storage LUN via 4Gbit iSCSI link (EqualLogic PS6000).