Emergency Help with Mission Critical Services

Help!

I have long been a user of Nod32 to protect my home PC and have never run into a problem. (knock on wood). I administer the network of a law firm in Atlanta and we have recently been infected with a number of virii. We had Symantec half-installed and I had planned to switch to Nod32 once a couple of critical cases had passed and we could upgrade our server.

Now my server and all of my networked PCs have so many virii that I can't keep one running for more than 5 minutes without all hell breaking loose. Even though I have installed NOD32 on a couple of machines, it is not solving the problem. Everytime I start up my server, NOD32 finds the same 40-50 viruses and deletes them (along with several of my local services). I reboot and it happens again. I do a fresh OS install on a fresh hard drive and within one booting, I'm infected again.

At this point I am willing to scrap all of the hard drives for all of the PCs on my Network to create a master disk image that I can replicate to all of my PCs that contains NOD32 installed from the beginning. Is this the best solution and is there any guarantee that with a new hard drive, OS, and Nod32 running on each PC that I won't have the same problem as soon as I try to access the old data only from the infected drives.

I have to have this solved by Noon on Sunday. Please help.

Thanks in advance for your support.

Jason

 

it would help if you could isolate the machines - I fix them offline so that other machines in the network can not re-infest - and to know what infestations you have (ie, what virus/trjojans are present)

 

Exactly - this is absolutely critical - and bring nothing back on the LAN until you're sure that all are fine.

If you used the old drives in a pure "data mode", you will probably be fine as well. The probably derives from - if the drives are "data only", you won't be launching executables from them, however, you still could be executing active content (VBA macros, etc.) if Office and other applications are set to do so and that could set the ball in motion once again. It depends on the source of the infections.

Isolate by moving all clients off the LAN - analyze - clean - verify while isolated - repeat for all clients, then the server - then reconnect to the LAN.

Blue

 

Thanks for the replies,

So, what my plan is this: I am going to reinstall the server from scratch on a new hard drive. Then I am going to purchase new hard drives for each PC and create a master image with everything installed correctly and installed from clean sources. Then I am going to deploy a new hard drive to each PC one at a time and bring each PC on-line clean one at a time.

Then once I have my clean network back and running with the basic software, I will extract all of the data from the old infected drives on a PC that is not connected to the network. Once I verify that all the data is Virus free, i will copy it to a Network Attached Storage device and share it amongst my clients.

Is this a good idea?

Jase

 

Sounds like a good plan
In the future, you should enforce stricter rules in your workgroup/domain (for example, disable autorun on removable devices, read/write permissions to network shares, filetypes allowed to travel in mail, web access, content filtering)

 

Okay. I can't get my boss on the phone to approve the 1000.00 in new hard drives, so what about this:

1. I have removed all PC's from the network including server.
2. I install one new hard drive in a PC with a clean OS and Tools installed on it
3. I get the clean PC setup the way I want it and make a "CLEAN IMAGE" of it
4. With the next PC, I remove the hard drive and attach it to the clean PC as a slave drive.
5. I'll run NOD32 on the infected slave drive
6. then back up all the data only (no macros, executables, etc) to a disk image on the CLEAN PC
7. Format the infected drive
8. Deploy the CLEAN IMAGE to the newly formatted drive
9. Restore the CLEANED ORIGINAL DATA to the drive
10. Return the drive to the original PC
11. Make any necessary adjustments to the rebuilt PC
12. Attach rebuilt PC to network
13. Next PC

This way I only have to have one large drive, I have a backup solution in case one of the drives fails, I am guaranteed clean installs/configurations are done correctly. I think it will work. Any suggestions?

Jase

 

Well said. Unfortunately, two months ago when they hired me to "get their server up and running", they were clear about not wanting even a part-time IT person on a regular basis. They wanted their server up and running and that's it.

Needless to say the last two months have been all prep work to get their infrastructure setup adequately (like RJ45 jacks for ethernet, lol) and resolving problems with their current service providers. Once I finally get things under control and have the basics stable enough to implement the new services a Server offers, I get this.

I don't know how to explain that this was HIS fault, not mine. I'm taking the heat for his years of neglect to his "network". It sucks.

Jase

 

Keep the clean image outside of the PCs. The virus may infect your freshly formated drive as soon as you plug a drive from a infected PC. I'd do the cleanup from a LiveCD environment, just to be safe.
Don't use the network for anything. If you need to download something (like AV updates), do it from a clean PC (your home PC for instance) and burn them to non-writeable media (CD).
Before formating, kill the partitions and zero the HDDs with the tools provided by the manufacturer.

 

Jase,
You do seem to have a pickel! This is a good plan, costly, but can work. How do you plan to save all the data on each of the client machines if you give them new fresh drives? Or, do your client users not store data on their PCs? I am familiar with law firm needs/desires. Tough. Hope you are not an IT department of one....

I think the correct answer is at answer #2 above. really think a complete server reinstall is overkill, but it is your decision. If it were me, I would:

1. unplug the clients from the lan.
2. shutdown any devices that allow inbound to your domain/lan. {like something that allows mobile systems access to your lan; they could be infected!)
2a. well you do need one "dmz" machine to talk to the www.
3. unplug the server from the lan. isolate the server. set it to scan/clean in safe mode. repeat as necessary.
4. scan/clean each client. if the client can not be cleaned to your liking, replace the drive and rebuild the client.
5. when all clients are clean, check the server. If the server cleaning does not suit you, replace drives and rebuild.
6. rebuild domain as necessary.

Yes, that should work fine. Make sure the NAS is clean also! All machines on your lan needs to be cleaned.
I live some 70 miles up I-75 from you. If you need a voice or help with nod32, send me a PM. It's DST shift so I'll be up late checking stuff here!
Best,
Duncan

 

Tell him that having networked PCs without control and some IT watching it is asking for trouble. An AV is only part of a security strategy.
I won't be surprised if he doesn't do proper backups

 

To speed things, if that's critical - branch out on the slaves for cleaning (i.e. first cleaning yields a second platform, those two working in parallel yield another pair, and so on). You don't want to be juggling too many scanning clients - but if you do this twice, then reserve a PC for the main backup/reinstall, you will significantly improve throughput (one dedicated for reimage/data restore, three others scanning in parallel). Just be sure to keep track of things.

Blue

 

@Blue,
A fine example of Taylorism

 

Actually, I thought bootstrapping would have been a somewhat closer, although clearly imprecise, analogy

Blue

 

I didn't think in bootstrapping. Nice catch
End of offtopic

 

Thank you all for your help and input. I am still just a wreck over this entire thing. I have been working on it non-stop for three days. I have tried everything that everyone has suggested and I am still infected with SOS and SYSMON.

I am going to try the clean install from a CD that is not infectable and see if that works better. My boss is understandably irate that he can't use his computer to even check his e-mail, let alone prepare a really important brief for court tomorrow.

I have to run to the store for a couple of things, but I will be back to ask for more help in a few minutes. Thanks again for everyone's input.

Jase