Severe PG 3 slowdown when Alerts tab fills up

Hi guys,
I did some heavy batch-filing today and there were several hundred executions, perhaps spaced out by about 5-10 seconds. As the batch file ran, I noticed my machine slowing down exponentially. I checked Task Manager and Procguard.exe was eating 70-80% cpu. The GUI window was closed, but the tray icon was running.

After the batch file completed, Procguard.exe continued to take 50-60% cpu, even though I was doing nothing. As soon as I cleared the "Alert" tab list, Cpu went back to 0%. So it seems that there may be some problem sorting or even maintaining in memory a large alert list.

Perhaps this code can be optimized to handle this better? It is not uncommon for me to have several hundred executions in a session. Or could something else be causing this? It is indeed strange.

 

Hi LuckMan212, There may well be some optimisation possible to the alert log but for now you could try switching of Execution protection temporally whilst running your batch files.

Jason will have to answer from the technical point of view.

Thanks. Pilli

 

perhaps there could be a commandline that could be run to temporarily disable PG from the batch file itself? it runs at night unattended so I am not there to manually disable PG usually.

some sort of secure command line disabling would be nice (of course with option to disable it for users who dont want the risk)

example:

Code:

>pgcontrol.exe -disable -password:xyzzy

of course password set via the GUI somehow beforehand and stored encrypted in pgdata files.

bad idea / good idea?

 

Jason will have to comment on your idea, using a command line sounds like an interesting idea.

Another thought occurred to me about this, ProcessGuard is still protecting your PC if the procguard.exe (GUI) is not running, so exiting the GUI completely may also stop some or all of the logging - Worth a try

Pilli

 

To be more specific (at least as far as I understood it): The service does the logging to the logfiles, but the logwindow is the gui's business. So you would most certainly avoid problems with the log window (as you seem to be having insofar as clearing the log window seems to help), but you would still have all the information in the log files...

Andreas

 

Thanks for the ideas guys, but... even if closing the GUI was a solution it's not an ideal one for 2 reasons:

1) program should be able to handle this gracefully without needing to close the GUI

2) I am not around in the middle of the night to manually close the GUI or exit the tray icon anyway

Hopefully Jason can comment on this

 

Agreed and I am sure Jason will comment on this

Surely if you are not around it does not matter? As the GUI does not need to be running for protection to work, your logfile will be there when you next open the GUI, just close the GUI before you leave the machine. As we have suggested it is worth a try and I am sure Jason will look into the matter ASAP.

Pilli

 

I would guess the issue is "sorting" the list with many thousands of items is causing the CPU usage issue. I might remove all sorting from the ALERTS list because it isn't as useful there and would resolve any CPU usage issue with it.

 

sounds good to me! thanks!

 

One caveat is that, if EXECUTION alerts cannot be disabled separately, it is a tedious job to browse hunderds or thousands of alerts looking for anything important.

 

Maybe a user defined option that is "clear all alerts more than xx hours old" at a user defined time of day? Just a thought.

 

I was thinking more of a case where a batch script runs all night, then trying to look through thousands of EXECUTION alerts the next morning. Not sure what's best to suggest. The log file is available, but Unicode makes it difficult to use with most programmer tools. Also (I think) the log file tends to omit the process name when a hook is blocked (but maybe that's getting fixed ).

Perhaps new alerts could be appended to the window, and the window's contents "marked as dirty". When the API send you a message to update the window, you could re-sort at that time. That may be a fairly long sort though (as opposed to insertions). Looks like a lousy idea the more I think about it.

 

Maybe a little options box somewhere that specified how many of the alerts are to be kept in memory could resolve the issue by avoiding the problem of CPU and memory use....

Something like a tickbox "limit alert messages in memory"
and a numeric counter "number of messages in memory"
for the full monty: a tickbox "disable alert sorting"
and make the default to allow sorting with 50 messages in memory

These options could go onto the new "options" tab when the "advanced" mode happens (ever hopeful) as one of the many little options

 

As a sidenote:
This is working on my comp.:
type pglog_11*.txt | grep something

(with grep and type from unxutils)

 

I'm no longer struggling to search/process log files now. I discovered that my editor (Vim) handles it as easily as it handles ascii. I thought I'd already tried that and failed, but it must have been before I lost my mind.

I do still have a concern about not having the program name in log files when PG blocks a program. It does appear (albeit, sometimes, erroneously) on the Alert Tab. Once that's fixed, I'd be fine with an abbreviated alert screen. Truth be told, I'm not too bothered either way.

 

Andreas, your sidenote looks like something interesting, but I may be confused. I would have expected either a DOS-style "type" or a UNIX-style "cat" to pipe the input to grep. The "type" in UnxUtils.zip is different than what I'm used to (MKS toolkit) but neither "type" writes the contents of a file to stdout.

I tried the UnxUtils' version of grep.exe, but could only match "EXECUTE" as "E.X.E.C.U.T.E" because of the multi-byte characters. Even though the second version matched, grep still complained, "Binary fiile (standard input) matches", and would not display the matching text. My grep didn't even cooperate that much.

I didn't see a switch for grepping multi-byte characters, so I'm wondering whether you convert log files to ascii before grepping, or perhaps that conversion happens transparaently on your system.

In any case, thank you for the pointer to UnxUtils.zip. I saw some handy new grep switches, and I'm sure there'll be other useful things, too.

Regards, Mike (sorry if OT)

 

sorry off-topic still
hi earth,
in fact, I've simply forgotten about 'cat'. I tried 'less' first, but it obviously couldn't handle unicode and produced garbage. Then I tried 'type' and it worked. But it seems that i was mistaken in thinking i was using the 'type.exe' that is in my unxutils folder. While that folder is in my search path, obviously what got called is w2k's cmd.exe's "internal" type. (I've now explicitly stated the path to the unxutils in calling and got a PG Execution protection request for type.exe - which i don't remember having gotten before. DOH!) Sorry for the confusion.

Andreas

And yes .. . . .. . vim rules!!

 

my thread's been hijacked!

 

Sorry Luckman, I got carried away trying to make sure I'd know what to do without my many thousands of sortable Alerts.

 

Hi earth1, Maybe you and Andreas should contact each other via the boards IM I still find the topic interesting though

 

LuckMan,
could you -- for the time being -- confirm whether or not closing the gui solves the CPU problem. I.a.w. does the CPU troubles also occur when only the service does the file logging?

and sorry for the ... ehm ... off-topicity

Andreas

 

I will check that out and post back when I have the results... and no hard feelings for the hijack, I was just kidding. Interesting stuff....

 

It would also be interesting to know if you have to click on any of the headings to do explicit sorting prior to the CPU usage happening, and also to know if that is done does the CPU usage get any worse ?

ie: if you sort by the program name then it will need to do some work and move things around in the list box so it should be more CPU intensive

 

I started poking around and got some surprising results. When I got past 2000 alerts things started slowing towards a crawl. At that point, it was pretty easy to start measuring CPU spikes with taskmgr on fast update. It took about half a second of procguard time to run a little do nothing program. The following is a rough estimation of what taskmgr attributed to procguard for CPU usage.

Inserting one entry (running the program once more) tended to be consistent in its time requrement. Clicking on a column heading in procguard's Alert Tab varied from a low that equaled the insert time, to a high of about 50% greater. The average seemed to be about 15% greater than the insert time.

I wonder if procguard may be re-sorting the entire list of alerts each time it adds one (as opposed to traditional insert logic) and/or sending one or more Windows messages for every entry when it adds one. The latter might best explain why there seems to be a point at which the system bogs down so completely.

 

earth your results are consistent with mine and I appreciate your thorough testing.... I have not had a chance yet to confirm that number but it "sounds" right. I will post my results but hopefully Jason can chime in. Perhaps the sort routines can be optimized? I know that for example MS Excel can sort 5,000 rows in a basically unmeasurably short time (at least on my system) so perhaps there is some room for improvement in the sort algorithm. Or perhaps a piece of 3rd party sort code can be used as I know most programmers do not like to write such routines themselves. Reinventing the wheel and all that....