PageGate stop at Midnight...

[logimpact]

We have a problem where PageGate stop often at Midnight for no reason. I checked the Logs and I do not understand what happen. In the task manager the process is runing but the program hang somewhere. I have to kill the process to be able to restart the program.

Do you have any ideas?

Here's some logs:

PageGate.log
2008-12-12 23:59:40 Module resumed
2008-12-12 23:59:44 Unable to synchronized time with scheduler. Using local time.
2008-12-12 23:59:44 Asking PGSched module to start
2008-12-12 23:59:47 PGSched start detected
2008-12-12 23:59:47 Asking PGGAscii module to start
2008-12-12 23:59:49 Asking Connector1 module to start
2008-12-12 23:59:56 Log ping...
2008-12-12 23:59:58 Module paused
2008-12-12 23:59:58 Running cleanup routine
2008-12-15 13:20:40 Module Started
2008-12-15 13:20:42 Asking PGSched module to start


PGsched.log
2008-12-12 23:59:05 Timeout... continuing
2008-12-12 23:59:05 Waiting for connectors to pause
2008-12-12 23:59:05 Connectors paused
2008-12-12 23:59:05 Waiting for interfaces to pause
2008-12-12 23:59:05 Interfaces paused
2008-12-12 23:59:05 Attempting to compact database
2008-12-12 23:59:10 Database in use, unable to compact
2008-12-12 23:59:10 Trimming log files
2008-12-12 23:59:10 Checking database for null values
2008-12-12 23:59:10 Checking database for invalid group entries and dups
2008-12-12 23:59:10 Removing old messages from database
2008-12-12 23:59:10 Removing old files from badmsg folder
2008-12-12 23:59:10 Resetting database flags
2008-12-12 23:59:11 Module resumed
2008-12-12 23:59:15 Program Halted. Database error: 3061 Too few parameters. Expected 2. in Procedure: check_license
2008-12-12 23:59:15 Restarting
2008-12-12 23:59:24 Module Started
2008-12-12 23:59:25 Running cleanup routine
2008-12-12 23:59:25 Waiting for server module to pause
2008-12-12 23:59:30 Server module paused
2008-12-12 23:59:30 Waiting for connectors to pause
2008-12-12 23:59:32 Connectors paused
2008-12-12 23:59:32 Waiting for interfaces to pause
2008-12-12 23:59:32 Interfaces paused
2008-12-12 23:59:32 Attempting to compact database
2008-12-12 23:59:38 Database compacted
2008-12-12 23:59:38 Trimming log files
2008-12-12 23:59:39 Checking database for null values
2008-12-12 23:59:39 Checking database for invalid group entries and dups
2008-12-12 23:59:39 Removing old messages from database
2008-12-12 23:59:39 Removing old files from badmsg folder
2008-12-12 23:59:39 Resetting database flags
2008-12-12 23:59:40 Module resumed
2008-12-12 23:59:40 Program Halted. Database error: 3061 Too few parameters. Expected 2. in Procedure: check_license
2008-12-12 23:59:40 Restarting
2008-12-12 23:59:45 Module Started
2008-12-12 23:59:46 Running cleanup routine
2008-12-12 23:59:46 Waiting for server module to pause
2008-12-14 23:59:59 Timeout... continuing
2008-12-14 23:59:59 Waiting for connectors to pause
2008-12-14 23:59:59 Connectors paused
2008-12-14 23:59:59 Waiting for interfaces to pause
2008-12-14 23:59:59 Interfaces paused
2008-12-14 23:59:59 Attempting to compact database
2008-12-15 00:00:05 Database in use, unable to compact
2008-12-15 00:00:05 Trimming log files
2008-12-15 00:00:05 Checking database for null values
2008-12-15 00:00:05 Checking database for invalid group entries and dups
2008-12-15 00:00:05 Removing old messages from database
2008-12-15 00:00:05 Removing old files from badmsg folder
2008-12-15 00:00:05 Resetting database flags
2008-12-15 00:00:06 Module resumed
2008-12-15 00:00:08 Initiating cleanup routines
2008-12-15 00:00:08 Module Stopped
2008-12-15 13:20:43 Module Started
2008-12-15 13:20:44 Running cleanup routine

PGDial1.log
2008-12-12 23:59:00 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:59:07 Running cleanup routine
2008-12-12 23:59:08 Module Stopped
2008-12-12 23:59:15 Module Started
2008-12-12 23:59:16 Log ping...
2008-12-12 23:59:27 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:59:31 Running cleanup routine
2008-12-12 23:59:32 Module Stopped
2008-12-12 23:59:50 Module Started
2008-12-12 23:59:55 Unable to synchronize time with scheduler. Using local time.
2008-12-13 00:47:00 Log ping...
2008-12-13 01:47:00 Log ping...
2008-12-13 02:47:00 Log ping...
2008-12-13 03:47:00 Log ping...
2008-12-13 04:47:00 Log ping...
2008-12-13 05:47:00 Log ping...
2008-12-13 06:47:00 Log ping...
2008-12-13 07:47:00 Log ping...
2008-12-13 08:47:00 Log ping...
2008-12-13 09:47:00 Log ping...
2008-12-13 10:47:00 Log ping...
2008-12-13 11:47:00 Log ping...
2008-12-13 12:47:00 Log ping...
2008-12-13 13:47:00 Log ping...
2008-12-13 14:47:00 Log ping...
2008-12-13 15:47:00 Log ping...
2008-12-13 16:47:00 Log ping...
2008-12-13 17:47:00 Log ping...
2008-12-13 18:47:00 Log ping...
2008-12-13 19:47:00 Log ping...
2008-12-13 20:47:00 Log ping...
2008-12-13 21:47:00 Log ping...
2008-12-13 22:47:00 Log ping...
2008-12-13 23:47:00 Log ping...
2008-12-14 00:47:00 Log ping...
2008-12-14 01:47:00 Log ping...
2008-12-14 02:47:00 Log ping...
2008-12-14 03:47:00 Log ping...
2008-12-14 04:47:00 Log ping...
2008-12-14 05:47:00 Log ping...
2008-12-14 06:47:00 Log ping...
2008-12-14 07:47:00 Log ping...
2008-12-14 08:47:00 Log ping...
2008-12-14 09:47:00 Log ping...
2008-12-14 10:47:00 Log ping...
2008-12-14 11:47:00 Log ping...
2008-12-14 12:47:00 Log ping...
2008-12-14 13:47:00 Log ping...
2008-12-14 14:47:00 Log ping...
2008-12-14 15:47:00 Log ping...
2008-12-14 16:47:00 Log ping...
2008-12-14 17:47:00 Log ping...
2008-12-14 18:47:00 Log ping...
2008-12-14 19:47:00 Log ping...
2008-12-14 20:47:00 Log ping...
2008-12-14 21:47:00 Log ping...
2008-12-14 22:47:00 Log ping...
2008-12-14 23:47:00 Log ping...
2008-12-15 00:00:02 Running cleanup routine
2008-12-15 00:00:03 Module Stopped
2008-12-15 13:20:55 Module Started
2008-12-15 13:20:59 Time synchronized with scheduler
2008-12-15 13:21:03 ========================================
2008-12-15 13:21:03 Start connection to carrier: PageNet

PGGascii.log
2008-12-12 23:54:28 Log ping...
2008-12-12 23:54:35 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:54:42 Running cleanup routine
2008-12-12 23:54:42 Module Stopped
2008-12-12 23:55:02 Module Started
2008-12-12 23:55:07 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:55:27 Time synchronized with scheduler
2008-12-12 23:55:42 Log ping...
2008-12-12 23:55:57 Running cleanup routine
2008-12-12 23:55:57 Module Stopped
2008-12-12 23:56:14 Module Started
2008-12-12 23:56:20 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:56:39 Time synchronized with scheduler
2008-12-12 23:57:11 Log ping...
2008-12-12 23:57:19 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:57:24 Running cleanup routine
2008-12-12 23:57:24 Module Stopped
2008-12-12 23:57:40 Module Started
2008-12-12 23:57:43 Time synchronized with scheduler
2008-12-12 23:58:18 Log ping...
2008-12-12 23:58:25 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:58:30 Running cleanup routine
2008-12-12 23:58:30 Module Stopped
2008-12-12 23:58:53 Module Started
2008-12-12 23:58:59 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:59:16 Log ping...
2008-12-12 23:59:25 Unable to synchronize time with scheduler. Using local time.
2008-12-12 23:59:30 Running cleanup routine
2008-12-12 23:59:30 Module Stopped
2008-12-12 23:59:48 Module Started
2008-12-12 23:59:54 Unable to synchronize time with scheduler. Using local time.
2008-12-13 00:29:00 Log ping...
2008-12-13 01:29:00 Log ping...
2008-12-13 02:29:00 Log ping...
2008-12-13 03:29:00 Log ping...
2008-12-13 04:29:00 Log ping...
2008-12-13 05:29:00 Log ping...
2008-12-13 06:29:00 Log ping...
2008-12-13 07:29:00 Log ping...
2008-12-13 08:29:00 Log ping...
2008-12-13 09:29:00 Log ping...
2008-12-13 10:29:00 Log ping...
2008-12-13 11:29:00 Log ping...
2008-12-13 12:29:00 Log ping...
2008-12-13 13:29:00 Log ping...
2008-12-13 14:29:00 Log ping...
2008-12-13 15:29:00 Log ping...
2008-12-13 16:29:00 Log ping...
2008-12-13 17:29:00 Log ping...
2008-12-13 18:29:00 Log ping...
2008-12-13 19:29:00 Log ping...
2008-12-13 20:29:00 Log ping...
2008-12-13 21:29:00 Log ping...
2008-12-13 22:29:00 Log ping...
2008-12-13 23:29:00 Log ping...
2008-12-14 00:29:00 Log ping...
2008-12-14 01:29:00 Log ping...
2008-12-14 02:29:00 Log ping...
2008-12-14 03:29:00 Log ping...
2008-12-14 04:29:00 Log ping...
2008-12-14 05:29:00 Log ping...
2008-12-14 06:29:00 Log ping...
2008-12-14 07:29:00 Log ping...
2008-12-14 08:29:00 Log ping...
2008-12-14 09:24:55 Message accepted. Recipient: 5141102201 from: 00297024-00/LENDEMAIN/
2008-12-14 09:29:00 Log ping...
2008-12-14 10:29:00 Log ping...
2008-12-14 11:29:00 Log ping...
2008-12-14 12:29:00 Log ping...
2008-12-14 13:29:00 Log ping...
2008-12-14 14:29:00 Log ping...
2008-12-14 15:29:00 Log ping...
2008-12-14 16:29:00 Log ping...
2008-12-14 17:29:00 Log ping...
2008-12-14 18:29:00 Log ping...
2008-12-14 19:29:00 Log ping...
2008-12-14 20:29:00 Log ping...
2008-12-14 21:29:00 Log ping...
2008-12-14 22:29:00 Log ping...
2008-12-14 23:29:00 Log ping...
2008-12-15 00:00:05 Running cleanup routine
2008-12-15 00:00:07 Module Stopped
2008-12-15 13:20:54 Module Started
2008-12-15 13:20:56 Time synchronized with scheduler
2008-12-15 13:21:00 Message accepted. Recipient: 5141102328 from: 00297137-00/LENDEMAIN/

[OzCom]

It is designed to perform a process every 24 hours at midnight automatically. This includes a local backup of the database (renamed pagegate.bak) and an access inspect/fix.

_________________
Aaron D. Osgood
Streamline Communications, LLC
* Authorized Notepage, Inc. Development Partner and Re-seller
TEL: 207-518-8455
AOsgood@streamline-solutions.net
Creators of AutoPage Spillman CAD interface for PageGate

[tordaan]

Good luck in getting this resolved. I have been fighting this for over a year. All I have ever gotten is "update to the latest build". This has never resolved the issue.

We run 2 dedicated recent production Dell servers and still have the problem. This is the third set of servers and we have run the program on Win2K WS, Win2k Server, and most recently Win2K 2003.

My observations have been that the failure is most likely to occur when the combined daily log file and badmsg file sizes climb to unusual levels. In my most recent cases this was 9MB on one occasion and 11MB on another occasion. I do run all incoming traffic through our corporate gateway to remove as much spam as possible before it hits the servers. If I didn't do that, I suspect pagegate would fail every night.

In order to reduce the log file size, I have taken the following steps.

I run a scheduled batch job to dump the contents of the badmsg file at 11 PM every night. This seems to have made a big difference in the failure rate. In my case, if these messages failed once, requeuing them for retry is not going to get them to go. This file, for us at least, can get quite large over a several day period. If you are not filtering for spam on the front end, I suppose it can become enormous.

Most recently I changed logging level from detail to summary. I do not have enough time with this setting to clearly state whether this will make a difference. However, if I am correct, this should make a dramatic difference. I checked the documentation, an find nothing that mentions performance degradation based on logging level, but experience dictates that the more logging, the greater the impact on performance.

My opinion is that this product for all the other nice features and the general good job that it does, does have this one stability issue that needs to be addressed.

[OzCom]

Under the LOGGINC section of ADMIN - what do you have the LOG LEVEL, the LOG SIZE, and the ARCHIVE LOGS settings at?

Also, is this version 4 or 5?

_________________
Aaron D. Osgood
Streamline Communications, LLC
* Authorized Notepage, Inc. Development Partner and Re-seller
TEL: 207-518-8455
AOsgood@streamline-solutions.net
Creators of AutoPage Spillman CAD interface for PageGate

[tordaan]

I've been running 5.0.8, but just moved my server with the lowest MX priority up to 5.0.9 at Notepages request. On Monday I will change MX priority to make this the highest priority server for test.

Also, I was logging detail but just changed to summary. Archive is set, so max file size is not applicable. File size is whatever daily size is. This is typically 4 to 6 MB total logging less badmsg log. After changing to summary, this total logging has dropped by about 60%. This may be an interim solution, but the fact still remains that the connector should not rollover and die based on log size.

There is no documentation that I have found concerning perfomance degradation based on logging level, however, I still feel that the log size in this configuration is the trigger. The connector failure occurs when we have had a high traffic day. 9MB or higher total logging for the day. Keep in mind that we pre filter all incoming SMTP traffic for spam.

I should mention that the only paging traffic that we run through these machines is mostly ADHOC SMTP with some alias addressing. We do not use the Web or client interface. Our SMTP traffic is probably far higher than the average Pagegate user. We are a major west coast regional paging carrier.

Also, Our connectors are set up for snpp and send to snpp servers on our LAN.

NotePage is now working with us and hopefully we will get to the bottom of this issue soon.

[jganley]

Tordaan, Did you ever hear back from Note Page?

[tordaan]

I have been working with Notepage on this issue. I am bginning to believe that database size may have a bearing on this. I have since reduced the archive to 30 days from previous higher levels and we seem to have stablized. We are watching the perfomance very closely and running debug logging to try and catch any anomolies.

We are fairly large West Coast regional paging carrier and probably handle much higher traffic than most uers.

If you are having an issue, I strongly reccomend that you run Notepage Pro in conjuction with the pagegate monitor to alert you if any of the services lock up during the midnight roll over. This works quite well.