ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » AMQ7469 and the missing log file directory /var/mqm/log/Qxxx

Post new topic  Reply to topic
 AMQ7469 and the missing log file directory /var/mqm/log/Qxxx « View previous topic :: View next topic » 
Author Message
LouML
PostPosted: Wed Jul 18, 2007 9:53 am    Post subject: AMQ7469 and the missing log file directory /var/mqm/log/Qxxx Reply with quote

Partisan

Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY

Just been made aware of the following messages in the error log.

-------------------------------------------------------------------------------
07/18/07 13:38:18
AMQ7469: Transactions rolled back to release log space.

EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive. Consider
increasing the size of the log to allow transactions to last longer before the
log starts to become full.
-------------------------------------------------------------------------------


They seem to have been going on as far back as the three AMQERR_.LOG files can show (and most likely a lot longer).

bash-2.05$ ls -la /var/mqm/qmgrs/QM\!jefomsapp\!01/errors
total 1572
drwxrwxrwx 2 mqm mqm 1024 Jul 17 10:05 .
drwxrwxr-x 18 mqm mqm 1024 Jul 15 06:27 ..
-rw-rw-rw- 1 mqm mqm 247550 Jul 18 13:40 AMQERR01.LOG
-rw-rw-rw- 1 mqm mqm 256305 Jul 17 10:03 AMQERR02.LOG
-rw-rw-rw- 1 mqm mqm 256412 Jul 13 13:28 AMQERR03.LOG
bash-2.05$


I went to look at the linear log files, but to my surprise, they do not exist. Not sure when or how the got deleted.

bash-2.05$ ls -la /var/mqm/log
total 4
drwxrwxr-x 2 mqm mqm 512 May 19 2005 .
drwxrwxr-x 11 mqm mqm 512 Feb 15 19:46 ..
bash-2.05$


I know I should see something like the following (from another system)

bash-3.00$ ls -laF /var/mqm/log/QM\!eqdbln2p\!01/active/
total 41124
drwxrwx--- 2 mqm mqm 512 Jul 11 14:58 ./
drwxrwx--- 3 mqm mqm 512 Jul 11 14:58 ../
-rw-rw---- 1 mqm mqm 4202496 Jul 18 12:24 S0000000.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 18 12:29 S0000001.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 18 12:34 S0000002.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 18 12:37 S0000003.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 18 12:21 S0000004.LOG
bash-3.00$


I know at a minimum, I will have to stop and restart the queue manager. My question is:

1 - Will the proper log file directory get re-created by a restart or will I have to delete and re-create the entire queue manager?

2 - Would I be able to create the directories manually and then create empty s0000000.log files?

3 - Would this problem contribute to an apparant slow-down of mq message traffic? I don't see a slow down myself, but a user complained that mq slowed down for a 15 minute period this morning.
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Wed Jul 18, 2007 10:11 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

LouML wrote:
1 - Will the proper log file directory get re-created by a restart or will I have to delete and re-create the entire queue manager?


You can do a "cold" start. Search here or IBM's site.

LouML wrote:
2 - Would I be able to create the directories manually and then create empty s0000000.log files?

That's what the coldstart does.

I suspect, maybe, that the filesystem hosting the log files has been unmounted, or another filesystem mounted over the existing log directory.

LouML wrote:
3 - Would this problem contribute to an apparant slow-down of mq message traffic? I don't see a slow down myself, but a user complained that mq slowed down for a 15 minute period this morning.

You're lucky it didn't cause everything to blow up and come to a screeching halt. Yes, avaliability of log space will cause response time impact.
_________________
I am *not* the model of the modern major general.
Back to top
View user's profile Send private message
LouML
PostPosted: Wed Jul 18, 2007 10:16 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Partisan

Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY

jefflowrey wrote:
LouML wrote:
1 - Will the proper log file directory get re-created by a restart or will I have to delete and re-create the entire queue manager?


You can do a "cold" start. Search here or IBM's site.

LouML wrote:
2 - Would I be able to create the directories manually and then create empty s0000000.log files?

That's what the coldstart does.

I suspect, maybe, that the filesystem hosting the log files has been unmounted, or another filesystem mounted over the existing log directory.

LouML wrote:
3 - Would this problem contribute to an apparant slow-down of mq message traffic? I don't see a slow down myself, but a user complained that mq slowed down for a 15 minute period this morning.

You're lucky it didn't cause everything to blow up and come to a screeching halt. Yes, avaliability of log space will cause response time impact.


Thank you Jeff, for the very quick response.
Back to top
View user's profile Send private message
LouML
PostPosted: Thu Jul 19, 2007 4:11 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Partisan

Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY

Latest info - I emailed our backup/restore manager to see if we could restore the missing files and he replied with the following:

That directory (/var/mqm/log) appears to be empty since the beginning of the year (which is as long as we have tapes)

Note that this machine is clustered, so the directories are set up in different places than the standard MQ install. Is this the directory you are looking for?

/MQHA/QM.jefomsapp.01/log/QM!jefomsapp!01/active


As a newbie here, and not having worked with clusters before, does the following mean that I'm not missing any log files? If so, why would I get the AMQ7469's?

bash-2.05$ ls -la /var/mqm/qmgrs/QM\!jefomsapp\!01/
lrwxrwxrwx 1 mqm mqm 48 Jul 5 2005 /var/mqm/qmgrs/QM!jefomsapp!01/ -> /MQHA/QM.jefomsapp.01/data/qmgrs/QM!jefomsapp!01
bash-2.05$ ls -la /MQHA/QM.jefomsapp.01/log/QM\!jefomsapp\!01/active
total 57458
drwxrwx--- 2 mqm mqm 1024 Mar 23 07:47 .
drwxrwx--- 3 mqm mqm 96 Jul 5 2005 ..
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:30 S0000000.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:33 S0000001.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:00 S0000002.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 18 22:09 S0000003.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:31 S0000004.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:34 S0000005.LOG
-rw-rw---- 1 mqm mqm 4202496 Jul 19 07:59 S0000006.LOG
bash-2.05$

The logs are linear and I don't appear to be out of disk space.

bash-2.05$ df -h /MQHA/QM.jefomsapp.01/log
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/mqmdg/mqmlog
6.0G 46M 5.6G 1% /MQHA/QM.jefomsapp.01/log
bash-2.05$ df -h /var/mqm/log
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d15 7.9G 5.7G 2.1G 73% /var
bash-2.05$
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 19, 2007 4:26 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

LouML wrote:

As a newbie here, and not having worked with clusters before, does the following mean that I'm not missing any log files? If so, why would I get the AMQ7469's?


I would theorise that he means a high availability cluster not an MQ one, and the logs have been placed on this (presumably shared) location to allow failover.

You're getting AMQ7469s because there's no more log space, just like the message says. Linear logging just means logs are not reused to allow media recording and recovery; the queue manager will still have been given a primary/secondary log allocation and it's full of uncommitted transactions.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
LouML
PostPosted: Thu Jul 19, 2007 4:53 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Partisan

Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY

Vitor wrote:
LouML wrote:

As a newbie here, and not having worked with clusters before, does the following mean that I'm not missing any log files? If so, why would I get the AMQ7469's?


I would theorise that he means a high availability cluster not an MQ one, and the logs have been placed on this (presumably shared) location to allow failover.

You're getting AMQ7469s because there's no more log space, just like the message says. Linear logging just means logs are not reused to allow media recording and recovery; the queue manager will still have been given a primary/secondary log allocation and it's full of uncommitted transactions.


Thank you.

So, how do the 7 log files (S0000000.LOG-S0000006.LOG) in the MQHA directory correlate to the 5 total files (LogPrimaryFiles=3 and LogSecondaryFiles=2) entries in my mqs.ini file?

LogDefaults:
LogPrimaryFiles=3
LogSecondaryFiles=2
LogFilePages=1024
LogType=LINEAR
LogBufferPages=17
LogDefaultPath=/var/mqm/log

As far as a solution is concerned, I assume that I have to:
1 - Get the users to figure out which app is not committing messages properly
2 - Change the mqs.ini file to allow more log files
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 19, 2007 4:57 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

LouML wrote:

So, how do the 7 log files (S0000000.LOG-S0000006.LOG) in the MQHA directory correlate to the 5 total files (LogPrimaryFiles=3 and LogSecondaryFiles=2) entries in my mqs.ini file?


3 primary, 2 secondary, 2 no longer used. You should be getting messages explaining which files can be archived.

LouML wrote:

As far as a solution is concerned, I assume that I have to:
1 - Get the users to figure out which app is not committing messages properly
2 - Change the mqs.ini file to allow more log files


I'd say 1 is more important than 2. Depending on how the application is failing to commit, it's plausable that it can fill any amount of log.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
LouML
PostPosted: Thu Jul 19, 2007 5:10 am    Post subject: Re: AMQ7469 and the missing log file directory /var/mqm/log/ Reply with quote

Partisan

Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY

Vitor wrote:
LouML wrote:

So, how do the 7 log files (S0000000.LOG-S0000006.LOG) in the MQHA directory correlate to the 5 total files (LogPrimaryFiles=3 and LogSecondaryFiles=2) entries in my mqs.ini file?


3 primary, 2 secondary, 2 no longer used. You should be getting messages explaining which files can be archived.

LouML wrote:

As far as a solution is concerned, I assume that I have to:
1 - Get the users to figure out which app is not committing messages properly
2 - Change the mqs.ini file to allow more log files


I'd say 1 is more important than 2. Depending on how the application is failing to commit, it's plausable that it can fill any amount of log.


Thank you for the prompt reply.

I assume you mean either an AMQ7467 or AMQ7468 message regarding archiving the log files. When I grep the log files for the AMQ7467 or AMQ7468 messages I do not find them:

bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR03.LOG | grep -i AMQ7467
bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR02.LOG | grep -i AMQ7467
bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR01.LOG | grep -i AMQ7467
bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR01.LOG | grep -i AMQ7468
bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR02.LOG | grep -i AMQ7468
bash-2.05$ cat /var/mqm/qmgrs/QM\!jefomsapp\!01/errors/AMQERR03.LOG | grep -i AMQ7468
bash-2.05$

Either way, I will follow up with the application people to see if they can find anything. Of course, that's the next problem - trying to figure out which app may be causing this.

Thank you again for your help.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » AMQ7469 and the missing log file directory /var/mqm/log/Qxxx
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.