ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » What causes /var/mqm/log to fill up?

Post new topic  Reply to topic Goto page 1, 2  Next
 What causes /var/mqm/log to fill up? « View previous topic :: View next topic » 
Author Message
sjclark1
PostPosted: Thu May 19, 2011 1:42 am    Post subject: What causes /var/mqm/log to fill up? Reply with quote

Apprentice

Joined: 06 Jan 2003
Posts: 35
Location: Scotland

Can anyone explain to me what causes either of the following partitions to fill up on linux:

/var/mqm/log
/var/mqm

I'd imagine the reasons for each partition filling up will be different.

I am not an MQ admin -- I don't have access to most of the directory structure. Basically we get occasional 2102 errors, and the MQ admin said something about the disk being full and having to be cleaned up every few hours. He didn't say which of the two partitions was full.

The queue manager itself is used by a large number of people so I don't have visibility of what everyone is doing.

But I am being pressed to stop these 2102 errors and I want to know whether that is being caused by the disk filling up.... and then I need to know what can cause the logs to fill up.

Side question: where do persistent messages get persisted to? Do they go on the filesystem? Excuse my ignorance.


Last edited by sjclark1 on Mon May 23, 2011 1:09 am; edited 1 time in total
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu May 19, 2011 8:01 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

2102 shows a resource problem. A disk being full would be one of the explanations....

Now you need to provision enough space for
/var/mqm/qmgrs This is where the messages are stored in the queues
/var/mqm/log This is where the transactional logs are written to.

Persistent messages are written to the logs and to the queue if not immediately consumed. Make also sure that if any of the qmgr on the box is using linear logging, that the MQ Admin is also running a support pack to maintain and archive unused logs (ex MS0L)

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
PeterPotkay
PostPosted: Thu May 19, 2011 9:25 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

/var/mqm/errors can start filling up with files that end with the .FDC extension. If there is an ongoing problem, these files can be enormous and / or there can be thousands of them.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Mr Butcher
PostPosted: Thu May 19, 2011 11:05 pm    Post subject: Reply with quote

Padawan

Joined: 23 May 2005
Posts: 1716

From the manual entry for 2102 returncode:

Quote:
On HP OpenVMS, OS/2, i5/OS, Compaq NonStop Kernel, and UNIX systems, consult the FFST record to obtain more detail about the problem.


so there should be a FDC file ?!? telling you more detals about the problem
_________________
Regards, Butcher
Back to top
View user's profile Send private message
sjclark1
PostPosted: Mon May 23, 2011 1:01 am    Post subject: Reply with quote

Apprentice

Joined: 06 Jan 2003
Posts: 35
Location: Scotland

fjb_saper wrote:

/var/mqm/log This is where the transactional logs are written to.


OK thanks -- so presumably if someone is loading a large number of messages before committing, could that be causing the issue?

We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from?

Thanks to everyone else who has replied.
Back to top
View user's profile Send private message
fschofer
PostPosted: Mon May 23, 2011 2:46 am    Post subject: Reply with quote

Knight

Joined: 02 Jul 2001
Posts: 524
Location: Mainz, Germany

Hi,

first thing you need to know is if your queue manager are using circular or linear logging. Take a look the qm.ini files of your queue managers.

If they are using circular logging then your logging file systems needs to hold enough space to hold the configured primary + secondary log files.

If they are using linear logging then additional log files will pile up
and superfluous log files have to be archived / deleted in a scheduled way.

See here for more details about MQ transaction logs
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa14480_.htm

Regards
Frank
Back to top
View user's profile Send private message Send e-mail
ipmqadm
PostPosted: Mon May 23, 2011 6:13 am    Post subject: Reply with quote

Acolyte

Joined: 18 Apr 2007
Posts: 68

"We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from?"

In my opinion, you really need to check your qm.ini file to see what type of logging and how many log files you have. Hopefully you are using circular logging and have sufficient bufferspace defined.

You can also have your mq admin start an mq trace on the qmgr too. He will need to research the mq trace details as it CAN also create out-of-space issues, but if executed and monitored correctly the mq trace will definately show you 'where these large transactions are running from".
Back to top
View user's profile Send private message
Mr Butcher
PostPosted: Mon May 23, 2011 9:16 pm    Post subject: Reply with quote

Padawan

Joined: 23 May 2005
Posts: 1716

Quote:
We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from?


running some script frequently issuing a display for queues with UCOM(YES) (uncomitted messages) may also help do identify the queue / application that holds a uow for a long time. of course you will occasionally also hit small uows.......
_________________
Regards, Butcher
Back to top
View user's profile Send private message
sarathmattam
PostPosted: Tue Apr 10, 2012 10:02 pm    Post subject: Identifying Unsed Logs Reply with quote

Voyager

Joined: 05 Sep 2008
Posts: 94


Dear All,

I am facing a similar issue where the disk is running out of space due to MQ logs. Am working in Unix environment and MQ 6.0. I have a queue manager handling 1 million messages/day. That too Persistent Messages. To handle this large number of messages Primary logs were kept as 200 and secondary as 100. I am using circular logging. But now we made a change in the design so that the messages are diverted through another path and hence we reduced the number of primary and secondary to 10 & 10 resp. Now the issue is, i have lot files in /var/mqm/log/<QMGR>/active path. How can i find out the unused log files from this? I got to know from one of the posts that we have a command to find out the last file in use in case of linear logging. Do we have anything similar for Circular Logging. Please help.

qm.ini file is as below

#*******************************************************************#
#* Module Name: qm.ini *#
#* Type : WebSphere MQ queue manager configuration file *#
# Function : Define the configuration of a single queue manager *#
#* *#
#*******************************************************************#
#* Notes : *#
#* 1) This file defines the configuration of the queue manager *#
#* *#
#*******************************************************************#
ExitPath:
ExitsDefaultPath=/var/mqm/exits/
ExitsDefaultPath64=/var/mqm/exits64/
#* *#
#* *#
Log:
LogPrimaryFiles=10
LogSecondaryFiles=10
LogFilePages=1024
LogType=CIRCULAR
LogBufferPages=0
LogPath=/var/mqm/log/TST!EXT!QMGR/
LogWriteIntegrity=TripleWrite
Service:
Name=AuthorizationService
EntryPoints=13
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm/lib64/amqzfu
ComponentDataSize=0


Regards,
Sarath
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 11, 2012 2:52 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Are you the system admin? Are you the WMQ admin? You, or your WMQ admin, need to calculate the size of logs required for each qmgr.

Log sizes are calculated as described in the WMQ System Administration documentation. The calculation involves the actual message data length + MQMD length * number of messages.

Circular log segments are overwritten (reused) when they no longer contain active units of work.

Have you read the WMQ System Administration manual (or equivalent InfoCenter) documentation?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sarathmattam
PostPosted: Wed Apr 11, 2012 4:12 am    Post subject: Reply with quote

Voyager

Joined: 05 Sep 2008
Posts: 94

Dear Bruce, Thanks for your reply.

But, unfortunately ur reply didn't give me the solution to the issue mnetioned below. My question was, with circular logging, can i find out the unused files in the QMGR log directory. ie, if i had initially 100 Primary files and its reduced to 10 now, by any means can i find out the unused ones ?

Regards,
Sarath
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Apr 11, 2012 4:26 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

It is generally an unsupported task to alter the number of primary and secondary log files on a queue manager.

It is generally useable to *increase* this number. It is more complicated to *decrease* this number.

The recommended practice in all cases is to recreate the queue manager with the new log values.

In your case, I would restart the queue manager, and note the date and times of log file modification and their file sizes.

Then I would wait a week and delete any files that have not changed.
Back to top
View user's profile Send private message
mvic
PostPosted: Wed Apr 11, 2012 7:27 am    Post subject: Re: Identifying Unsed Logs Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2080

sarathmattam wrote:
Now the issue is, i have lot files in /var/mqm/log/<QMGR>/active path. How can i find out the unused log files from this?

Leave the qmgr for a few days, come back and look again. MQ might remove the files it no longer needs. No guarantees, though.

But you should not remove the files manually!

If MQ does not remove the files itself, then the supported action is to delete/recreate the queue manager with the new parameters.

On the other hand, it is a pity that you have to do so much work/planning on this when disk space is so cheap. I think your time is worth more $$ than the disk space you would save.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Apr 11, 2012 8:15 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

I've dealt with this problem before. We altered the # of primary and seconday log files down for a QM with Circular Logging. No change after a day. Restarted the QM, no change. Opened a PMR. Once I drove enough traffic thru the QM for the QM to have to 'touch' all the log files*, only then did it start cleaning them up. Eventually the QM did shrink the number of log files down to the new values I put in. It took a while though. And the PMR did not offer any faster way.

This was a few years ago. Newer versions of MQ might deal with this differently and more quickly.


* I just wrote a little looping program that put a message under syncpoint, commited, deleted the message under syncpoint, committed, and round and round. This forced the QM to advance thru the log files. It took a long time. The logs were huge, and the QM had very little other activity.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Apr 11, 2012 8:41 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Circular logs are re-used. Primary logs are allocated and formatted at qmgr creation.

Secondary logs are allocated, formatted and used, where/when/if needed because primary logs were exhausted (contained active units of work).

The theory is that, when no longer needed, secondary logs are deleted.

Running out of disk space means that you need more disk space.

Running out of primary logs (having many secondary logs) means that you didn't go a good (enough) job of calculating primary log file size and number requirements. Having a few secondary logs likely means that you hit a spike in workload.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » What causes /var/mqm/log to fill up?
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.