| Author |
Message
|
| sjclark1 |
 Posted: Thu May 19, 2011 1:42 am Post subject: What causes /var/mqm/log to fill up? |
 |
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Can anyone explain to me what causes either of the following partitions to fill up on linux:
/var/mqm/log
/var/mqm
I'd imagine the reasons for each partition filling up will be different.
I am not an MQ admin -- I don't have access to most of the directory structure. Basically we get occasional 2102 errors, and the MQ admin said something about the disk being full and having to be cleaned up every few hours. He didn't say which of the two partitions was full.
The queue manager itself is used by a large number of people so I don't have visibility of what everyone is doing.
But I am being pressed to stop these 2102 errors and I want to know whether that is being caused by the disk filling up.... and then I need to know what can cause the logs to fill up.
Side question: where do persistent messages get persisted to? Do they go on the filesystem? Excuse my ignorance.
Last edited by sjclark1 on Mon May 23, 2011 1:09 am; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu May 19, 2011 8:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
2102 shows a resource problem. A disk being full would be one of the explanations....
Now you need to provision enough space for
/var/mqm/qmgrs This is where the messages are stored in the queues
/var/mqm/log This is where the transactional logs are written to.
Persistent messages are written to the logs and to the queue if not immediately consumed. Make also sure that if any of the qmgr on the box is using linear logging, that the MQ Admin is also running a support pack to maintain and archive unused logs (ex MS0L)
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu May 19, 2011 9:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
/var/mqm/errors can start filling up with files that end with the .FDC extension. If there is an ongoing problem, these files can be enormous and / or there can be thousands of them.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu May 19, 2011 11:05 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
From the manual entry for 2102 returncode:
| Quote: |
| On HP OpenVMS, OS/2, i5/OS, Compaq NonStop Kernel, and UNIX systems, consult the FFST record to obtain more detail about the problem. |
so there should be a FDC file ?!? telling you more detals about the problem
_________________
Regards, Butcher |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Mon May 23, 2011 1:01 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
| fjb_saper wrote: |
/var/mqm/log This is where the transactional logs are written to.
|
OK thanks -- so presumably if someone is loading a large number of messages before committing, could that be causing the issue?
We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from?
Thanks to everyone else who has replied. |
|
| Back to top |
|
|
| fschofer |
| Posted: Mon May 23, 2011 2:46 am Post subject: |
|
|
Knight
Joined: 02 Jul 2001
Posts: 524
Location: Mainz, Germany
|
Hi,
first thing you need to know is if your queue manager are using circular or linear logging. Take a look the qm.ini files of your queue managers.
If they are using circular logging then your logging file systems needs to hold enough space to hold the configured primary + secondary log files.
If they are using linear logging then additional log files will pile up
and superfluous log files have to be archived / deleted in a scheduled way.
See here for more details about MQ transaction logs
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa14480_.htm
Regards
Frank |
|
| Back to top |
|
|
| ipmqadm |
| Posted: Mon May 23, 2011 6:13 am Post subject: |
|
|
Acolyte
Joined: 18 Apr 2007
Posts: 68
|
"We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from?"
In my opinion, you really need to check your qm.ini file to see what type of logging and how many log files you have. Hopefully you are using circular logging and have sufficient bufferspace defined.
You can also have your mq admin start an mq trace on the qmgr too. He will need to research the mq trace details as it CAN also create out-of-space issues, but if executed and monitored correctly the mq trace will definately show you 'where these large transactions are running from". |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Mon May 23, 2011 9:16 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
| Quote: |
| We are basically a middleware app and we provide MQ infrastructure to other apps within the organisation... any tips on how we might find where these large transactions are running from? |
running some script frequently issuing a display for queues with UCOM(YES) (uncomitted messages) may also help do identify the queue / application that holds a uow for a long time. of course you will occasionally also hit small uows.......
_________________
Regards, Butcher |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Tue Apr 10, 2012 10:02 pm Post subject: Identifying Unsed Logs |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Dear All,
I am facing a similar issue where the disk is running out of space due to MQ logs. Am working in Unix environment and MQ 6.0. I have a queue manager handling 1 million messages/day. That too Persistent Messages. To handle this large number of messages Primary logs were kept as 200 and secondary as 100. I am using circular logging. But now we made a change in the design so that the messages are diverted through another path and hence we reduced the number of primary and secondary to 10 & 10 resp. Now the issue is, i have lot files in /var/mqm/log/<QMGR>/active path. How can i find out the unused log files from this? I got to know from one of the posts that we have a command to find out the last file in use in case of linear logging. Do we have anything similar for Circular Logging. Please help.
qm.ini file is as below
#*******************************************************************#
#* Module Name: qm.ini *#
#* Type : WebSphere MQ queue manager configuration file *#
# Function : Define the configuration of a single queue manager *#
#* *#
#*******************************************************************#
#* Notes : *#
#* 1) This file defines the configuration of the queue manager *#
#* *#
#*******************************************************************#
ExitPath:
ExitsDefaultPath=/var/mqm/exits/
ExitsDefaultPath64=/var/mqm/exits64/
#* *#
#* *#
Log:
LogPrimaryFiles=10
LogSecondaryFiles=10
LogFilePages=1024
LogType=CIRCULAR
LogBufferPages=0
LogPath=/var/mqm/log/TST!EXT!QMGR/
LogWriteIntegrity=TripleWrite
Service:
Name=AuthorizationService
EntryPoints=13
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm/lib64/amqzfu
ComponentDataSize=0
Regards,
Sarath |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 11, 2012 2:52 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
Are you the system admin? Are you the WMQ admin? You, or your WMQ admin, need to calculate the size of logs required for each qmgr.
Log sizes are calculated as described in the WMQ System Administration documentation. The calculation involves the actual message data length + MQMD length * number of messages.
Circular log segments are overwritten (reused) when they no longer contain active units of work.
Have you read the WMQ System Administration manual (or equivalent InfoCenter) documentation?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Wed Apr 11, 2012 4:12 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Dear Bruce, Thanks for your reply.
But, unfortunately ur reply didn't give me the solution to the issue mnetioned below. My question was, with circular logging, can i find out the unused files in the QMGR log directory. ie, if i had initially 100 Primary files and its reduced to 10 now, by any means can i find out the unused ones ?
Regards,
Sarath |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Apr 11, 2012 4:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It is generally an unsupported task to alter the number of primary and secondary log files on a queue manager.
It is generally useable to *increase* this number. It is more complicated to *decrease* this number.
The recommended practice in all cases is to recreate the queue manager with the new log values.
In your case, I would restart the queue manager, and note the date and times of log file modification and their file sizes.
Then I would wait a week and delete any files that have not changed. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Apr 11, 2012 7:27 am Post subject: Re: Identifying Unsed Logs |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| sarathmattam wrote: |
| Now the issue is, i have lot files in /var/mqm/log/<QMGR>/active path. How can i find out the unused log files from this? |
Leave the qmgr for a few days, come back and look again. MQ might remove the files it no longer needs. No guarantees, though.
But you should not remove the files manually!
If MQ does not remove the files itself, then the supported action is to delete/recreate the queue manager with the new parameters.
On the other hand, it is a pity that you have to do so much work/planning on this when disk space is so cheap. I think your time is worth more $$ than the disk space you would save. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Apr 11, 2012 8:15 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
I've dealt with this problem before. We altered the # of primary and seconday log files down for a QM with Circular Logging. No change after a day. Restarted the QM, no change. Opened a PMR. Once I drove enough traffic thru the QM for the QM to have to 'touch' all the log files*, only then did it start cleaning them up. Eventually the QM did shrink the number of log files down to the new values I put in. It took a while though. And the PMR did not offer any faster way.
This was a few years ago. Newer versions of MQ might deal with this differently and more quickly.
* I just wrote a little looping program that put a message under syncpoint, commited, deleted the message under syncpoint, committed, and round and round. This forced the QM to advance thru the log files. It took a long time. The logs were huge, and the QM had very little other activity.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 11, 2012 8:41 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
Circular logs are re-used. Primary logs are allocated and formatted at qmgr creation.
Secondary logs are allocated, formatted and used, where/when/if needed because primary logs were exhausted (contained active units of work).
The theory is that, when no longer needed, secondary logs are deleted.
Running out of disk space means that you need more disk space.
Running out of primary logs (having many secondary logs) means that you didn't go a good (enough) job of calculating primary log file size and number requirements. Having a few secondary logs likely means that you hit a spike in workload.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
