Author |
Message
|
MQMB&WAS |
Posted: Tue Nov 12, 2019 12:37 pm Post subject: Long running transactions |
|
|
Centurion
Joined: 12 Jun 2016 Posts: 130
|
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?
Once found the culprit transaction, what can a mq admin do about it?
Couldn't find the info online, appreciate your inputs.
Thanks |
|
Back to top |
|
|
Vitor |
Posted: Tue Nov 12, 2019 1:32 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
One quick way is to look for a local queue with a high depth & UNCOM(YES), then ask the application that puts to that queue if there's anything they should be telling you.
(You can of course identify this queue from your documentation, the naming convention of the queue or one of the other controls you have in place......) _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
bruce2359 |
Posted: Tue Nov 12, 2019 1:33 pm Post subject: Re: Long running transactions |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
MQMB&WAS wrote: |
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem? |
Control command dspmqtrn
MQMB&WAS wrote: |
Once found the culprit transaction, what can a mq admin do about it? |
Cancel the offending application.
MQMB&WAS wrote: |
Couldn't find the info online, appreciate your inputs.
Thanks |
really? I googled 'mq long running transactions' and found it. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
hughson |
Posted: Wed Nov 13, 2019 2:18 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013 Posts: 1948 Location: Bay of Plenty, New Zealand
|
|
Back to top |
|
|
rekarm01 |
Posted: Wed Nov 13, 2019 6:48 pm Post subject: Re: Long running transactions |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 1415
|
MQMB&WAS wrote: |
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem? |
If the queue manager forces a commit/backout to resolve a long-running transaction, then shouldn't it be fairly easy to identify the application responsible, from the MQ error logs?
Vitor wrote: |
One quick way is to look for a local queue with a high depth & UNCOM(YES), then ask the application that puts to that queue if there's anything they should be telling you. |
Looking for a queue with high depth doesn't always work, when a queue manager has low volume queues with occasionally long running transactions, but also has very high volume queues with short running transactions. And it doesn't have to be that long; one long running transaction could fill up the log space within a matter of minutes, without filling up any queues. |
|
Back to top |
|
|
hughson |
Posted: Wed Nov 13, 2019 8:42 pm Post subject: Re: Long running transactions |
|
|
Padawan
Joined: 09 May 2013 Posts: 1948 Location: Bay of Plenty, New Zealand
|
MQMB&WAS wrote: |
Once found the culprit transaction, what can a mq admin do about it? |
One other option if you can't identify the culprit to talk to them in person, might be the STOP CONN command, but that will also depend on whether 1PC or 2PC transaction, i.e. whether they are still connected.
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
|
gbaddeley |
Posted: Thu Nov 14, 2019 3:04 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003 Posts: 2527 Location: Melbourne, Australia
|
Run "display qstatus('*') type(queue) where(uncom ne no) all".
For each queue that has uncommitted messages, run
"display qstatus('QNAME') type(handle) all" and correlate against
"display conn(*) type(conn) where(qmurid ne 0.0) all"
Look for oldest time when log was last written (UOWLOGDA & UOWLOGTI).
This will be your long running UOW. _________________ Glenn |
|
Back to top |
|
|
bruce2359 |
Posted: Fri Nov 15, 2019 5:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
What errors did you see in the active qmgr error log?
I'd have expected an error written to the active qmgr error log, such as AMQ6709: The log for the queue manager is full.
I'd have expected an error returned to the app, such as an MQCC_FAILED with 2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM. Developers are responsible for catching a wide variety of ReasonCodes.
If you are lucky, there may be only one badly behaving app filling your logs. Been there, seen this.
Any new or recently modified apps? It's also possible that a single app with multiple concurrent users is at play here. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
Andyh |
Posted: Mon Nov 18, 2019 9:21 am Post subject: |
|
|
Master
Joined: 29 Jul 2010 Posts: 239
|
The queue manager should have dealt with the long running transaction (either backed out, or rolled forwards (if indoubt)) BEFORE the logs actually became full. If the logs did really fill to capacity (new work being refused due to log resouce constraints) as a result of a long running transaction then you should raise a service ticket with IBM.
A more common reason for a shortage of log space is that the queue manager has been created with linear logging, and the linear logs created are not being properly managed.
Management of linear log extents was a customer responsibility until 9.0.2, at which point the QMgr introduced options to allow automatic log management. |
|
Back to top |
|
|
gbaddeley |
Posted: Mon Nov 18, 2019 3:21 pm Post subject: Re: Long running transactions |
|
|
Jedi Knight
Joined: 25 Mar 2003 Posts: 2527 Location: Melbourne, Australia
|
MQMB&WAS wrote: |
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?
Once found the culprit transaction, what can a mq admin do about it?
Couldn't find the info online, appreciate your inputs.
Thanks |
It can be too late if the file system is full, as the qmgr may no longer be responsive for you to do anything useful.
The only alternative may be to end the qmgr, increase the number of logs, increase the file system size, and start the qmgr. Start up may take a long time, as the qmgr may need to reconcile all the logs.
Assuming you have a separate file system for the active logs, you should have alerting that detects say 80% or 90% full. This gives time to investigate and resolve the issue.
You should also have alerting on the qmgr error logs for forced roll back (AMQ7469) or roll forward (AMQ7485) or existence (AMQ7486) of long running transactions.
HTH _________________ Glenn |
|
Back to top |
|
|
HubertKleinmanns |
Posted: Fri Nov 22, 2019 12:11 am Post subject: Re: Long running transactions |
|
|
Shaman
Joined: 24 Feb 2004 Posts: 732 Location: Germany
|
gbaddeley wrote: |
MQMB&WAS wrote: |
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?
Once found the culprit transaction, what can a mq admin do about it?
Couldn't find the info online, appreciate your inputs.
Thanks |
It can be too late if the file system is full, as the qmgr may no longer be responsive for you to do anything useful.
The only alternative may be to end the qmgr, increase the number of logs, increase the file system size, and start the qmgr. Start up may take a long time, as the qmgr may need to reconcile all the logs.
Assuming you have a separate file system for the active logs, you should have alerting that detects say 80% or 90% full. This gives time to investigate and resolve the issue.
You should also have alerting on the qmgr error logs for forced roll back (AMQ7469) or roll forward (AMQ7485) or existence (AMQ7486) of long running transactions.
HTH |
You could have a look at the error logs (AMQERR01.LOG in the QMgrs data volume), have a look, which logs are still needed:
- AMQ7467 Logs needed for Restart.
- AMQ7468 Logs needed for Media Recovery.
You could at least remove the logs with a number lower than shown at AMQ7468 without any loss of functionality.
You could also remove the logs with a number lower than shown at AMQ7467, but you wouldn't be able ro perform media recovery (repairing damaged queues). _________________ Regards
Hubert |
|
Back to top |
|
|
bruce2359 |
Posted: Fri Nov 22, 2019 4:12 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
Odd .. the OP hasn’t offered details or responded to questions we’ve posed. I’m wondering if this is a college or tech school research assignment.
When did this symptom begin? What apps were new or recently updated just before the symptom appeared? What is the configuration of the qmgr experiencing the symptom? Was/is this post a theoretical issue?
I hope we get a good grade from the professor. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
|