| Author |
Message
|
| dutchman |
 Posted: Mon Jul 18, 2011 10:52 pm Post subject: SCTQ UNCOM(YES) but no input/output processes |
 |
|
Acolyte
Joined: 15 May 2001
Posts: 71
Location: Netherlands
|
Ladies/Gents ... interesting problem here. Our Full Repos SYSTEM.CLUSTER.TRANSMIT.QUEUE is monitored by the IBM Tivoli product. We recently activated a new situation in which the oldest msg on the SCTQ was checked. If it was older than,say, 2 minutes (3 times) then an alert would be raised.
This was done last Friday. On Monday I noticed the alert had been raised but there were no msgs on the queue - not that I could initially see anyway.
Turns out that there was an uncommitted msg on the queue:
dis qstatus(SYSTEM.CLUSTER.TRANSMIT.QUEUE) type(queue) all
1 : dis qstatus(SYSTEM.CLUSTER.TRANSMIT.QUEUE) type(queue) all
AMQ8450: Display queue status details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.QUEUE) TYPE(QUEUE)
CURDEPTH(0) IPPROCS(0)
LGETDATE(2011-07-1 LGETTIME(17.42.54)
LPUTDATE(2011-07-1 LPUTTIME(17.42.54)
MEDIALOG(S0008333.LOG) MONQ(MEDIUM)
MSGAGE(40167090) OPPROCS(0)
QTIME(25793979, 300353550) UNCOM(YES)
dis qstatus(SYSTEM.CLUSTER.TRANSMIT.QUEUE) type(handle) all
2 : dis qstatus(SYSTEM.CLUSTER.TRANSMIT.QUEUE) type(handle) all
AMQ8565: Queue Status not found.
I have tried the following:
- restart qmgr
- kill repos process
- stop all cluster channels and channel init
- dspmqtrn shows nothing
A PMR has now been raised, but i was interested whether other people have seen this.
MQ version 7.0.1.2
Cheers ... Ruud |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Jul 19, 2011 1:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
You do realize that for 7.0.1 the "GA" version is 7.0.1.3 ?
So you might want to think about upgrading to the latest version.
Surprised that bouncing the qmgr did not clear the uncommitted message.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jul 19, 2011 4:35 pm Post subject: Re: SCTQ UNCOM(YES) but no input/output processes |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
Uncommitted operations when there are no OPPROCS and no IPPROCS... looks like an indoubt channel or an incomplete XA transaction.
Any helpful output from dspmqtrn, or DIS CHS(*) ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 19, 2011 7:49 pm Post subject: Re: SCTQ UNCOM(YES) but no input/output processes |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
| dutchman wrote: |
I have tried the following:
- restart qmgr
- kill repos process
- stop all cluster channels and channel init
- dspmqtrn shows nothing
A PMR has now been raised, but i was interested whether other people have seen this.
MQ version 7.0.1.2
Cheers ... Ruud |
@ mvic, you must have missed the above...
I figure that it is indeed an uncommitted XA transaction but if the prepare commit got never executed and the client connection was cut, all you have is queue depth with no possibility to access it.
Usually this gets cleared up, either when the application disconnects or in case of a JMS web app (pooling) when the web app disconnects or the qmgr gets bounced...
You may also want to verify and ensure that you have enough logspace
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| dutchman |
| Posted: Tue Jul 19, 2011 8:38 pm Post subject: |
|
|
Acolyte
Joined: 15 May 2001
Posts: 71
Location: Netherlands
|
Guys, good morning. Everything checked out, no logspace issues, no FDCs, AMQERR* long overwritten.
Nothing useful from IBM as yet. They are asking for all the usual stuff like dis channels when the display I sent them quite clearly stated that there was no current activity!
C'est la vie. Will keep you posted... Ruud |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jul 20, 2011 12:38 am Post subject: Re: SCTQ UNCOM(YES) but no input/output processes |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| fjb_saper wrote: |
| @ mvic, you must have missed the above... |
correct  |
|
| Back to top |
|
|
| dutchman |
| Posted: Wed Jul 20, 2011 3:08 am Post subject: |
|
|
Acolyte
Joined: 15 May 2001
Posts: 71
Location: Netherlands
|
Update: When I initially looked at the MSGAGE parm I thought it was expressed in milliseconds and as such it worked out to be around 4.5 days ... which
matched the date of the alert.
But I was wrong - it turns out this unit-of-work is 465 days old!
I like the comment I got from IBM: In a very small number of cases, it has been observed that uncommitted messages have remained on a transmission queue after a channel has re-started and the in-doubt status of the channel has been resolved
If I go back 465 days we may well have been running 7.0.0.1 at that time.
I reckon the only way round this is to delete the SCTQ and recreate it.
However, the 'purists' amongst us might argue that runmqsc should not allow that - after all, there is an infnished transaction out there.
We could, of course, go to the file system and delete it there, which would cause runmqsc to report it as corrupt... which then necessitates a
delete/recreate after all.
Keep smiling! ... Ruud |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jul 20, 2011 3:26 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| dutchman wrote: |
| In a very small number of cases, it has been observed that uncommitted messages have remained on a transmission queue after a channel has re-started and the in-doubt status of the channel has been resolved |
Is that very small number the number 1: ie. you?
It is hard to believe this isn't a bug. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 20, 2011 8:03 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
I hate to cheese out and say "Upgrade and see if it goes away" if you don't have a specific APAR resolved in a specific Fix Pack, but you are at 7.1.0.2. You can't get that version even if you wanted to. 7.0.1.3 is the oldest version of MQ 7 available for new installs on Passport Advantage. Makes me wonder what was lurking in the previous 7.0.1 versions that was so bad.
Rather than deleting/recreating, what about using rsvmqtrn to force the QM to resolve this transaction. I have to assume if its > 400 days old and no one is complaining about a missing message you can get rid of it.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa15890_.htm?resultof=%22%64%73%70%6d%71%74%72%6e%22%20
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 20, 2011 11:38 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
| PeterPotkay wrote: |
I hate to cheese out and say "Upgrade and see if it goes away" if you don't have a specific APAR resolved in a specific Fix Pack, but you are at 7.1.0.2. You can't get that version even if you wanted to. 7.0.1.3 is the oldest version of MQ 7 available for new installs on Passport Advantage. Makes me wonder what was lurking in the previous 7.0.1 versions that was so bad.
Rather than deleting/recreating, what about using rsvmqtrn to force the QM to resolve this transaction. I have to assume if its > 400 days old and no one is complaining about a missing message you can get rid of it.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa15890_.htm?resultof=%22%64%73%70%6d%71%74%72%6e%22%20 |
Peter, looks like you too missed the obvious:
| dutchman wrote: |
I have tried the following:
- restart qmgr
- kill repos process
- stop all cluster channels and channel init
- dspmqtrn shows nothing
A PMR has now been raised, but i was interested whether other people have seen this.
MQ version 7.0.1.2
Cheers ... Ruud |
Or do you know of any cases where rsvmqtrn solved some problem when dspmqtrn showed nothing?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Wed Jul 20, 2011 3:02 pm Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
Hi Ruud,
there must be some 'evidence' around of what the message actually was... did you find anything in the Q file or LOG's (assuming the message was persistent ofcourse...)
given that it's more then 400 days old, not many people will complain... and the company did not go bankrupt... but for purists sake... I would still want to try and find out what the message was...
remember sometimes you see these small articles in the paper about a letter being delivered after 10,20,30,40 sometimes even 50 years...
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 20, 2011 6:05 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
FJ, dspmqtrn shows nothing, but runmqsc's UNCOM does, so who's right? I don't know of a specific situation where rsvmqtrn cleans up things that dspmqtrn doesn't show (because why would you look to run rscmqtrn if dspmqtrn showed nothing), but would it hurt to try? I dunno, maybe it would!
Ruud, what was IBM's official recommended course of action for you?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 20, 2011 8:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
| PeterPotkay wrote: |
FJ, dspmqtrn shows nothing, but runmqsc's UNCOM does, so who's right? I don't know of a specific situation where rsvmqtrn cleans up things that dspmqtrn doesn't show (because why would you look to run rscmqtrn if dspmqtrn showed nothing), but would it hurt to try? I dunno, maybe it would!
Ruud, what was IBM's official recommended course of action for you? |
Peter, AFAIK dspmqtrn only shows something if the XA prepare commit has been executed but the commit itself has not. In this situation you can force rollback or commit using rsvmqtrn.
In my experience if the connection was cut before the XA prepare commit call was ever made, you have a "ghost" message (uncommitted, unreachable, but present in the qdepth count) and the only way to get rid of it is to bounce the qmgr.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| dutchman |
| Posted: Wed Jul 20, 2011 9:57 pm Post subject: |
|
|
Acolyte
Joined: 15 May 2001
Posts: 71
Location: Netherlands
|
Guys ... nice to see your thoughts and replies. I got a response from IBM this morning: Pls run
rsvmqtrn -m QMGR_NAME -c 0,341736
That got me thinking: how is it that dspmqtrn showed nothing and yet there is this transaction. One command I was asked to issue was this:
amqldmpa -m QMGR-NAME -c A -o 2 -f/tmp/atm
amqldmpa -m QMGR-NAME -c K -d 3 -f /tmp/kern
The atm output showed the following:
StructID "ATXN"
State PREPARED|HARDENED|LOGGED
TranNum 0.341736
Tranid 2.11.11.2eb30
hmtxTranData.RequestCount 100
FirstLSN 0.8.9043.56790
LastLSN 0.8.9043.57042
RestartLSN 0.8.9043.56790
BrowseLockCount 95
BrowseUnlockCount 95
CreateTime 2011-07-18 11:31:42.000
SLE(0)
{
hsmbQHandle 2.9.9.2e7958
ObjName %CHLBATCH.478
ObjType 512
Index: 0
Reason: aqtQHTSPAD
Priority 0
Flags adtSLEINUSE|adtSLEACTIVE|adtSLEVALID|adtSLEPERS
PrevLSN 0.8.9043.56790
LSN 0.8.9043.56790
Qid 82221248.27
}
SLE(1)
{
hsmbQHandle 2.9.9.3ad58
ObjName SYSTEM.CLUSTER.TRANSMIT.QUEUE
ObjType 65537
Index: 1
Reason: aqtQHTGETOP
Priority 0
Flags adtSLEINUSE|adtSLEACTIVE|adtSLEVALID|adtSLEPERS
LSN 0.8.9043.56790
Qid a1abb59d.0
}
TRANSACTIONSYNONYM
{
StructID "ATTS"
HashValue FEC1EAB8
Tranid 2.11.11.2eb30
}
}
{
This quite clearly shows PREPARED - so I have asked the obvious question: why did dspmqtrn not show this?
Cheers ... Ruud |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Thu Jul 21, 2011 1:25 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
did you get any pointers to retrieve or get a hold of the message data itself? resolving the uncom by committing could result in a business txn being sent, I'd like to know what the actual message was before 'releasing' it... 
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
|
|
