| Author |
Message
|
| yadav.neeraj01 |
 Posted: Mon Feb 29, 2016 1:51 pm Post subject: Queue manager not starting..ended unexpectedly |
 |
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
Hello...We are using MQ V6 on Linux env...Queue manager ended unexpectedly...i am not able to start it... Below are the Log files and FDC related to this... what is the possible root cause and resolution of this issue..Please help
mqm@TTLAPDTRV02[/var/mqm/errors]strmqm DTRQMT02
WebSphere MQ queue manager 'DTRQMT02' starting.
AMQ6109: An internal WebSphere MQ error has occurred.
mqm@TTLAPDTRV02[/var/mqm/errors]mqrc AMQ6109
1077960969 0x40406109 STOP
1077960969 0x40406109 xecSTOP
MESSAGE:
An internal WebSphere MQ error has occurred.
EXPLANATION:
An error has been detected, and the MQ error recording routine has been called.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.
Mq Log:-
----- amqxfdcx.c : 808 --------------------------------------------------------
02/29/2016 10:18:47 AM - Process(14047.1) User(mqm) Program(strmqm)
AMQ6119: An internal WebSphere MQ error has occurred ('43 - Identifier removed'
from semop.)
EXPLANATION:
MQ detected an unexpected error when calling the operating system. The MQ error
recording routine has been called.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.
----- amqxfdcx.c : 769 --------------------------------------------------------
02/29/2016 10:18:47 AM - Process(14047.1) User(mqm) Program(strmqm)
AMQ6184: An internal WebSphere MQ error has occurred on queue manager DTRQMT02.
EXPLANATION:
An error has been detected, and the WebSphere MQ error recording routine has
been called. The failing process is process 14047.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.
----- amqxfdcx.c : 808 --------------------------------------------------------
FDC Files:-
WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Monday February 29 10:18:46 MST 2016 |
| Host Name :- TTLAPDTRV02 (Linux 2.6.18-406.el5PAE) |
| PIDS :- 5724H7204 |
| LVLS :- 6.0.2.12 |
| Product Long Name :- WebSphere MQ for Linux (x86 platform) |
| Vendor :- IBM |
| Probe Id :- XC175012 |
| Application Name :- MQM |
| Component :- xllSemReq |
| SCCS Info :- lib/cs/unix/linux_2/amqxlpix.c, 1.40.1.3 |
| Line Number :- 1789 |
| Build Date :- Dec 5 2012 |
| CMVC level :- p600-212-121204 |
| Build Type :- IKAP - (Production) |
| UserID :- 00011001 (mqm) |
| Program Name :- amqzxma0 |
| Addressing mode :- 32-bit |
| Process :- 14050 |
| Thread-Process :- 14050 |
| Thread :- 1 |
| ThreadingModel :- PosixThreads |
| QueueManager :- DTRQMT02 |
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 27 |
| Last HQC :- 3.0.0-671564 |
| Last HSHMEMB :- 1.0.0-865664 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| ('43 - Identifier removed' from semop.) |
| FDCSequenceNumber :- 0 |
| Arith1 :- 43 2b |
| Comment1 :- '43 - Identifier removed' from semop. |
 |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Mon Feb 29, 2016 2:23 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
Did you search google for the Probe Id? The first hit I found suggested making sure that all mq internal processes were no longer running before attempting to start the qmgr. You can run this command: ps -ef | grep amq | grep DTRQMT02
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 2:26 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
below is the output
========================
mqm@GTWTTLAPDTRV02[/var/mqm/errors]ps -ef | grep amq | grep DTRQMT02
mqm 32523 1 0 Feb22 ? 00:00:00 /opt/mqm/bin/amqzmgr0 -m DTRQMT02
mqm 32560 1 0 Feb22 ? 00:00:00 /opt/mqm/bin/amqrmppa -m DTRQMT02
=========================== |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 2:27 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
| i searched google about this issue...somebody suggested to run amqiclen -x, i ran but no luck |
|
| Back to top |
|
|
| Andyh |
| Posted: Mon Feb 29, 2016 2:38 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Why/How are mq owned semaphores being deleted from under a running queue manager. Were other mq owned IPC resources also deleted ?
MQ uses shared memory to track what MQ processes are active. If MQ owned shared memory is deleted then the queue manager can become confused about what queue manager related processes are still running, which might lead to this sort of error (there's not enough detail in your appends to reliably identify what's going on).
I would suggest ending ALL mq related processes, then see if the queue manager will restart. As some illegal activity has already occurred (resulting in semaphores being deleted from under a running queue manager) then it's likely that you'll need to use kill -9 to end any remaining MQ processes.
There should be no need to manually remove MQ related IPC resources and this should be avoided whenever possible. |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 3:09 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
| Ended both the process of queue manager..still not able to start the queue manager...i ran that clean command when both the queue manager was stopped |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Feb 29, 2016 3:37 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
Re-boot.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 4:02 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
Thanks everyone...its been sorted out...  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Feb 29, 2016 5:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
| yadav.neeraj01 wrote: |
| Thanks everyone...its been sorted out... |
Exactly how?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 5:53 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
| By Killing the MQ Processes and reboot |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Feb 29, 2016 6:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
| yadav.neeraj01 wrote: |
| By Killing the MQ Processes and reboot |
You posted that there were no amq processes, and strmqm still failed. I'm a bit confused.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| yadav.neeraj01 |
| Posted: Mon Feb 29, 2016 6:44 pm Post subject: |
|
|
Apprentice
Joined: 23 Feb 2011
Posts: 49
|
| I killed the mqm processes but did not reboot.....that's why I was not able to start..now I rebooted the system and able to start the queue manager... |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Mar 01, 2016 12:04 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
The important lesson here is NEVER to manually delete MQ owned IPC resources.
If you are ever in a situation where this appears "necessary" you should be raising a PMR with MQ support.
The queue manager design is that it should never be necessary to manually remove IPC resources and any valid situation requiring this action would be considered APARable.
Similarly, once ALL MQ RELATED (including applications) processes are ended, a reboot should never be required. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Mar 01, 2016 1:07 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
There is of course the small matter of using ancient versions of MQ (which are not supported and therefore not APAR'able and not PMR'able).
If you kill MQ processes (last resort) then there is a preferred order to do this:
amqzmuc0 Critical process manager
amqzxma0 Execution controller
amqzfuma OAM process
amqzlaa0 LQM agents
amqzlsa0 LQM agents
amqzmuf0 Utility Manager
amqzmur0 Restartable process manager
amqzmgr0 Process controller
amqfqpub Publish Subscribe process
amqfcxba Broker worker process
amqrmppa Process pooling process
amqcrsta Non-threaded responder job process
amqcrs6b LU62 receiver channel and client connection
amqrrmfa The repository process (for clusters)
amqzdmaa Deferred message processor
amqpcsea The command server
runmqtrm Invoke a trigger monitor for a server
runmqdlq Invoke dead-letter queue handler
runmqchi The channel initiator process
runmqlsr The channel listener process
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
