ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » CLUSRCVR channel in paused state

Post new topic  Reply to topic
 CLUSRCVR channel in paused state « View previous topic :: View next topic » 
Author Message
yanaK
PostPosted: Thu Jun 11, 2020 2:54 am    Post subject: CLUSRCVR channel in paused state Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

Our cluster setup for a request response flow is like this:

Client ->
PS.SVR (Channel) -> SYSTEM.CLUSTER.TRANSMIT.QUEUE (QLOCAL) -> TO.PS.B (CLUSSDR) ->

TO.PS.B (CLUSRCVR) -> PS.REQ (QLOCAL) ->

[Application] ->

SYSTEM.DEF.SVRCONN (CHANNEL) -> SYSTEM.CLUSTER.TRANSMIT.QUEUE (QLOCAL) -> TO.PS.F (CLUSSDR) ->

TO.PS.F (CLUSRCVR) -> PS.RSP (QLOCAL) -> PS.SVR (Channel) ->
Client

Now suddenly I see TO.PS.B (CLUSRCVR) is in paused state. Everything else is in running state.

CLUSRCVER error shows this:

Code:
----- amqrrmfa.c : 30286 ------------------------------------------------------
06/11/2020 03:12:46 AM - Process(11713.1) User(mqm) Program(amqrrmfa)
                    Host(t4231) Installation(Installation1)
                    VRMF(7.1.0.1) QMgr(PS.B)

AMQ9435: Unable to put repository manager message.

EXPLANATION:
The repository manager tried to send a message to the
SYSTEM.CLUSTER.COMMAND.QUEUE on another queue manager whose identifier is
MB_2012-07-23_11.24.30, but the MQPUT call was unsuccessful. MQCC=2,
MQRC=2085. Processing continues, but the repository information may be out of
date.
ACTION:
Refer to the Application Programming Reference manual for information about
MQCC 2 and MQRC 2085. Check the channel and cluster definitions on the local
and target queue managers, and ensure that the channels between them are
running. When the problem is corrected, the repository information will
normally be updated automatically. The REFRESH CLUSTER command can be used to
ensure that the repository information is up to date.

----- amqrrmfa.c : 30286 ------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
                    Host(t4231) Installation(Installation1)
                    VRMF(7.1.0.1) QMgr(PS.B)

AMQ7234: 10000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.

EXPLANATION:
10000 messages from queue SYSTEM.DEAD.LETTER.QUEUE have been loaded on queue
manager PS.B.
ACTION:
None.
-------------------------------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
                    Host(t4231) Installation(Installation1)
                    VRMF(7.1.0.1) QMgr(PS.B)

AMQ7234: 20000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
ACTION:
None.
----- amqccita.c : 3688 -------------------------------------------------------
06/11/2020 03:15:35 AM - Process(11758.57) User(mqm) Program(amqrmppa)
                    Host(t4231) Installation(Installation1)
                    VRMF(7.1.0.1) QMgr(PS.B)

AMQ9544: Messages not put to destination queue.

EXPLANATION:
During the processing of channel 'TO.PS.B' one or more messages could not
be put to the destination queue and attempts were made to put them to a
dead-letter queue.  The location of the queue is 1, where 1 is the local
dead-letter queue and 2 is the remote dead-letter queue.
ACTION:
Examine the contents of the dead-letter queue.  Each message is contained in a
structure that describes why the message was put to the queue, and to where it
was originally addressed. Also look at previous error messages to see if the
attempt to put messages to a dead-letter queue failed. The program identifier
(PID) of the processing program was '11758'.


On the 1st error one question is that queue manager is not in use - how to remove it? (I tried to strmqm but it won't even start)

On the CLUSSDR error I see this:
Code:
----- amqrccca.c : 956 --------------------------------------------------------
06/11/2020 03:34:08 AM - Process(9143.13573) User(mqm) Program(amqrmppa)
                    Host(t4232) Installation(Installation1)
                    VRMF(7.1.0.1) QMgr(PS.F)

AMQ9544: Messages not put to destination queue.

EXPLANATION:
During the processing of channel 'TO.PS.B' one or more messages could not
be put to the destination queue and attempts were made to put them to a
dead-letter queue.  The location of the queue is 2, where 1 is the local
dead-letter queue and 2 is the remote dead-letter queue.
ACTION:
Examine the contents of the dead-letter queue.  Each message is contained in a
structure that describes why the message was put to the queue, and to where it
was originally addressed. Also look at previous error messages to see if the
attempt to put messages to a dead-letter queue failed. The program identifier
(PID) of the processing program was '9143'.



The Application is throwing "No message available (timeout) for queue PS.RSP".

I tried restarting the PS.B channel - no luck - then the queue manager - still no luck.

I saw the SYSTEM.CLUSTER.TRANSMIT.QUEUE (the 1st one) had a huge CURDEPTH - cleared it - still no luck.

Channel details:
Code:
     1 : dis chl (TO.PS.B)
AMQ8414: Display Channel details.
   CHANNEL(TO.PS.B)                        CHLTYPE(CLUSRCVR)
   ALTDATE(2019-12-19)                     ALTTIME(15.01.04)
   BATCHHB(0)                              BATCHINT(0)
   BATCHLIM(5000)                          BATCHSZ(40)
   CLUSNL( )                               CLUSTER(REPPST)
   CLWLPRTY(0)                             CLWLRANK(0)
   CLWLWGHT(50)                            COMPHDR(NONE)
   COMPMSG(NONE)                           CONNAME(t4232(1416))
   CONVERT(NO)                             DESCR( )
   DISCINT(600)                            HBINT(120)
   KAINT(AUTO)                             LOCLADDR( )
   LONGRTY(999999999)                      LONGTMR(1200)
   MAXMSGL(4194304)                        MCANAME( )
   MCATYPE(THREAD)                         MCAUSER( )
   MODENAME( )                             MONCHL(QMGR)
   MRDATA( )                               MREXIT( )
   MRRTY(10)                               MRTMR(1000)
   MSGDATA( )                              MSGEXIT( )
   NETPRTY(0)                              NPMSPEED(FAST)
   PROPCTL(COMPAT)                         PUTAUT(DEF)
   RCVDATA( )                              RCVEXIT( )
   RESETSEQ(NO)                            SCYDATA( )
   SCYEXIT( )                              SENDDATA( )
   SENDEXIT( )                             SEQWRAP(999999999)
   SHORTRTY(10)                            SHORTTMR(60)
   SSLCAUTH(REQUIRED)                      SSLCIPH( )
   SSLPEER( )                              STATCHL(QMGR)
   TPNAME( )                               TRPTYPE(TCP)
   USEDLQ(YES)


My questions are 1. why did it enter this state ? 2. Is there a way to know which channel links to which queues ? 3. Is there a way to force it to running state ?

Thanks.[/code]
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jun 11, 2020 5:02 am    Post subject: Re: CLUSRCVR channel in paused state Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

yanaK wrote:
My questions are 1. why did it enter this state ?


The queue manager is not in use to the point you can't even start it.

yanaK wrote:
2. Is there a way to know which channel links to which queues ?


Channels don't link to queues, channel link to queue managers. T

yanaK wrote:
3. Is there a way to force it to running state ?


If there was a way to fix a bad configuration by just forcing it to work, we'd all be out of a job.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Jun 11, 2020 5:18 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Code:
MB_2012-07-23_11.24.30
This seems to be a qmid. Do you know what the qmgr name is and why are you not able to start it?
Does it have damaged queues? Do you need to start it with the -c option?
What did the qmgr log say after the attempt to start the qmgr failed? Why did the qmgr not start?
Was there a error message when you ran strmqm ?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Thu Jun 11, 2020 5:20 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The errors you posted describe an ACTION:.
Did you take the actions? What were the results?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Jun 11, 2020 5:55 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Moved to Clustering forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Jun 11, 2020 7:57 pm    Post subject: Re: CLUSRCVR channel in paused state Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

yanaK wrote:
CLUSRCVER error shows this:

[code]06/11/2020 03:12:46 AM - Process(11713.1) User(mqm) Program(amqrrmfa)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)

AMQ9435: Unable to put repository manager message.

EXPLANATION:
The repository manager tried to send a message to the
SYSTEM.CLUSTER.COMMAND.QUEUE on another queue manager whose identifier is
MB_2012-07-23_11.24.30, but the MQPUT call was unsuccessful. MQCC=2,
MQRC=2085. Processing continues, but the repository information may be out of
date.
ACTION:
Refer to the Application Programming Reference manual for information about
MQCC 2 and MQRC 2085. Check the channel and cluster definitions on the local
and target queue managers, and ensure that the channels between them are
running. When the problem is corrected, the repository information will
normally be updated automatically. The REFRESH CLUSTER command can be used to
ensure that the repository information is up to date.

----- amqrrmfa.c : 30286 ------------------------------------------------------

On the 1st error one question is that queue manager is not in use - how to remove it? (I tried to strmqm but it won't even start)


If the queue manager is decommissioned, you should remove it from the cluster so that other members of the cluster don't try to continue using it. This error messages shows that they are still trying to use it.

This link Removing a queue manager from a cluster is how you should normally remove a queue manager from a cluster before you decommission it. It also contains another link within it to remove a queue manager when it hasn't been done correctly. I suspect you need the alternative method since you cannot start the queue manager.

yanaK wrote:
Now suddenly I see TO.PS.B (CLUSRCVR) is in paused state. Everything else is in running state.


STATUS(PAUSED) shows that a receiving type channel is failing to a MQPUT a message to a target queue, and has paused to see if the MQPUT will succeed if it tries again in a moment.

Given that you have just loaded 1000s of messages from the Dead Letter Queue, I suspect, they were on the DLQ because they couldn't be MQPUT to their target queue before and perhaps that problem has not been fixed, so you have to go through these 1000s messages, with the Message Retry timer for each one, and then they will end up on the DLQ again. Do you know why the messages were on the DLQ, and was the problem shown in the DL Header fixed before they were loaded again?

yanaK wrote:
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)

AMQ7234: 10000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.

EXPLANATION:
10000 messages from queue SYSTEM.DEAD.LETTER.QUEUE have been loaded on queue
manager PS.B.
ACTION:
None.
-------------------------------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)

AMQ7234: 20000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
ACTION:
None.
----- amqccita.c : 3688 -------------------------------------------------------

Channel details:
1 : dis chl (TO.PS.B)
AMQ8414: Display Channel details.
CHANNEL(TO.PS.B) CHLTYPE(CLUSRCVR)
MRRTY(10) MRTMR(1000)




yanaK wrote:
The Application is throwing "No message available (timeout) for queue PS.RSP".

This is not a surprise given the above - the messages this application is waiting on will be queued up behind all the re-queued messages from the DLQ.

yanaK wrote:
I tried restarting the PS.B channel - no luck - then the queue manager - still no luck.

This will have only delayed the processing of the 1000s of re-queued messages for a time while the channel restarted.

yanaK wrote:
I saw the SYSTEM.CLUSTER.TRANSMIT.QUEUE (the 1st one) had a huge CURDEPTH - cleared it - still no luck.

This does not seem like a good idea. You could have deleted both application and cluster messages.

yanaK wrote:
My questions are 1. why did it enter this state ?

The channel entered STATUS(PAUSED) because it is having problems putting messages to target queues.

yanaK wrote:
2. Is there a way to know which channel links to which queues ?

Channels can deliver messages to any queues on the queue manager. Normally there is only one cluster channel between any two queue managers, so that one channel will be delivering messages to all the cluster queues.

yanaK wrote:
3. Is there a way to force it to running state ?

It will go into running state when it gets through the backlog of failing puts. If you don't want to retry puts, you can remove the MRRTY and MRTMR attributes.

I would suggest that the root cause will be that messages from the DLQ were reloaded without the issue reported in the DLQ-ed messages being fixed first. You should look into that.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » CLUSRCVR channel in paused state
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.