ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Performance Monitoring » amqrmppa and amqzlaa0 processes are consuming high CPU

Post new topic  Reply to topic
 amqrmppa and amqzlaa0 processes are consuming high CPU « View previous topic :: View next topic » 
Author Message
mqlover
PostPosted: Wed Apr 30, 2014 2:03 am    Post subject: amqrmppa and amqzlaa0 processes are consuming high CPU Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Hi,

I have seen this topic being discussed quite few sometime earlier but still I havent got any solution to resolve the above problem.
There are 2 qmgrs which are full repos and are in a cluster. These qmgrs process i.e amqrmppa and amqzlaa0 are consuming lot of CPU.
I have restarted the qmgr quite a few times, killed these processes and also listener but still nothing seems to be working.

I checked there arent many SVRCONN connections to the qmgrs.
No messages piling up in any of the queues

Kindly anybody can provide me solution for this?

Thanks in advance
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Apr 30, 2014 6:33 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY

How many qmgrs in the cluster? Any big changes happening in the cluster that would have lots of communications to the FRs?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
PeterPotkay
PostPosted: Wed Apr 30, 2014 8:11 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Define "lots" of CPU.

Whats in the MQ error logs?
What do the FDC files say?

Please don't go around killing processes. Often it can make more problems than it solves.

A QM with active QM to QM channels that is processing lots messages as fast as it can is going to drive the CPU on a server.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Wed Apr 30, 2014 8:22 am    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

As I'm sure you know the amqrmppa processes are the processes which move messages - ie. the channels. The amqzlaa0 processes are the agent processes. If I saw 'high' CPU on these processes my first thought would be that I was just moving a lot of messages down my channels.

Have you ruled this out ? You seem to be suggesting that the channels are misbehaving in some way, can you say what leads you to this conclusion.

If you issue use Explorer/MO71 to display the sequence number of all running channels and then refresh the display every few seconds, do you see the numbers go up ? If so, by how many. In other words, roughly speaking, how many messages per second are you transmitting/receiving.

If the number is fairly large then you can use RESET QSTATS or equivalent to work out which queues are being used heavily. From there you can use DIS CONN(*) etc to see which applications are involved.

Of course it is possible that you have a looping message. Something that I have done in performance testing is define two remote queues that 'point' to each other. You can then inject a few messages and they will bounce backwards and forwards between the two Queue Managers as fast as possible. Handy for testing but can be confusing if done by accident.

Like Peter says, don't kill MQ processes. It should be a last resort. If you suspect a channel problem then it should be possible to find out which are having problems and, stop them manually, if necessary. Looking at the transmission queue would also be a good way of seeing what these messages were and where they were coming from,

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
mqlover
PostPosted: Sun May 04, 2014 9:49 pm    Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks for the reply and sorry for the delay.
Well, lots of CPU i meant is 2 processes of each qmgr i.e 4 process taking 25% of CPU each.
I could see the nujmber of messages on channels increasing upto 30k-40k.
Even the message sequence number increasing.
There are no FDC generated. Error logs are not explaining anything which I can understand.
I tried using RESET QMGR TYPE(STATISTICS) but doesnt seem to working as its the same.

What else approach can I take?

Thanks much for detailed explainations all this while.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sun May 04, 2014 10:32 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY

In your flow check the delivery count. Route anything above the threshold to the failure flow. You do need to take care of poison messages...
Run a debug level trace. This should show you if your message is in an endless process loop. Sometimes quite easy to program... just missing 1 line of code right?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
PaulClarke
PostPosted: Sun May 04, 2014 11:04 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Well, if you have two channels constantly moving messages and nothing else going on in the system then it does sound to me as though you have two channels looping.

What I would do is.....

  • Stop the channels in question.
  • Issue a resolve channel action(backout)
  • Then modify the transmission queues for GET(ENABLE)
  • Then browse the transmission queues and see what the messages are. Look at the XQH header on the front of the message and see where they are destined.
  • Then look at the definition of that queue and ensure that it is not a remote queue pointing right back where it came from. Naturally I would use MO71 for this but I suspect you could use Explorer. Other loops can be caused by default transmission queues pointing to the 'other' Queue Manager.
  • If you do have looping definitions then fix the queue definitions
  • Lastly, you can delete the messages off the transmission queue or move them to a different queue or unload them to a file etc, again I would use MO71 for this.


Hope that helps,

Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
mqlover
PostPosted: Mon May 05, 2014 12:04 am    Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks for your suggestions. I shall try the mentioned steps and update the result.
Back to top
View user's profile Send private message
mqlover
PostPosted: Wed May 21, 2014 6:05 pm    Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks for all your suggestions. Finally I could solve it. There was one of the channels which was taking more of resources. Could zero in only after observing the number of msgs coming and passing in through which channel.

Thanks again for your help.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu May 22, 2014 12:56 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.

How did you solve this? What exactly did you do?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ Performance Monitoring » amqrmppa and amqzlaa0 processes are consuming high CPU
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.