ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » getting persistent messages from the queue is very slow

Post new topic  Reply to topic Goto page Previous  1, 2, 3, 4, 5, 6
 getting persistent messages from the queue is very slow « View previous topic :: View next topic » 
Author Message
an4ous
PostPosted: Fri Feb 18, 2022 3:41 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

only some internal ibm mq services have get and put values > 0
for example

Code:

...
MonitoringType: MQIAccounting
QueueManager: 'MGR_PROD'
IntervalStartDate: '2022-02-17'
IntervalStartTime: '14.16.01'
IntervalEndDate: '2022-02-17'
IntervalEndTime: '14.16.01'
CommandLevel: 910
ConnectionId: x'414d51435247534d47525f50524f442090970d62a102f822'
SeqNumber: 0
ApplicationName: 'amqsmon'
ApplicationPid: 11468
ApplicationTid: 1
UserId: 'mqm'
ConnDate: '2022-02-17'
ConnTime: '14.16.01'
DiscDate: '2022-02-17'
DiscTime: '14.16.01'
DiscType: Normal
OpenCount: [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
OpenFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseCount: [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
PutCount: [0, 0]
PutFailCount: 0
Put1Count: [0, 0]
Put1FailCount: 0
PutBytes: [0, 0]
GetCount: [1, 0]
GetFailCount: 1
GetBytes: [800, 0]
BrowseCount: [1, 0]
BrowseFailCount: 0
BrowseBytes: [800, 0]
CommitCount: 1
CommitFailCount: 0
BackCount: 0
InqCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
InqFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DurableSubscribeCount: [0, 0, 0]
NonDurableSubscribeCount: [0, 0, 0]
SubscribeFailCount: 0
DurableUnsubscribeCount: [0, 0]
NonDurableUnsubscribeCount: [0, 0]
UnsubscribeFailCount: 0
SubRqCount: 0
SubRqFailCount: 0
CbCount: [0, 0, 0, 0]
CbFailCount: 0
CtlCount: [0, 0, 0, 0]
CtlFailCount: 0
StatCount: 0
StatFailCount: 0
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]

...
MonitoringType: MQIAccounting
QueueManager: 'MGR_PROD'
IntervalStartDate: '2022-02-17'
IntervalStartTime: '14.16.02'
IntervalEndDate: '2022-02-17'
IntervalEndTime: '14.16.02'
CommandLevel: 910
ConnectionId: x'414d51435247534d47525f50524f442090970d629fadf522'
SeqNumber: 0
ApplicationName: 'java'
ApplicationPid: 2180
ApplicationTid: 2662
UserId: 'mqm'
ConnDate: '2022-02-17'
ConnTime: '14.16.02'
DiscDate: '2022-02-17'
DiscTime: '14.16.02'
DiscType: Normal
OpenCount: [0, 2, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0]
OpenFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseCount: [0, 2, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0]
CloseFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
PutCount: [1, 0]
PutFailCount: 0
Put1Count: [0, 0]
Put1FailCount: 0
PutBytes: [0, 0]
GetCount: [1, 0]
GetFailCount: 0
GetBytes: [0, 0]
BrowseCount: [0, 0]
BrowseFailCount: 0
BrowseBytes: [0, 0]
CommitCount: 2
CommitFailCount: 0
BackCount: 1
InqCount: [0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0]
InqFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DurableSubscribeCount: [0, 0, 0]
NonDurableSubscribeCount: [0, 0, 0]
SubscribeFailCount: 0
DurableUnsubscribeCount: [0, 0]
NonDurableUnsubscribeCount: [0, 0]
UnsubscribeFailCount: 0
SubRqCount: 0
SubRqFailCount: 0
CbCount: [0, 0, 0, 0]
CbFailCount: 0
CtlCount: [0, 0, 0, 0]
CtlFailCount: 0
StatCount: 0
StatFailCount: 0
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
...
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Feb 18, 2022 3:49 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

I suspect that you DO have applications actually doing work on this queue manager.

Have you waited for a full Accounting Interval? Long running application connections write their Accounting records every interval. Short running application connections write their Accounting records at disconnect time.

Perhaps you have long running applications doing work and you haven't yet captured their accounting records?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Andyh
PostPosted: Fri Feb 18, 2022 4:10 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

an4ous wrote:
How can I to distinguish real rollbacks and "speculative MQBACK of svrconn channels"?


You'd have to either look at MQ traces, or at the MQ recovery logs.
Neither are very attractive options for someone without significant MQ expetise.

It's also highly unlikely that this is your issue, you stated earlier that the performance was OK for a while following a QMgr restart, and then tailed off after a while. I tried to explain the most likely cause of this in an earlier append. If you were to collect MQ stats from when the QMgr restarts, to when the issue starts to occur you could look to see if the application behaviour changes (i.e the mix and rate of MQI calls). You could also monitor the current depth of the problematic queue over that time to try and see if the problem occurs because the queue gets deep, or vice-versa.

IBM support can give you much more insight into how the queue is being used by looking at internal metrics associated with the queue. Have you actually raised a case with IBM ?
Back to top
View user's profile Send private message
an4ous
PostPosted: Fri Feb 18, 2022 4:29 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

Main answer of IBM with them recomendations in my first post


Last answer of IBM
"From this, they seem to be saying that when they increased the number of getters, it didn't help, but increased CPU usage.

"chnage of getters count don not increase speed of reading from queue (but increase load avarfe and cpu utilization on mq)"
Well yes we would expect increasing the number of getters working in parallel would increase CPU usage, and they ought to have expected that, too.
However, I would also expect that, as a payback for the increased CPU usage, they would have processed more gets per second in this configuration.
And that's what they need to do.

I think we may have run out of road for the support process here.. we made plenty of suggestions for how they might be able to move forwards so that getters could keep up with putters, and if they want more, I think they ought to explain this to an IBM Services consultant, and get a wider-ranging discussion, though of course this attracts a fee."

"As noted by the MQ Platform Team, they have provide a few recommendations to help you more forward. Anything more is beyond the defect support process. If you require further assistance, you will need to engage the IBM Services Team. Note: This is a fee based service which you can arrange through your IBM Account Rep."

Thay say that with IBM MQ not problems so our case out of standart support.
Back to top
View user's profile Send private message
Andyh
PostPosted: Fri Feb 18, 2022 6:58 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

It's true that the L2/L3 support teams role is not to help you with tuning and monitoring and that you could pay for a services engagement to investigate further. Pointing out to the IBM support team that performance degrades over time might yield a bit more help from that source (degrading performance might be considered a potential defect, but as stated earlier it's more likely to be because the queue is becoming deep and the gets are having to do more work to find the requested messages).

I'm not aware of any easy way to share detailed performance data over this forum. I have sufficient knowledge to suggest what to collect and to interpret at this data, but I think you need to be very careful about what you share with unknown individuals on the internet! It's a bit beyond the scope of a quick forum chat to try and explain (in a generic manner )what to look for in this data. Although the utilities that generate the data are shipped with the product they're not really intended for direct customer use and are not documented. Because of this I doubt IBM would be keen to present on this topic at any of the MQ related conferences, but it might be worth suggesting.
Back to top
View user's profile Send private message
an4ous
PostPosted: Fri Feb 18, 2022 11:09 pm    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

It doesn't matter if queue deep or not - getting speed still falling
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sat Feb 19, 2022 6:16 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

an4ous wrote:
It doesn't matter if queue deep or not - getting speed still falling

Do you suspect that depth doesn’t matter? Or have you proven that it doesn’t matter?

Two local queue events you should consider enabling as part of problem determination:
DEPTH EVENTS and SERVICE INTERVAL EVENTS https://www.ibm.com/docs/en/ibm-mq/7.5?topic=objects-attributes-queues
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
an4ous
PostPosted: Sun Feb 20, 2022 10:44 pm    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

I am sure that queue depth doesn't matter. In friday queue depth was 11 millions messages. In during of weekends when business activity is low queue decreased to zero (to queue have put around 30 messages per second and got around 130 messages per second)

Now when beginning new work day and when in quue 0 mesasages to queue putting around 200 messages per second and getting less then 100 messages per second so quue grows again.
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Feb 21, 2022 2:25 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

More recent MQ versions (several years now) give a bit of priority to getters over putters when waiting for queue locks (on the basis that an empty queue is a good queue). Even then it's possible for a large number of putters to overwhelm a smaller number of getters. You're reporting around 200 puts and 100 gets per second in the constrained period (i.e. 300 ops/sec) but only 130 ops/sec during the low put load period. An IO limit would typically result in a higher total when gets were outstripping puts (more IO involved in a put than a get). Similarly the CPU costs of most puts and gets are typically similar (unless searching for messages on a deep queue). Your observation that the total ops/sec reduces significantly when the put rate is low might therefore further suggest a performance issue in the getting application.

There are a number of earlier questions (and some confusuon on my part) about how this app preserves message order with multiple getters. Please would you try to explain how the getting applications works. What options are specified on the MQOPEN and the MQGET and what serialization exists in the app itself.
Back to top
View user's profile Send private message
an4ous
PostPosted: Mon Feb 21, 2022 3:14 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

Andyh wrote:


There are a number of earlier questions (and some confusuon on my part) about how this app preserves message order with multiple getters. Please would you try to explain how the getting applications works. What options are specified on the MQOPEN and the MQGET and what serialization exists in the app itself.


Thanks , I try to get this information from developers (it is difficult)
Back to top
View user's profile Send private message
an4ous
PostPosted: Mon Feb 21, 2022 4:01 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

Our developers say that they don't know "What options are specified on the MQOPEN and the MQGET and what serialization exists in the app itself." because they use apache camel framework and jms, rely on them and they don't see what's "under the hood"
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Feb 21, 2022 6:01 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

You could observe the app's MQI behaviour by taking an activity trace, and that's a fairly simple thing to review yourself.

In order to guess at what's happening from a serialization viewpoint a full MQ trace might be more helpful, but that's a bit harder for you to diagnose yourself, and not the sort of thing you'd share on a public forum.

Without a good set of doc and some expert help this is likely to be a long running saga, there have been lots of well intentioned suggestions in this thread, but there's too much guesswork going on for my taste (inevitable given the limited doc available) and it's like hoping to stumble across root cause rather than a more structured approach to narrow down the possible causes.

MQ is capable of tens of thousands of persistent messages per second, even on quite modest hardware, and 130 msg/sec suggests an issue that should be quite visible to someone with some knowledge and access to the diagnostic information MQ is able to make available.
Back to top
View user's profile Send private message
an4ous
PostPosted: Mon Feb 21, 2022 7:27 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

In MQI trace for SPLUNK QUEUE I see such records

Tid Date Time Operation CompCode MQRC HObj (ObjName)
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:20 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:20 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:21 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )
5477210 2022-02-21 18:03:21 MQXF_CMIT MQCC_OK 0000 -
5477210 2022-02-21 18:03:21 MQXF_GET MQCC_OK 0000 2 (SPLUNK.Q )


but sometimes such (not often)
...
5724637 2022-02-21 17:53:12 MQXF_GET MQCC_WARNING 2080 2 (SPLUNK.Q )
5724637 2022-02-21 17:53:12 MQXF_GET MQCC_FAILED 2033 2 (SPLUNK.Q )
...

MQ reason codes 2080 and 2033 mean MQRC_TRUNCATED_MSG_FAILED and MQRC_NO_MSG_AVAILABLE
For SPLUNK QUEUE and SPLUNK CHANNEL MAXMSGL is 104857600 and there are messages in queue so I do not know why 2033 and 2080 warining/erros apeare but in general, there are few of them

For SPLUNK SVRCONN CHANNEL I also see many records as

Tid Date Time Operation CompCode MQRC HObj (ObjName)
4161099 2022-02-21 17:53:08 MQXF_CONNX MQCC_OK 0000 -
4161099 2022-02-21 17:53:08 MQXF_OPEN MQCC_OK 0000 2 ( )
4161099 2022-02-21 17:53:08 MQXF_INQ MQCC_OK 0000 2 ( )
4161099 2022-02-21 17:53:08 MQXF_CLOSE MQCC_OK 0000 2 ( )
4161099 2022-02-21 17:53:08 MQXF_DISC MQCC_OK 0000 -


or as

4366695 2022-02-21 17:53:08 MQXF_CONNX MQCC_OK 0000 -
4366695 2022-02-21 17:53:08 MQXF_CMIT MQCC_OK 0000 -
4366695 2022-02-21 17:53:08 MQXF_BACK MQCC_OK 0000 -
4366695 2022-02-21 17:53:08 MQXF_DISC MQCC_OK 0000 -

I don't know if it's bad or good, but it doesn't look bad

Why in trace there are no PUT events?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Feb 21, 2022 7:48 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

an4ous wrote:
In MQI trace for SPLUNK QUEUE I see such records ...

but sometimes such (not often)
...
5724637 2022-02-21 17:53:12 MQXF_GET MQCC_WARNING 2080 2 (SPLUNK.Q )
5724637 2022-02-21 17:53:12 MQXF_GET MQCC_FAILED 2033 2 (SPLUNK.Q )
...

MQ reason codes 2080 and 2033 mean MQRC_TRUNCATED_MSG_FAILED and MQRC_NO_MSG_AVAILABLE
For SPLUNK QUEUE and SPLUNK CHANNEL MAXMSGL is 104857600 and there are messages in queue so I do not know why 2033 and 2080 warining/erros apeare but in general, there are few of them


https://www.ibm.com/support/pages/mqrc-2080-resize-buffer-and-continue-get-mqrctruncatedmsgfailed addresses the 2080, and offers application changes to accommodate - including open-input-exclusive for getters.

11 million messages is a deep queue. Perhaps you should consider creating a qmgr cluster to spread the workload, with 3 or 4 qmgrs, each with an instances of the same named queue.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Apr 03, 2022 7:11 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Any resolution on this issue?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3, 4, 5, 6 Page 6 of 6

MQSeries.net Forum Index » General Discussion » getting persistent messages from the queue is very slow
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.