ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » getting persistent messages from the queue is very slow

Post new topic  Reply to topic Goto page Previous  1, 2, 3, 4, 5, 6  Next
 getting persistent messages from the queue is very slow « View previous topic :: View next topic » 
Author Message
hughson
PostPosted: Thu Feb 17, 2022 2:26 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

an4ous wrote:
Where can I find commit/rollback statistic for a specific queue?

There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.

You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
an4ous
PostPosted: Thu Feb 17, 2022 3:02 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

hughson wrote:
an4ous wrote:
Where can I find commit/rollback statistic for a specific queue?

There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.

You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.

Cheers,
Morag


CommitCount/CommitFailCount from MQIAccounting that's what I need?
or BackCount?
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Feb 17, 2022 3:45 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

an4ous wrote:
hughson wrote:
an4ous wrote:
Where can I find commit/rollback statistic for a specific queue?

There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.

You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.

Cheers,
Morag


CommitCount/CommitFailCount from MQIAccounting that's what I need?
or BackCount?


I suspect all of those would be interesting, but you were specifically interested in why you have so many rollbacks.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
markt
PostPosted: Thu Feb 17, 2022 4:30 am    Post subject: Reply with quote

Knight

Joined: 14 May 2002
Posts: 502

Remember that svrconn channels will often issue a speculative MQBACK as they end to ensure that commits don't happen even if the app has abended. So counting calls to that verb are not necessarily useful. (An MQBACK when there's no actual work to really be rolled back is not expensive.)
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 17, 2022 5:04 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

markt wrote:
Remember that svrconn channels will often issue a speculative MQBACK as they end to ensure that commits don't happen even if the app has abended. So counting calls to that verb are not necessarily useful. (An MQBACK when there's no actual work to really be rolled back is not expensive.)

gbaddeley wrote:
bruce2359 wrote:
an4ous wrote:
around 1400 commits and 500 rollbacks per second
Seems to me to be a high percentage of rollbacks.
I'd suggest an application trace to identify what is driving so many rollbacks.

There should not be ANY rollbacks. This indicates an issue that needs to be found and fixed. Rollbacks will be impacting overall performance.

I agree. Both of these statements can be true. There can be badly written applications that issue rollbacks; and svrcon channels can issue rollbacks.

This long-running thread has been a wild ride presuming (as I read it) that some MQ defect is responsible for this performance issue. This, too, can also be true.

We've not yet heard from the app developer to be able to exclude the app as the problem source. Does the app issue rollbakcs? I so, why?

I’ve enabled channel stats for SDR-RCVR channels, but not SVRCONNs.

I recall working with a client whose app put persistent "transaction state messages" to a "transaction state queue" in a multi-legged transaction. From time to time one or another of the legs would fail (bad data, whatever), and the transaction would be rolled back. From time to time, but nowhere near the percentage of rollbacks in the OPs app.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
an4ous
PostPosted: Thu Feb 17, 2022 6:25 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

How can I to distinguish real rollbacks and "speculative MQBACK of svrconn channels"?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 17, 2022 8:49 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

bruce2359 wrote:
We've not yet heard from the app developer to be able to exclude the app as the problem source. Does the app issue rollbakcs? I so, why?

_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 17, 2022 9:19 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

What is splunk's involvement? I found a few hits searching google for splunk+performance.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
an4ous
PostPosted: Thu Feb 17, 2022 10:25 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

splunk is not a bottleneck. Splunk is ready to accept many more requests
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 17, 2022 10:37 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

an4ous wrote:
splunk is not a bottleneck. Splunk is ready to accept many more requests

Are you a Splunk guru? What is your evidence that Splunk is not a bottleneck?

We have yet to hear from the developers of the app.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
an4ous
PostPosted: Thu Feb 17, 2022 10:56 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

I am not splunk guru, but in our company to one splunk server write many other buesness systems from other projects (not via ibm wmq), they don't have the same problems as we do with perfomance.

Also our splunk administrators says that splunk server has resources for more activity.

Our developers remain silent
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 17, 2022 12:06 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

an4ous wrote:
Our developers remain silent

You appear target-locked on finding MQ at fault. It is quite common for developers to lay blame on MQ until MQ folks prove that MQ is not the problem.

Problem source identification demands full participation from all involved: developers, hardware, network, o/s, database, MQ, splunk, … clearly not happening here.

I’d suggest that you push back - point out to management that you (and the users of the poorly performing app) are not being fully supported in this effort.

Welcome to the MQ world.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Feb 18, 2022 2:22 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

an4ous wrote:
How can I to distinguish real rollbacks and "speculative MQBACK of svrconn channels"?

I'm not sure you can, but we have yet to hear what your developer has to say about when they might themselves issue an MQBACK. That information will help. For example, they might have chosen to write to an application log whenever they issue an MQBACK, as that would be an unusual path for the application to take.

Another interesting factor that will help is whether the disconnects are explicit, or implicit. An implicit disconnect will cause a rollback. That statistic is also part of the MQI Accounting record you should capture.

Be very interesting to see these records. Keep us posted.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
an4ous
PostPosted: Fri Feb 18, 2022 2:40 am    Post subject: Reply with quote

Apprentice

Joined: 14 Jan 2022
Posts: 38

In MQI accounting, I see a lot of similar records
Code:

...
MonitoringType: MQIAccounting
QueueManager: 'MGR_PROD'
IntervalStartDate: '2022-02-17'
IntervalStartTime: '14.15.54'
IntervalEndDate: '2022-02-17'
IntervalEndTime: '14.15.54'
CommandLevel: 910
ConnectionId: x'414d51435247534d47525f50524f442090970d6272cdf522'
SeqNumber: 0
ApplicationName: 'CamelApp'
ApplicationPid: 31473
ApplicationTid: 2596926
UserId: 'USER_NAME'
ConnDate: '2022-02-17'
ConnTime: '14.15.54'
ConnName: 'CLIENT_IP'
ChannelName: 'SPLUNK.SVRCONN'
RemoteProduct: 'MQJM'
RemoteVersion: '09020300'
DiscDate: '2022-02-17'
DiscTime: '14.15.54'
DiscType: Normal
OpenCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
OpenFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
PutCount: [0, 0]
PutFailCount: 0
Put1Count: [0, 0]
Put1FailCount: 0
PutBytes: [0, 0]
GetCount: [0, 0]
GetFailCount: 0
GetBytes: [0, 0]
BrowseCount: [0, 0]
BrowseFailCount: 0
BrowseBytes: [0, 0]
CommitCount: 2
CommitFailCount: 0
BackCount: 1
InqCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
InqFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DurableSubscribeCount: [0, 0, 0]
NonDurableSubscribeCount: [0, 0, 0]
SubscribeFailCount: 0
DurableUnsubscribeCount: [0, 0]
NonDurableUnsubscribeCount: [0, 0]
UnsubscribeFailCount: 0
SubRqCount: 0
SubRqFailCount: 0
CbCount: [0, 0, 0, 0]
CbFailCount: 0
CtlCount: [0, 0, 0, 0]
CtlFailCount: 0
StatCount: 0
StatFailCount: 0
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
...

what should I pay attention to?
For every 2 CommitCount there is 1 BackCount it is normally?
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Feb 18, 2022 3:11 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

This application doesn't appear to have done any work. It disconnected in the same second that it connected and has done no puts or gets.

Are all your application connections like this? Do you have any that actually did some "work"?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3, 4, 5, 6  Next Page 5 of 6

MQSeries.net Forum Index » General Discussion » getting persistent messages from the queue is very slow
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.