| Author |
Message
|
| Andyh |
 Posted: Mon Feb 14, 2022 9:58 am Post subject: |
 |
|
Master
Joined: 29 Jul 2010
Posts: 237
|
"after restart IBM MQ Queue manager (for change cpu core counets and change qm.ini) getting speed from problem queue up to ~275 messages per second, but after few days uptime getting speed go down less then 150 messages per second. - Why IBM MQ can not keep getting speed on one level? "
The most common reason for decaying MQGET performance would be if the queue was becoming deep and the getting application isn't simply asking for the first available message. The amqldmpa utility can display a variety of internal counters relating to how much searching of various internal queue state is occurring. IBM MQ support would typically advise on how to use this (internal, undocumented) utility and to interpret the output.
"xecL_W_LONG_LOCK_WAIT/AMQ6150W" A long lock wait (LLW) indicates that some internal mutex has remained locked for an unusually long time. MQ is very heavily multithreaded and there are a very large number of locks (there has been a transition to more fine grained locking where possible over the last 10+ years) in MQ. The FDC's in /var/mqm/errors will identify exactly which locks are suffering from contention, and again it would be typical for IBM MQ support to analyse these files to determine what is leading to the lock contention. It's far too generic an error to analyze without access to the FDC's. |
|
| Back to top |
|
 |
| an4ous |
| Posted: Wed Feb 16, 2022 3:24 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
"Spread message load across multiple queue where possible:
Can we balance between multiple queues in one QM (non cluster) without rewritng code of clients applications?
For examle pplications put and get meesages in/from one "virtual/alias" queu, then meesages roundrobin sents by MQ to several real local queues?
I do understand correctly that IBM MQ can't do that? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 5:09 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| an4ous wrote: |
In MQ QM logs last records
"AMQ7469I: Transactions rolled back to release log space
AMQ7486I: Transaction was preventing log space from being released.
long running transaction was detected/The log space for the queue manager is becoming full" 12/14/2021 (two month ago)
There are no new errors
Also I watch dspmqtrn - time of current transactions around 2 minutes (our developers set max transcation time 180s)
DISPLAY CONN(*) ALL WHERE(UOWLOGTI NE ' ') also not shows long transactions
So now long transactions are not reason low perfomance |
Maybe or maybe not.
A single long running transaction may cause enqueues across multiple log files. Multiple (many) concurrent long running transactions can cause enqueues that span all of your primary log files, and then your secondaries. This would exhaust all log files - the two error messages above.
Presuming that you’ve sized correctly the number of primaries, then three secondaries may be insufficient to handle peak workload.
Presuming that you have lots of disk space, increase both primary and secondary values. Alternatively, change from circular to linear.
This long running post has identified a variety of bottlenecks that each contribute to poor performance.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 5:25 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
I monitor logs files usage/utilization, now less than 20% at peak.
Last long transactions was in december and no matter how much I set the number of log files there were always few due to clients error (endless transactions). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 5:34 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| an4ous wrote: |
I monitor logs files usage/utilization, now less than 20% at peak.
Last long transactions was in december and no matter how much I set the number of log files there were always few due to clients error (endless transactions). |
What were the original log specs? Who calculated that size?
How did you arrive at your current specs? Did you actually calculate the number and size of your logs? Or, are you just increasing values hoping that somehow the symptom disappears?
How to calculate logs is well documented here: https://www.ibm.com/docs/en/ibm-mq/9.2?topic=lost-calculating-size-log
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 5:48 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
Early I had 150/primory/50 secondary logs files then, 250 primory/75 secondary circular logs files 256 MB each
Recently I calculated real normal usage by formul https://www.ibm.com/docs/en/ibm-mq/9.1?topic=csl-how-large-should-i-make-my-active-log with formul
logsize >= 2 * (N+1) * B
In my case
2 * 360s (real transaction time less than 180s) * 12MB (real logical writes less than 6 mb/s) /256MB =~ max 33 primory logs 256 MB each (it's stock, actually 10 primore logs should be enough) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 6:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
Earlier in this thread you posted stats that showed a large number of rollbacks. Were the log configurations the 33 and 3 then?
Please re- capture stats now, and post results here.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Last edited by bruce2359 on Thu Feb 17, 2022 10:35 am; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 6:11 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
Please refresh my memory... are your apps still MQOPENing the queue INPUT_EXCLUSIVE? What do they do when they don't successfully MQOPEN the queue (because some other app instance has it open-exclusive)?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 6:37 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
Queue for sending messages to splunk about I wrote in my first post does not work in EXCLUSIVE (NOSHARE) mode.
Many clients can put/get messages in/from this queue.
| Code: |
bash-4.2$ echo "DISPLAY QLOCAL (*) IPPROCS OPPROCS where (OPPROCS GT 50)" | runmqsc MGR_PROD
5724-H72 (C) Copyright IBM Corp. 1994, 2020.
Starting MQSC for queue manager MGR_PROD.
1 : DISPLAY QLOCAL (*) IPPROCS OPPROCS where (OPPROCS GT 50)
AMQ8409I: Display Queue details.
QUEUE(SPLUNK.Q) TYPE(QLOCAL)
IPPROCS(18) OPPROCS(167)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed. |
Now 167 clinets putting to this queue amd 18 gettting so I want split this queue becouse IBM says ""Spread message load across multiple queue where possible:" |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 6:46 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
| bruce2359 wrote: |
Earlier in this thread you posted stats that showed a large number of rollbacks. We’re the log configurations the 33 and 3 then?
Please re- capture stats now, and post results here. |
Now for QM and all Queues I have
[code][
Enter Type selection
==> SYNCPOINT
Publication received PutDate:20220216 PutTime:14425678 Interval:1 minutes,41.404 seconds
Commit count 139343 1374/sec
Rollback count 47679 470/sec
Publication received PutDate:20220216 PutTime:14430678 Interval:10.000 seconds
Commit count 13604 1360/sec
Rollback count 4680 468/sec
Publication received PutDate:20220216 PutTime:14431678 Interval:10.000 seconds
Commit count 14397 1440/sec
Rollback count 5037 504/sec
Publication received PutDate:20220216 PutTime:14432678 Interval:10.000 seconds
Commit count 14250 1425/sec
Rollback count 4821 482/sec
Publication received PutDate:20220216 PutTime:14433678 Interval:10.000 seconds
Commit count 15229 1523/sec
Rollback count 4761 476/sec
Publication received PutDate:20220216 PutTime:14434678 Interval:10.000 seconds
Commit count 14669 1467/sec
Rollback count 4889 489/sec
Publication received PutDate:20220216 PutTime:14435678 Interval:10.000 seconds
Commit count 13811 1381/sec
Rollback count 4848 485/sec
/code]
around 1400 commits and 500 rollbacks per second |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 7:45 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| an4ous wrote: |
| around 1400 commits and 500 rollbacks per second |
Seems to me to be a high percentage of rollbacks.
I'd suggest an application trace to identify what is driving so many rollbacks.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 8:06 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
| What I must search in FMT files after trace? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Feb 16, 2022 8:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| an4ous wrote: |
| What I must search in FMT files after trace? |
I’d suggest that you first ask developers to explain why so many rollbacks.
What, if anything, do these rollbacks have in common? For example: are they from a particular client? Similar data?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Feb 16, 2022 2:34 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2494
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| an4ous wrote: |
| around 1400 commits and 500 rollbacks per second |
Seems to me to be a high percentage of rollbacks.
I'd suggest an application trace to identify what is driving so many rollbacks. |
There should not be ANY rollbacks. This indicates an issue that needs to be found and fixed. Rollbacks will be impacting overall performance.
_________________
Glenn |
|
| Back to top |
|
|
| an4ous |
| Posted: Wed Feb 16, 2022 11:49 pm Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022
Posts: 38
|
| Where can I find commit/rollback statistic for a specific queue? |
|
| Back to top |
|
|
|
|
