Author |
Message
|
melis |
Posted: Mon Apr 08, 2002 6:55 am Post subject: |
|
|
Newbie
Joined: 07 Apr 2002 Posts: 3
|
Hi,
The normal use for a queue would be to handle messages on a FIFO-basis. However in our project the need arises to get a specific message based on a certain value. The MQ-gurus advise against this solution mainly because it wouldn't perform when using a large number of messages. However they don't seem to have any papers which could give their advice some more "body".
Can anybode direct me to some kind of paper which gives some test results on this situation ? Or does anybode have some hands-on experience in what the performance penalty will be when using this ?
Thanks in advance
_________________ Regards,
Melis |
|
Back to top |
|
|
bduncan |
Posted: Mon Apr 08, 2002 4:02 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001 Posts: 1554 Location: Silicon Valley
|
I had some technical papers that I presented at a former company which compared selective versus non-selective gets. While there are times you must use selective gets, I say avoid it at all costs, especially when dealing with high volumes of messages. This comes down to a very basic computer science question of constant time versus n time. However there are other factors which make the practical search time closer to constant versus n squared! Don't get me wrong, selective gets for MQSeries are still very fast (compared to the other messaging systems out there), but I recently worked for a fortune 500 company, where for nearly 3 days our entire supply-chain/procurement system came to a halt - causing empty trucks to sit at loading docks around the country for one simple reason - our queues had gotten too big! Basically, a system outage caused our mainframe to be unable to process messages coming in from our warehouses. These messages piled up on queues, and when the mainframe started processing messages again, we hit a wall. During the outage, the queue depths had gone to over 400,000 apiece (there were 16 of them). For each queue, we were doing selective gets. Normally, the rate at which the mainframe could process the messages was faster than the rate at which they arrived from the warehouse systems. Unfortunately, because the queue depth had gotten so large, the search time took much longer, and the rate at which the mainframe could process the messages fell below the rate at which the messages were arriving (because the warehouses had never stopped sending their messages even during the outage) and so our queues started getting even bigger, rather than smaller, which made the throughput even slower! Before our queues hit 640,000 (and all hell would break loose) we shut down all the channels from the warehouses (save one) and only process that one queue on the mainframe side. The funny thing about the whole affair was that the time to process a given message seemed to increase exponentially as the queue depth went up, rather than linearly (as one would expect). So it ended up taking 3 days for our mainframe to slowly whittle away at each queue and catch back up. We had to have admins sitting there 24/7 throttling the channels going to mainframe so we could keep the queue depths at managable levels until it caught up with the warehouses. This little mistake cost us a LOT of time and money.
_________________ Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
Back to top |
|
|
PeterPotkay |
Posted: Tue Apr 09, 2002 9:23 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
Brandon, just trying to understand...
If the mainframe queue had 0 + n messages on a queue, why wouldn't it procees all those requests FIFO? WHat was it matching on? This implies that some messages never got looked at or there was some specific order for these things to process in, right?
_________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
oz1ccg |
Posted: Tue Apr 09, 2002 11:18 pm Post subject: |
|
|
Yatiri
Joined: 10 Feb 2002 Posts: 628 Location: Denmark
|
Hi,
I say the same situation on one of my big customers, that the processing time way increasing dramaticly, but our reason was that the queue was defined without index.... (CorrelId) After enabling that our problem disapered.
We handled in the same aount of messages every day in a crossplatform application.
_________________ Regards, Jørgen
Home of BlockIP2, the last free MQ Security exit ver. 3.00
Cert. on WMQ, WBIMB, SWIFT. |
|
Back to top |
|
|
zpat |
Posted: Wed Apr 10, 2002 1:51 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001 Posts: 5856 Location: UK
|
Index the queues on your selection criteria! Use expiry times where possible and purge expired messages (this is another subject in itself). |
|
Back to top |
|
|
melis |
Posted: Wed Apr 10, 2002 4:22 am Post subject: |
|
|
Newbie
Joined: 07 Apr 2002 Posts: 3
|
Brian,
Thanks for your sharing your experience. Is there any chance that you could send me the papers you mentioned in your reply (perhaps by omitting the client in question, but with all te relevant performance data) ?
Another question: you mentioned that the problem begun when the queues had a depth of 400.000 messages (or more). Is there some kind of break-even-point at which rate you shouldn't be doing selective reads but should be handling them on a FIFO-basis ?
Someone else mentioned the use of indexing could be a solution to this problem. Was that used in your case and what where the results of that ?
_________________ Regards,
Melis |
|
Back to top |
|
|
melis |
Posted: Wed Apr 10, 2002 4:24 am Post subject: |
|
|
Newbie
Joined: 07 Apr 2002 Posts: 3
|
Jorgen,
> I say the same situation on one of my big customers, that the processing time way increasing dramaticly, but our reason was that the queue was defined without index.... (CorrelId) After enabling that our problem disapered.
We handled in the same aount of messages every day in a crossplatform application.
Can you tell something about the number of messages on that queue and the hardware/software-combination you were using at the time ?
_________________ Regards,
Melis |
|
Back to top |
|
|
bduncan |
Posted: Wed Apr 10, 2002 11:14 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001 Posts: 1554 Location: Silicon Valley
|
Peter, to answer your question about WHY (which is a very good question indeed!) let me just preface it with saying I WASN'T the person who designed the architecture - I just had to live with the aftermath. Basically we were running MQV2 on the mainframe (and let me tell you how afraid everyone was of the prospect of upgrading!) furthermore, they designed this system before the patch for V2 was available that allowed message grouping. So the guys who designed the system wanted to do message grouping, even though their version of MQ on the mainframe didn't support it. So what did they do? They had one queue, where a single message for each logical group would arrive. This message would contain the Correlation IDs of all the messages belonging to that group. An application would get this message (in FIFO order) and then begin doing MQGETs on another queue (specifying each of the CorrelIds) to get the messages that belonged to that group. But (for reasons which would take forever to describe) those MQGETs didn't actually remove the messages - they just BROWSED them! Another application would come back and remove (MQGET) the messages that had been "processed" again doing another search on the queue by CorrelID... As you can imagine this was terribly inefficient. 1+(n^m)+1+(n^m) MQGETs for a single group, where n is the number of messages in the queue, and m is the number of messages for the current group.
I'm not sure if I still have my performance data... I'll have to look to see if I kept those visio charts...
_________________ Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
Back to top |
|
|
lnm |
Posted: Fri Apr 26, 2002 9:17 am Post subject: |
|
|
Apprentice
Joined: 12 Mar 2002 Posts: 43 Location: Florida
|
Is indexing an option that is set on a queue?
Indexing like in a table?
I'm confused.
Where is there documentation on this? |
|
Back to top |
|
|
oz1ccg |
Posted: Fri Apr 26, 2002 9:26 am Post subject: |
|
|
Yatiri
Joined: 10 Feb 2002 Posts: 628 Location: Denmark
|
When you define the queue it's posibble to tell MQSeries how you normally will retrieve messages (by messageid or correlid), when MQSeries is told that you want to use one of those options, MQseries will keep an instorage index of those messages so it very quick can retrive the correct one without having to read thru the whole queue. (that was my problem with the big customer +100.000 msgs per queue) The senario was OS/390 and AIX.
_________________ Regards, Jørgen
Home of BlockIP2, the last free MQ Security exit ver. 3.00
Cert. on WMQ, WBIMB, SWIFT. |
|
Back to top |
|
|
|