| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Getting Expired messages |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Fri May 18, 2001 2:56 pm Post subject: |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
A quote from the online MQ reference manual:
After a message's expiry time has elapsed, it becomes eligible to be discarded by the queue manager. In the current implementations, the message is discarded when a browse or nonbrowse MQGET call occurs that would have returned the message had it not already expired. For example, a nonbrowse MQGET call with the MatchOptions field in MQGMO set to MQMO_NONE reading from a FIFO ordered queue will cause all the expired messages to be discarded up to the first unexpired message. With a priority ordered queue, the same call will discard expired messages of higher priority and messages of an equal priority that arrived on the queue before the first unexpired message.
My test:
1. I put a message to a Q with a CorrelID of JOESHMOE and an Expiry of 120 seconds.
2. For the next couple of minutes, I can look at the message in QPASA and if I look at the properties of the Q, the QDEPTH of the Q is one.
3.During these 2 minutes I also do GETS with a CORRELID of PETERPOTKAY. I keep getting back 2033s, as I should, because the only message on the Q has a CORRELID of JOESHMOE. After each of these GET attempts, the Q Depth remains at 1. (As a seperate test I did GETS with a JOESHMOE CORRELID just to make sure I can get the message.)
4. We wait a bit and now the message has expired. If I look at the Q thru QPASA, there are no messages to look at, BUT the depth is still 1. This is expected and is acting as documented, i.e. expired messages count towards Q depth.
5. I repeat step 3. I get a 2033. I go back to QPASA and look at the Q Depth, it is now ZERO. A GET that would not have returned the message had it not already expired has cleaned up the Q, it direct contrast to what the manual says above!
So the next thing I did was put 2 messages with 60 second EXPIRY times and JOESHMOE CORRELIDs, an UNLIMITED EXPIRY message with PETERPOTKAY, and 2 more JOESHMOE messages with 60 second EXPIRY times. The order of the messages in the FIFI Q would look like the following...........
JOESHMOE
JOESHMOE
PETERPOTKAY
JOESHMOE
JOESHMOE
After 2 minutes all the JOESHMOE messages are expired and PETERPOTKAY is still unlimited. I check the Q DEPTH and it is 5. I BROWSE the Q in QPASA and only see PETERPOTKAY. I do a get with CORRELID of PETERPOTKAY and get back PETERPOTKAY message. Go look at the Q, and it looks empty. Check the Q DEPTH and it says 2! The first 2 JOESHMOE messages got cleaned up as MQ went thru the FIFO Q looking for PETER. The last 2 were ignored (until I did another GET for PETER and then they went away).
Is the manual wrong? Is QPASA wrong?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Fri May 18, 2001 3:09 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
OS/390 Version 2.1, if it makes any difference.....
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bduncan |
| Posted: Fri May 18, 2001 3:48 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Peter,
I may be wrong, but I believe the documentation AND your observations are correct, but the documentation could have been a little more specific when describing the rules about discarding expired messages. I think an example may most easily illustrate what I have observed in the past:
1. queue has 100 messages on it.
2. all messages have the same priority.
3. the first 50 messages are expired, the second 50 have not.
4. all 100 messages have a unique correlid.
5. now, I do an MQGET matching against correlid=BRANDON, and there is one message on the queue with this correlid. More specifically, this is one of the 50 that are NOT expired. In other words, it is in the second group of 50.
6. Since all the messages on the queue have the same priority, the FIFO rules should apply, and MQSeries will begin matching against each message starting with the first looking for a correlid of BRANDON.
7. Assuming the correct message is message #75, then MQ will have to "touch" the first 74 messages on the queue before it comes to the message with the proper correlid.
8. As each expired message is "touched" it is discarded - in accordance with what you observed.
9. When the manual explains the conditions for discarding an expired message: "an MQGET that would otherwise give you the message were it not expired" it is explaining the most simplistic situation where there are no matching options. In the same way that message priorities can affect the discarding algorithm, so can matching options. In the simplest sense, MQSeries is searching through an unsorted array, element by element, looking for a specific value. By searching in this fashion, it must "touch" every element in the array until it arrives at the desired element. Part of this touching function says that because we are doing an MQGET (that COULD yield this message were we not using any matching options), and this particular message is expired, it should be discarded.
This also explains the results of your second test. Because the PETER message was #3, MQSeries had to match against 1 and 2 before it arrived at 3. 1 and 2 had correlids of JOE, and were expired, so in touching them it removed them. Then the third message yielded the correct correlid, and it was returned to you. Messages 4 and 5 were never "touched" so even though they are expired, they are still there.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 21, 2001 4:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Thanks Brandon, I suppose if you read between the lines in the online manual instead of taking it literally, then it all makes sense. The real reason all this came about was because of a discussion on nightly clean up jobs. In other words, should a shop have a job that runs once every 24 hours that simply does a plain vanilla GET to flush the Queue of any Expired messsages.
With my previous interpetation of the manual, I would say yes. After my tests and your explanation of the manual, I would now say no. Do you agree?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bduncan |
| Posted: Mon May 21, 2001 9:22 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Peter,
We have had similar issues with queues filling up with expired messages over time. If you just start putting messages to a queue, and they expire, but you never do a get against that queue, then yes, eventually the queue will fill up, or the filesystem will fill up. If you are worried about either of these two scenarios, then you are on the right track with a "getter" program, but keep in mind that if you do a vanilla get, you will not only "touch" the expired messages, but pull off any non-expired messages. A better solution may be to do a get against the queue matching for a correllid that you know won't exist. This way you will touch every message without actually removing any non-expired ones...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 21, 2001 9:26 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Ah, but wouldn't the next "real" GET, done the next morning, accomplish the same thing, negating the need for these clean-up type utilities?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| HarveyHarrison |
| Posted: Wed May 23, 2001 7:09 am Post subject: |
|
|
Newbie
Joined: 22 May 2001
Posts: 1
|
Peter is correct - the get would discard the expired messages.
The expired message discarding behavior is different on some platforms - for example MQ 5.1 for Compaq NSK, the physical removal of messages is based on a housekeeping timed event rather than when GETs occur.
Harvey Harrison
MQSeries Kernel Architect |
|
| Back to top |
|
|
| bduncan |
| Posted: Wed May 23, 2001 9:36 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
But Peter, the next "real" GET might not run through the entire queue. In other words, if the message you eventually pull off is halfway through the FIFO queue, then all the messages that come after it were never touched, hence if any are expired they won't be discarded. The advantage of a "getter" program is that it is guaranteed to "touch" every message on the queue.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Jun 02, 2001 1:14 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5856
Location: UK
|
Either do a MQGET for a MSGID and CORRELID combination that will not exist, or loop around doing MQGETs under syncpoint until "no msg available" then BACKOUT the entire UOW.
This will work for all platforms. See the May or June issue (I forget which) of MQ Update, see "Journals" at http://www.xephon.com
[ This Message was edited by: zpat on 2001-06-02 14:16 ] |
|
| Back to top |
|
|
| RichArnfield |
| Posted: Mon Jun 04, 2001 4:22 am Post subject: |
|
|
Newbie
Joined: 24 May 2001
Posts: 4
Location: Currently London, soon Frankfurt
|
Hi Folks,
How about this for a possible solution, for cleaning a queue of expired messages.
Put a dummy message with a specific CorrelId or even MsgId.
Make sure it is committedto the queue.
Then get it off again.
It's gone on the end of the queue, so every single message on the queue will be 'touched' in the process of getting it off again. If you make the Correl/MsgId specific or unique enough, the chances of any of the other messages matching it should be pretty low.
Reckon that would work?
Cheers,
Rich.
_________________
Rich Arnfield, Castlemay Ltd.
Certified MQSeries Solutions Expert |
|
| Back to top |
|
|
| Cliff |
| Posted: Mon Jul 02, 2001 4:13 am Post subject: |
|
|
Centurion
Joined: 27 Jun 2001
Posts: 145
Location: Wiltshire
|
| Why not simply browse the whole queue? The manual states 'In the current implementations, the message is discarded when a browse or nonbrowse MQGET call occurs that would have returned the message had it not already expired. ' so surely a browse program accepting truncated messages will work for all? |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jul 02, 2001 6:38 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5856
Location: UK
|
| The browse method does not work on OS/390 (or so I understand). |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
