Author |
Message
|
queuetip |
Posted: Tue Mar 24, 2009 12:19 pm Post subject: Can You Intentionally Delay a Message's Availabilty? |
|
|
Acolyte
Joined: 03 Feb 2005 Posts: 67
|
Hi!
I have an application requirement to delay a message's availability for a configurable amount of time (e.g. after a msg is put on queue, wait 15 seconds before processing). Messages are less than 1K in size.
I was wondering if MQSeries V7 had any built-in administrative/ application interface options that could accomplish this delay for for us? I didn't see any. Looking for this forum's help for alternatives & recommendation.
Here are the 2 techniques I'm thinking about:
1) Creator application puts message on the data FIFO queue...MQSeries triggers consuming application on FIRST...consuming application opens data queue and gets message off immediately...add the delay time (e.g. 15 seconds) to the PutApplicationDateTime field...retrieve the system current datetimestamp.
If the current time is less than calculated time, application issues a wait for the difference before continue processing; otherwise continue processing.
2) Creator application builds message with delay time (e.g. 15 secs) in the expiry field, specifiy report option of MQRO_EXPIRATION_WITH_FULL_DATA & specify a ReplyToQ in the MQMD. Creator puts message on FIFO queue...MQSeries triggers consuming application on FIRST...consuming application opens both data queue (only to activate discard process) and ReplyToQ....consuming application gets messages on ReplyToQ with delay time as wait interval (knowing that delay has expired).
Any other alternative techniques? Recommendations?
Thank you!
Mike
|
|
Back to top |
|
|
RogerLacroix |
Posted: Tue Mar 24, 2009 12:43 pm Post subject: Re: Can You Intentionally Delay a Message's Availabilty? |
|
|
Jedi Knight
Joined: 15 May 2001 Posts: 3258 Location: London, ON Canada
|
queuetip wrote: |
I have an application requirement to delay a message's availability for a configurable amount of time (e.g. after a msg is put on queue, wait 15 seconds before processing). Messages are less than 1K in size. |
Why? This is a bad idea that will require a really bad design to be implemented.
Please tell your boss/management that it is best to simply consume the messages as they arrive.
Regards,
Roger Lacroix
Capitalware Inc. _________________ Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
Back to top |
|
|
mqjeff |
Posted: Tue Mar 24, 2009 12:45 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
It's not clear at all why it's a good idea to delay a message after it's been sent, rather than delay sending it in the first place.
No, MQ doesn't have a function for this because it's a really wasteful idea.
You can easily do this with Mesage Broker, though, and the Timeout nodes. If you don't see the need for a full enterprise license, you can purchase the very reasonably priced Starter Edition. |
|
Back to top |
|
|
Vitor |
Posted: Tue Mar 24, 2009 12:48 pm Post subject: Re: Can You Intentionally Delay a Message's Availabilty? |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
queuetip wrote: |
Recommendations? |
Ask whoever gave you the requirement if they're serious? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
queuetip |
Posted: Tue Mar 24, 2009 1:36 pm Post subject: |
|
|
Acolyte
Joined: 03 Feb 2005 Posts: 67
|
C'mon guys...in general, I agree with you but if you are truly experienced, you'd be a little more humble & realize there are always some unique processing requirements!
Besides if having message delay is so wrong, maybe we should ask why Oracle AQ extended their product with this option?
Seriously, I really would like your help...you guys have helped me out many times before. I will elaborate to help you better understand my scenario a little more...
The reason the application creating the message can't wait is that it is a real-time transaction and a clerk is expecting a sub-second response (we have SLAs)! On the back-end, we are coordinating data updates with a 3rd party system (outside the MQSeries unit of work). These messages are used to drive a background process that reads databases to build data messages that are fed near-real time to this other system. Since the same message can be produced by multiple clerks working independently and concurrently, we have determined we can eliminate creating a significant amount of "duplicate" data messages if we simply wait a few seconds due to the nature of the activity. We needed to establish a boundary for what is classified as duplicate vs. changes for the sake of the 3rd party system.
So this is why I am interested in possible delay techniques. Hoping this helps support our need for the intentional delay.
Again, thanks for understanding about the business environment & requirements. I look forward to any ideas you may have to help me accomplish the delay.
Thanks!
Mike
|
|
Back to top |
|
|
bruce2359 |
Posted: Tue Mar 24, 2009 2:01 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
Quote: |
(e.g. after a msg is put on queue, wait 15 seconds before processing)... it is a real-time transaction ... |
If you are wondering why we all look so confused, the above is why.
Let me see if I understand the requirement: a message arrives AND needs to be processed immediately (real time); processing of the input message creates a new message that is used to update a back-end database. OK, so far?
If the output message is the one that needs to be delayed, the application that consumes it can MQGET it from the queue, look at the create date/time, and wait (using o/s facilities possibly) until the 15 seconds pass since its creation, then do the back-end work.
If I didn't understand the requirement, try again. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
kevinf2349 |
Posted: Tue Mar 24, 2009 6:02 pm Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003 Posts: 1311 Location: USA
|
Write an application to consume messages and only send on the ones you want it to to the back end 3rd party process. Either use a database to store the messages (thorw away duplicate keys) or use whatever criteria you use to decide it is a wasted message to throw away the worthless ones.
How are you going to determine which are the 'chosen ones?' amongst your messages and which can be discarded?
If you simply want to delay all of them then write an application with a wait coded into it....not a MQ GET WAIT... and loop around it, but this could lead to serious backlogs.
If we are voting on this as a MQ requirement in the futre ....I vote Nay! |
|
Back to top |
|
|
Vitor |
Posted: Tue Mar 24, 2009 9:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
queuetip wrote: |
we have determined we can eliminate creating a significant amount of "duplicate" data messages if we simply wait a few seconds due to the nature of the activity. |
I'll ask again if these people are serious.
Waiting "a few" seconds is a ghastly way of determining this. You've clearly set "a few" at 15 but on what is that based? If the auditors come for you, what will be your defence?
If you're worried about significant numbers of "duplicate" updates to the backend system, then put a simple filtering mechanism either into the system or as a gateway in front of it. Not only is this defensible in audit terms, but will save you weeks of tuning as you struggle to determine the optimum "a few" setting.
And I have no comment on why Oracle AO has been extended with this option. Expect to point out on a previous site, with a lot of Oracle databases, the comment was "why Oracle AO?" when changing the middleware was mooted! _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
rxm8778 |
Posted: Wed Mar 25, 2009 4:24 pm Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005 Posts: 37
|
My two cents on this...
I don't think delaying the processing of a message should ever be the responsibilty of MQ.
Here is an analogy...
you can't go to the post office to mail a package to a friend and ask the post office to hold the package for 10 days before sending it. If you want to truly control when the person get the package, it is up to the package sender not to go to the post office until he is ready to send the package, or it is up to the package recepient to pick up the package when he is ready, or pick it up and not open it until he is ready. Those are the three options available.
With that said you should look into delaying the message producer and not produce the message until ready....keep the message in the queue and not getting it until ready or pick up the message but delaying processing the message until ready.
That would be my approach to your requirement....
p.s you may argue that in the post office analogy, you can select overnight vs ground transportation. But remember these options were all designed to optimized the package delivery (based on how much you are willing to pay) they were not created to specifically introduce a delay betwen package posting and delivery.
Last edited by rxm8778 on Fri Mar 27, 2009 10:18 am; edited 1 time in total |
|
Back to top |
|
|
Michael Dag |
Posted: Thu Mar 26, 2009 1:30 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002 Posts: 2602 Location: The Netherlands (Amsterdam)
|
actually ... I have seen this sort of requirement on MQ in the past,
it's also known as the "do not open until christmas" requirement.
basically it allows for a "pattern" to prefeed information into an organisation/unit, but all participants need to have access at the same
time, this can not be done in a normal way due to volume/processing without prefeeding (i.e. someone will get the information first or before others and will be at an advantage)
so given the above pattern I can see a "need", but no the requirement was never fulfilled. _________________ Michael
MQSystems Facebook page |
|
Back to top |
|
|
Vitor |
Posted: Thu Mar 26, 2009 1:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Michael Dag wrote: |
actually ... I have seen this sort of requirement on MQ in the past,
it's also known as the "do not open until christmas" requirement.
|
Even with this, wouldn't you be better off processing the information immediately (i.e. not delaying the inbound message), warehousing the data someplace until it was complete / it was Christmas and then sending it (pub/sub??) to all interested parties? You could secure the warehouse to stop recepients "shaking the box" and getting a clue.
What an odd requirement. The ingenuity and imagination of end users is a source of constant amazement to me. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
rtsujimoto |
Posted: Thu Mar 26, 2009 9:55 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004 Posts: 119 Location: Lake Success, NY
|
I don't want to judge whether or not your biz reqm'ts make sense or not. Sometimes the ultimate decision is truly out of your hands and it's your job to meet that reqm't. That aside, I find this sort of thing challenging. How about this:
1. Use 1 pgm to monitor the queue
2. Said pgm builds an internal table, where entry is: msgid, delay, delta
3. Table entries order by delta
4. Said pgm scans internal table, finds candidate, gets msg and puts to a
triggered queue, updates table
5. Said pgm also scans inbound queue periodically, updating table as
needed
Issues such as recoverability, speed, etc. need to be considered, but this is a start. |
|
Back to top |
|
|
PeterPotkay |
Posted: Fri Mar 27, 2009 9:44 am Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
A weird requirement, so here's a weird solution....
The consuming app is constantly checking the Oldest Message Age of the queue. Anytime it sees this value > 15 seconds, it gets one message, processes it, and then goes back to check the Oldest Message Age of the queue in a loop.
This design does not preclude multiple instances of the consuming app all running concurrently.
_________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
mqjeff |
Posted: Fri Mar 27, 2009 10:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
And if every single message is delayed by 15 seconds, then messages that would have conflicted 15 seconds ago because they arrived concurrently will... still conflict because they still arrived concurrently. Just 15 seconds later.
Best bet is to use WMB, and use the Timeout nodes to reprocess individual messages. |
|
Back to top |
|
|
PeterPotkay |
Posted: Fri Mar 27, 2009 10:10 am Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
Jeff, I don't understand your comment. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
|