Author |
Message
|
mqdogsbody |
Posted: Tue May 26, 2015 2:40 am Post subject: Efficiency of processing messages in two phases |
|
|
Acolyte
Joined: 01 Jun 2010 Posts: 71
|
Current we use MQ in a simple way – getting and consuming each message in separate work-unit in a FIFO manner.
A colleague is seeking input on the feasibility of changing to a scheme whereby messages are browsed but not consumed from the queue, and then sometime later removed/consumed from the queue based on their MessageId (or other properties read from the message when it is originally browsed). The application would be responsible for retaining the MessageIds of browsed and processed messages until they have been consumed/deleted from the queue and it might be a considerable time later that the message is eventually consumed/deleted, even up to the end-of-day.
From research and manual pages the basic mechanisms appear to exist in terms of being able to MQGET’ing messages that match a MessageId. However given that the general thrust of the documentation seems to be based on obtaining the ‘First message that matches MsgId’, My colleague is concerned about the scalability of application issuing MQGET calls for MessageIds that are a long way from the front of the queue. Can anyone advise of the relative performance of MQGET’ing messages that are far from the front of the queue?
Similarly, in a recovery scenario, my colleague is concerned about the time it might take to issuing ‘Browse-next’ operations to wind-forward past possibly 10’s of thousands of previously browsed and processed messages to place in the queue where the next un-processed message can be found if this is not generally the way MQ is supposed to be used. Can anyone advise on the sensibility of such an approach? _________________ -- mqDB --
Last edited by mqdogsbody on Tue May 26, 2015 3:32 am; edited 1 time in total |
|
Back to top |
|
|
PeterPotkay |
Posted: Tue May 26, 2015 3:02 am Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
You've proposed a solution - browsing messages, storing message IDs, coming back later to get them by message ID.
But you haven't explained why you would want to do this. What is the problem you are trying to solve? _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
pmb999 |
Posted: Tue May 26, 2015 4:04 am Post subject: Re: Efficiency of processing messages in two phases |
|
|
Newbie
Joined: 16 Jul 2009 Posts: 5
|
The time that takes to fully process a message is relatively long, where that processing entails writing to a complex SQL database. So the problem is to decouple the (partially) processing of a message and sending a response from the eventual completion of persisting data to the SQL database, and then removing the originating MQ message. In effect, we'd like to be able to keep the input message on MQ until much later. Also, to further aid the performance and response-time of the application, we'd like to minimise the MQ overhead incurred before sending a response, hence the preference for simply leaving it on the received queue, rather than redirecting it to another for later removal. |
|
Back to top |
|
|
fjb_saper |
Posted: Tue May 26, 2015 4:45 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003 Posts: 20729 Location: LI,NY
|
What's wrong with moving the message to a different queue for later removal? It keeps your main process running smooth, fast (in memory hand off) and efficiently.
Leaving the message on the queue will on the contrary maximize the impact on the MQ side. Reading constantly the queue, having to load the catalog, forcing incoming messages to be written to disk, etc... _________________ MQ & Broker admin |
|
Back to top |
|
|
pmb999 |
Posted: Fri Jun 05, 2015 8:21 am Post subject: Re-queueing message to different queue...? |
|
|
Newbie
Joined: 16 Jul 2009 Posts: 5
|
If, as the last posting suggests, we re-queue the received message to a different queue, is there any particular support for doing that, or is it a case of issueing an MQ PUT to a different queue to on-forward the incoming message?
I was wondering whether MQ had any 'fast-path' like feature to re-queue a message on a different queue without needing to transfer the message into and out of the application? |
|
Back to top |
|
|
mqjeff |
Posted: Fri Jun 05, 2015 9:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Is it clear that you don't want to use syncpoint?
I know of customers that are using a browse followed by a later get. They get reasonable performance.
But depending on your timing, you should probably just look at holding the MQ message in a syncpoint. |
|
Back to top |
|
|
RogerLacroix |
Posted: Fri Jun 05, 2015 10:58 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001 Posts: 3258 Location: London, ON Canada
|
Hi,
Well, here's a solution but it involves a commercial product (since you do not want to handle the movement of a message from queue A to queue B).
Let's say your original input queue is 'Q.A'. Create a new queue called 'Q.B'.
- Purchase and install MQ Message Replication (MQMR)
- Configure MQMR to replicate messages from 'Q.A' to 'Q.B'
- Have your application continue to perform destructive MQGets (not browse) against 'Q.A' (the original queue) and save the MessageID as you mentioned above.
- Have your other slow backend system process messages from 'Q.B' using the saved MessageID.
Problem solved. Your original queue ('Q.A') will always be at or near zero for queue depth, your application does not need to copy/forward messages to another queue and your slow backend system can process the messages whenever it gets to it.
Personally, I don't think you need to save the MessageID because the slow backend application will simply get the next message off 'Q.B' and process it. Both queue 'Q.A' (for your speedy frontend application) and queue 'Q.B' (for the slow backend application) get the EXACT same message and in the EXACT same order. So, just have each application do their thing.
Note: When MQMR replicates a message both the message data and the message's MQMD structure will be cloned. (i.e. PutTime, MessageId, CorrelId, UserId, etc..)
Food for thought.
Regards,
Roger Lacroix
Capitalware Inc. _________________ Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
Back to top |
|
|
pmb999 |
Posted: Sat Jun 06, 2015 5:10 am Post subject: Thanks |
|
|
Newbie
Joined: 16 Jul 2009 Posts: 5
|
Thank you for the further replies.
I don't think retaining messages within a sync-point can address a sustained discrepancy between the required (initial) processing speed and the down-stream processing speed. You may question how such a discrepancy will ever be resolved, but it can be addressed by periodically leveraging 'bulk' operations on (the results of) batches of MQ requests. I don't think it would be practical to retain sync-points while those batches are assembled, as the accumulation period might be several minutes or more in the event of site failure and disaster recovery.
The MQ replicator product could be a useful get out of jail solution if we take the suggested approach of replicating the input to another queue at the application level but run into performance barriers (though unsure how open my client is to introducing new products into its MQ infrastructure.) |
|
Back to top |
|
|
fjb_saper |
Posted: Sat Jun 06, 2015 7:43 am Post subject: Re: Re-queueing message to different queue...? |
|
|
Grand High Poobah
Joined: 18 Nov 2003 Posts: 20729 Location: LI,NY
|
pmb999 wrote: |
If, as the last posting suggests, we re-queue the received message to a different queue, is there any particular support for doing that, or is it a case of issueing an MQ PUT to a different queue to on-forward the incoming message? |
If you are able to change the application, doing a put to a different queue to on-forward the message is ideal.
You can then handle the next queue with syncpoint and batch size as desired without impacting the fast response of the first app. _________________ MQ & Broker admin |
|
Back to top |
|
|
|