| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Handling undeliverable messages? |
« View previous topic :: View next topic » |
| Author |
Message
|
| Boomn4x4 |
 Posted: Fri Dec 09, 2011 7:39 am Post subject: |
 |
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
| Vitor wrote: |
And how will the message get to the local queue? Does it have a different network to use? Or is it actually local to the queue manager? |
There will be two distinct networks. One internal local network that will host a single queue manager and its own local queues. And, one external network that will connect the clustered queue managers.
There will be several thousand queue managers each on their own local network. These queue managers will be relaying data to a block of queue managers. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Dec 09, 2011 7:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You're talking about a complex MQ network topology crossing multiple hard network boundaries.
I'd suggest you engage the services of an expert to assist. Especially since you are expressing some lack of understand of how MQ moves messages across a network. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 7:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| There will be two distinct networks. One internal local network that will host a single queue manager and its own local queues. And, one external network that will connect the clustered queue managers. |
So if all the clustered queue managers are on the same unreliable external netwrok then the point about performance impact becomes irrelevant, as until the network comes back no message is going anywhere.
| Boomn4x4 wrote: |
| There will be several thousand queue managers each on their own local network. These queue managers will be relaying data to a block of queue managers. |
That's a lot of queue managers; even more important to leverage WMQ's features given the king's ransom you'll be paying in license fee!
You'll want to think about topology. If all these thousands of queue managers are in a single cluster they'll cross connect. You may not want that but prefer to group them, especially as:
| Boomn4x4 wrote: |
| These queue managers will be relaying data to a block of queue managers. |
implies a pub/sub function. You'll do better with that if the block of queue managers are logically grouped.
All this has nothing to do with how messages are handled if the network is down and the fact that WMQ deals with that just fine. Even if there's one single cluster.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Boomn4x4 |
| Posted: Fri Dec 09, 2011 8:18 am Post subject: |
|
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
| mqjeff wrote: |
You're talking about a complex MQ network topology crossing multiple hard network boundaries.
I'd suggest you engage the services of an expert to assist. Especially since you are expressing some lack of understand of how MQ moves messages across a network. |
The contract is in our legal department bouncing back and forth between us and IBM...once the ink is dry, those levels of expertise will be available as will be the training to understand. However, in the mean time, I'm doing what I can to learn what I can. Which is why I'm soliciting help here.
I apologize for being a little vauge as to exactly what I'm trying to accomplish, however, due to the nature of business, I have an ethical responsibility to maintain a level of confidentiality.... stock prices, current vendor relations, competitors, yada yada yada....especially considering that there is no signed contract in place.
| Vitor wrote: |
| as until the network comes back no message is going anywhere. |
This is where my problem lies.... since no messages are going anywhere, where do I put them in the mean time? If I'm putting those messages on a clustered queue on a different queue manager, and that queue manager is unaccessible, I still need to do something with that message.... I cannot wait for the external network to come back up as the local system still needs to process data and generate more and more messages.
If I can take a step back, my initial research led me to local/remote queue functionality. Which worked perfectly... if a local message could not get to the remote queue, it was held locally on the transmission queue until it could be delivered. Perfect! However, being as there will be several thousand queue managers and system maintance concern came up in needed to administrate thousands of local queues, remote queue definitions, transmission queues, and channels.
So, I moved my focus to clustering as it simplifies this removing the need to have transmission queues and channels. However, there is a network reliability concern... and it also now seems that several thousands queue managers on the same cluster can cause inter-connectivity / resource problems.
This led me to the solution in which I'm exploring within this thread in having assured delivery to a local "staging" queue so as not to interrupt local processing AND having assured delivery outside of the local network. This would essentially bypass the need to administrate the overhead of a massive remote/local/channel topology. But, the more I'm reading into this, the more its sounding like I'm recreating a wheel... a wheel in which will never be as reliable as the original, albeit, less complicated to maintain. Simply put, yes, I can write a program to operate outside of WMQ... however, in doing so, I am intorducing my own point of potential failure... one in which WMQ obviously cannot handle, which essential flushes that whole "assured delivery" system down the toilt... thereby making WMQ a multi-million dollar piece of garbage. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 09, 2011 8:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Boomn4x4 wrote: |
| Vitor wrote: |
| as until the network comes back no message is going anywhere. |
This is where my problem lies.... since no messages are going anywhere, where do I put them in the mean time? If I'm putting those messages on a clustered queue on a different queue manager, and that queue manager is unaccessible, I still need to do something with that message.... I cannot wait for the external network to come back up as the local system still needs to process data and generate more and more messages.
If I can take a step back, my initial research led me to local/remote queue functionality. Which worked perfectly... if a local message could not get to the remote queue, it was held locally on the transmission queue until it could be delivered. Perfect! However, being as there will be several thousand queue managers and system maintance concern came up in needed to administrate thousands of local queues, remote queue definitions, transmission queues, and channels.
So, I moved my focus to clustering as it simplifies this removing the need to have transmission queues and channels. |
No. It merely creates the xmit queue and channels for you.
From an application point of view, whether or not it is sending messages to a QLOCAL, a QREMOTE, a QALIAS, or a QCLUSTER is invisible.
So the fact that you are perfectly satisfied with the application logic when you write to a QREMOTE means you should be perfectly satisfied with the application logic when you write to a QCLUSTER.
In either case, you have to deal with the question of what to do when there are lots of messages on a transmission queue. And this question has two parts - a business logic part and an administrative part. The administrative part is to make sure you have enough storage for the expected volume and then configure the queue manager and objects to handle that expected volume.
The business logic part is when is a message "too old"?
The significant difference to keep in mind between managing many sets of sdr/rcvr channel definitions and using an MQ cluster is that an MQ cluster will a) allow message traffic between any two members of the cluster, b) consequently form a fully connected network over time.
You may not want your end-points, particularly if they are external business partners, knowing that they can talk to each other through you. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 09, 2011 8:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20763
Location: LI,NY
|
| Boomn4x4 wrote: |
This is where my problem lies.... since no messages are going anywhere, where do I put them in the mean time? If I'm putting those messages on a clustered queue on a different queue manager, and that queue manager is unaccessible, I still need to do something with that message.... I cannot wait for the external network to come back up as the local system still needs to process data and generate more and more messages. |
No you don't. The pattern here is store and forward. The queue manager will handle that for you. You will however need to make sure you provision for it in queue size, max depth, log size, etc...
| Boomn4x4 wrote: |
If I can take a step back, my initial research led me to local/remote queue functionality. Which worked perfectly... if a local message could not get to the remote queue, it was held locally on the transmission queue until it could be delivered. Perfect! However, being as there will be several thousand queue managers and system maintenance concern came up in needed to administrate thousands of local queues, remote queue definitions, transmission queues, and channels.
So, I moved my focus to clustering as it simplifies this removing the need to have transmission queues and channels. However, there is a network reliability concern... and it also now seems that several thousands queue managers on the same cluster can cause inter-connectivity / resource problems. |
Much less so if the Full Repository are only used for Full Repository duty.
However as your network is unstable, you will probably need one FR in each "network" location. All the PR's from that location will then need to attach to that FR.
| Boomn4x4 wrote: |
| This led me to the solution in which I'm exploring within this thread in having assured delivery to a local "staging" queue so as not to interrupt local processing AND having assured delivery outside of the local network. This would essentially bypass the need to administrate the overhead of a massive remote/local/channel topology. But, the more I'm reading into this, the more its sounding like I'm recreating a wheel... a wheel in which will never be as reliable as the original, albeit, less complicated to maintain. Simply put, yes, I can write a program to operate outside of WMQ... however, in doing so, I am eliminating the assured delivery... thereby defeating the purpose. |
Set up some proof of concept and make sure you include the unreliable network... you'll see things much more clearly then... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 8:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| those levels of expertise will be available as will be the training to understand. |
At least training and mentoring are on the agenda.
| Boomn4x4 wrote: |
| I apologize for being a little vauge as to exactly what I'm trying to accomplish, however, due to the nature of business, I have an ethical responsibility to maintain a level of confidentiality.... stock prices, current vendor relations, competitors, yada yada yada....especially considering that there is no signed contract in place. |
No apology necessary. We've all been there (or are still there).
| Boomn4x4 wrote: |
| This is where my problem lies.... since no messages are going anywhere, where do I put them in the mean time? |
No. Your problem is that you've not grasped that WMQ is dealing with this for you. This is why you're paying the money for it.
| Boomn4x4 wrote: |
| if a local message could not get to the remote queue, it was held locally on the transmission queue until it could be delivered. Perfect! |
So why have you got an issue with a cluster which uses exactly the same principle but doesn't require you to set up the transmission queues??? 
| Boomn4x4 wrote: |
| But, the more I'm reading into this, the more its sounding like I'm recreating a wheel |
You think? Given that one of the key points IBM has been selling WMQ for a decade is it's ability to deal with unrealiable networks?
| Boomn4x4 wrote: |
| ... a wheel in which will never be as reliable as the original, albeit, less complicated to maintain. |
Maintaining a cluster is very, very simple.
| Boomn4x4 wrote: |
| Simply put, yes, I can write a program to operate outside of WMQ... however, in doing so, I am intorducing my own point of potential failure... one in which WMQ obviously cannot handle, which essential flushes that whole "assured delivery" system down the toilt... thereby making WMQ a multi-million dollar piece of garbage. |
Well yes, frankly. Stop fighting this very expensive product, and at least determine it's weaknesses before trying to fix them. Or you end up where you are now, not just fixing a problem that doesn't exist but potentially recreating a problem that wasn't there before you inserted your own code into the design.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Boomn4x4 |
| Posted: Fri Dec 09, 2011 8:57 am Post subject: |
|
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
| Vitor wrote: |
No. Your problem is that you've not grasped that WMQ is dealing with this for you. This is why you're paying the money for it. |
Yes.... that is correct. I was under the misunderstanding that when I put a message on the cluster queue, it went directly there.... When in fact, it went through my local queue manager and it handled the delivery for me... I was not understanding this step.
I just pulled my network cable out of my local box... processed several transactions offline, plugged my network cable back in... and guess what, they all made it to the cluster queue.
Thanks for all of your help... and patience. I'm sure I'll be talking to you again. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Dec 09, 2011 9:00 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Boomn4x4 wrote: |
| Thanks for all of your help... and patience. I'm sure I'll be talking to you again. |
Welcome to the mad-house! 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 9:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| guess what, they all made it to the cluster queue. |
Colour me amazed. 
Now next time you wonder "how will WMQ handle ... ", you'll think back to this, perform a few minutes of experimentation and save yourself hours of posting.
| Boomn4x4 wrote: |
| Thanks for all of your help... and patience. I'm sure I'll be talking to you again. |
I feel sure of it. Until the training and mentoring turns up.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
