| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Appending message records to a pre-existing file |
« View previous topic :: View next topic » |
| Author |
Message
|
| zpat |
 Posted: Fri Feb 12, 2010 2:58 am Post subject: |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
The file system space could fill up in the same way, for the interim file.
Basic capacity planning needs to be applied. What is the max daily volume - ensure enough queue space (I generally allow 3 times as much).
Queue depths are easy to alert on. In any case if the queues filled - MQ would stop accepting messages - they would not be lost.
This is a batch design (not ideal in the first place) - continous processing is better, but someone wanted a file once a day.
I don't think there is a risk of thousands becoming millions without some sort of major change which would need appropriate planning for.
I agree with not storing huge volumes on queues but the volumes have not been stated here. The queue maxdepth should be chosen not to exceed the queue storage space available (ditto for the DLQ).
Otherwise any looping application could cause problems. |
|
| Back to top |
|
 |
| Gralgrathor |
| Posted: Fri Feb 12, 2010 3:41 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| zpat wrote: |
| In any case if the queues filled - MQ would stop accepting messages - they would not be lost. |
Which is exactly the problem, in my opinion. Most broker architectures are optimized for continuous processing. Wherever MQ would stop collecting messages, there would come into being a backlog of unprocessed messages. This could be for one flow, it could be for any flow that uses a certain transmission queue.
I am currently working on a broker flow which directs messages during a nightly batch processing. Because of queue limitations, I have decided not to use queues, but in stead attach extra drive space (in a storage area) to the broker file system, and have the broker read directly from files on this space, and write outgoing messages directly to files located on this space.
I too would like to find some way of having the broker append records to an already existing file. But it's not a showstopper if this isn't possible: the target applications are able to process partial files, or otherwise a script running on the broker or target machine could concatenate the files. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 12, 2010 4:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Gralgrathor wrote: |
Because of queue limitations, I have decided not to use queues, but in stead attach extra drive space (in a storage area) to the broker file system, and have the broker read directly from files on this space, and write outgoing messages directly to files located on this space.
|
It's an interesting decision. Where I've been faced with this I've ensured there's adequate space for the queues to fill, seeing this as easier and cleaner than introducing a separate file step into the design to get round a perceived technical problem. I've also sized the space to allow for 3-5 days of messages depending on the customer requirements to allow for outages. Yes this wastes a lot of space in normal operation but disc is cheap. So even with daily messages in the millions (which I've seen on a number of occassions) it's cost effective. And any queue, irrespective of potential depth, should be monitored to detect abnormal depths.
So I stand by my advice. Hold 1 day's worth of messages in the queue then process them, have enough space for a few days and a properly monitored system.
There's certainly no reason not to do that just because "the MQ admin's don't like big queue depths".
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Feb 12, 2010 4:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
Expediency usually causes long-term pain.
I prefer to accomodate the best application design, rather than taking the path of least implementation resistance.
If the queue file space needs increasing, then it's worth doing it as it will then allow future designs to use it.
Make it a lot more than you think you will need!
Last edited by zpat on Fri Feb 12, 2010 6:12 am; edited 1 time in total |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 12, 2010 4:32 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Vitor wrote: |
| Where I've been faced with this I've ensured there's adequate space for the queues to fill. |
Unfortunately, the broker is not ideally suited for the kind of batch processing I'm working on. We're talking up to 1G of data to be processed in a window of about 8 hours. We would have considered alternatives to using the message broker for these batches, but due to management concerns and time limitations, we're stuck with using WMB.
Since there is no way to "time" the reading of the next line in a FileInput node, what happens is that it will start reading and not stop until the file is completely read. If I use queues, that means that in the worst case scenario the entire file (1G) will at some time reside on a queue. Clearly, even if the broker has the capacity for such large queues, this is not a desirable situation.
My solution was to integrate all steps in the processing chain into one transaction, using subflows. One line (csv) in equals one line out, without intervening queues. The source file is first split up into segments (using a seperate broker flow), and the segments are then processed by multiple instances of the main flow. I'm curious to see what performance issues will rise from this approach...
| Vitor wrote: |
| Yes this wastes a lot of space in normal operation but disc is cheap |
Not really. In an organization as the one I am working in, there are many hundreds of applications requiring fast access diskspace. Even a request for one single G of space takes some doing to get approved.
| Vitor wrote: |
| So I stand by my advice |
It is good advice, provided you can guarantee that the batches remain limited in size, and that messages are either periodically processed, or thrown away, as to prevent queues flooding. I can do neither in my case. But then, as I already said, the broker is not ideally suited for this kind of processing.
| Vitor wrote: |
| There's certainly no reason not to do that just because "the MQ admin's don't like big queue depths" |
But admins usually have good reasons to dislike them. Having a large queue fill up completely can cause serious performance drawbacks. Keeping queues small forces one to design processes so that there is little risk of large amounts of data becoming lodged on the broker. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 12, 2010 5:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Gralgrathor wrote: |
| My solution was to integrate all steps in the processing chain into one transaction, using subflows. One line (csv) in equals one line out, without intervening queues. The source file is first split up into segments (using a seperate broker flow), and the segments are then processed by multiple instances of the main flow. I'm curious to see what performance issues will rise from this approach... |
Whatever works for you is clearly good for you. FWIW when I've used the kind of scenario you describe I've seen no particular performance issues aside from those you would get with any flow (poor code, slow database, etc, etc). I've linked them together using queues, but that doesn't stop the basic comparison.
| Gralgrathor wrote: |
| Not really. In an organization as the one I am working in, there are many hundreds of applications requiring fast access diskspace. Even a request for one single G of space takes some doing to get approved. |
Interesting. In a high volume environment such as you describe I'd expect otherwise. Certainly my experience is that disc is at the cheaper end of the resource tree. Case in point; customer few days back got an out of space warning on an AIX machine. They're using linear logs & don't have housekeeping (!) so there was a plan to do a media image and archive the old files. Customer decided that as the disc filling wasn't a problem (like an application looping) it was easier to mount another 100Gb to the log directory "for now" and revisit the problem when they were less busy.....
We also have a z/OS customer who has each local queue on a separate 3390 but that's just silly 
| Gralgrathor wrote: |
| It is good advice, provided you can guarantee that the batches remain limited in size, and that messages are either periodically processed, or thrown away, as to prevent queues flooding. |
Hence my comment about monitoring. If a queue's supposed to be cleared out by a business process on a regular schedule and that doesn't happen you'll have problems long before the queue fills up
"WMQ's not processing my messages"
| Gralgrathor wrote: |
| Having a large queue fill up completely can cause serious performance drawbacks. |
Really? Cite your sources, where the queue's being used properly and not as a mock database with a depth of thousands and applications browsing by msg id. I've not seen a situation where a queue degrades by size & am interested to know where this happens (as additional evidence against people who want to store stuff in them indefinately).
| Gralgrathor wrote: |
| Keeping queues small forces one to design processes so that there is little risk of large amounts of data becoming lodged on the broker. |
Pedantically nothing gets lodged in broker, it gets lodged in the queue manager. My comments about monitoring apply, and I'd sooner have a broker design that reflects best business need than technical ones.
But that's just me. Not you. And you know your situation best. Site conditions and all that. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 12, 2010 7:58 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Vitor wrote: |
| Really? Cite your sources |
i should think it is a given that anything that consumes resources is a drain on those resources. however, i'll grant that a large queue in itself may not be an immediate problem. but in an environment where over a hundred flows use many more queues, this cannot be a solution to every problem involving capacity.
there are other risks to having large queues, and i'll remember what they are next time i cause our production environment to hang  |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 12, 2010 8:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Gralgrathor wrote: |
| Vitor wrote: |
| Really? Cite your sources |
i should think it is a given that anything that consumes resources is a drain on those resources. |
Ah - by "drain on resources" I took you to mean that doing a get on a large queue consumes more CPU than a get on a small queue. I tend not to think of disc as "resource" in that sense; typicially I encounter out of memory errors or poor response times due to "resource" shortage before I hit disc limits!
I conceed that a queue with a large depth does consume disc resource.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
