| Author |
Message
|
| solomon_13000 |
 Posted: Tue Jul 22, 2008 12:22 am Post subject: Compression |
 |
|
Master
Joined: 13 Jun 2008
Posts: 284
|
| I usually transfer large messages and the network is usually slow. So is it advisable to do compression before putting the messages on the network. Reason to improve performance. Therefore is compression a feasible technique?. I find that it is time consuming because it takes time to compress and decompress a message. |
|
| Back to top |
|
 |
| manicminer |
| Posted: Tue Jul 22, 2008 12:49 am Post subject: |
|
|
Disciple
Joined: 11 Jul 2007
Posts: 177
|
It depends on what your bottleneck is. If you sending and receiving machine have lots of spare CPU available, but your network is very slow or congested, then compression will probably help.
However if you have a very fast network, but your queue managers are CPU bound then compression will probably just slow things down.
Probably the only way you will know if it will help will be by analysing your system to find out what the source of the bottleneck is. |
|
| Back to top |
|
|
| sami.stormrage |
| Posted: Tue Jul 22, 2008 4:02 am Post subject: |
|
|
Disciple
Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore
|
how large?.. probably, refrence messages could help u.
_________________
*forgetting everything  * |
|
| Back to top |
|
|
| solomon_13000 |
| Posted: Tue Jul 22, 2008 4:30 am Post subject: |
|
|
Master
Joined: 13 Jun 2008
Posts: 284
|
|
| Back to top |
|
|
| AkankshA |
| Posted: Tue Jul 22, 2008 7:52 pm Post subject: |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
| solomon_13000 wrote: |
| above 0.5 mb. |
0.5 MB is not really that big a size for MQ i feel...
_________________
Cheers |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Tue Jul 22, 2008 8:01 pm Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
The bottom line is compression costs...regardless of if it is an MQ Message or a file. Your coice is really...'pick your poison', slow in the network, or slow in the processing.
My own choice would be to upgrade the network to meet the business need. Your milage may vary  |
|
| Back to top |
|
|
| skytrain |
| Posted: Fri Jul 25, 2008 2:10 am Post subject: |
|
|
Newbie
Joined: 25 Jul 2008
Posts: 4
|
hello,
i am trying to put 120 MB message over TCP on my Queue, the fact is when i am working local the message i can put will be segmented in 2 segments and it works formidable...the problem now is that i am using in my second test the SVRCONN channel so i increased my maximum message length on this channel to 104 MB this is the maximum and now i am getting the error message: MQRC_DATA_LENGTH_ERROR, do you think that i have to make a message compression on my MQ? i have tried some message compression attributes like(any, RLE, ZLIBFAST and ZLIBHIGH) but i am still getting the same error!!!????
any help??!!
thx |
|
| Back to top |
|
|
| AkankshA |
| Posted: Fri Jul 25, 2008 2:24 am Post subject: |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
|
| Back to top |
|
|
| skytrain |
| Posted: Fri Jul 25, 2008 2:42 am Post subject: |
|
|
Newbie
Joined: 25 Jul 2008
Posts: 4
|
Hello AkankshA,
thx for ur reply
regarding the first link u displayed, i cant use this solution coz i dont want to create a cluster this would mean new meetings and new planning for the MQ infrastucture that we have .. it woudnt sound good to say now well coz of couple MBs we need to create a cluster and the supported Plattform is Microsoft® Windows® XP Professional SP1 or later (32-bit versions only). and i am using AIX and Win2000...
the sescond link i am now reading it and trying to understand it
but now a quick question do u think using the message compression woudnt solve the problem? |
|
| Back to top |
|
|
| AkankshA |
| Posted: Fri Jul 25, 2008 2:58 am Post subject: |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
No doubt compression would work too....
Analyse both the approaches based on your requirement and go for the feasible one
_________________
Cheers |
|
| Back to top |
|
|
| skytrain |
| Posted: Fri Jul 25, 2008 3:20 am Post subject: |
|
|
Newbie
Joined: 25 Jul 2008
Posts: 4
|
hi again..and thx for answerng
here a little desciption of my test:
i developed a java programm that can put and get messages on and from the MQ from and to the filesystem...using the Server Channel the hashtable i use when i create an instance of my queue manager:
Hashtable table = new Hashtabe();
table.put(MQC.CHANNEL_PROPERTY, myChannelname);
table.put(MQC.HOST_NAME_PROPERTY, myHostname);
table.put(MQC.PORT_PROPERTY, Port);
//here now my compression
Collection msgComp = new Vector();
msgComp.add(new Integer(MQC.MQCOMPRESS_ZLIBHIGH));
MQEnvironement.hdrCompList = msgComp;
MQQueueManeger mq = new MQQueueManager(myqueuemang, table);
the compression by doing this is not working..i dont know why...
any idea?
thx |
|
| Back to top |
|
|
| David.Partridge |
| Posted: Fri Jul 25, 2008 6:00 am Post subject: |
|
|
Master
Joined: 28 Jun 2001
Posts: 249
|
While you may be able to put a 120MB message with automatic segmentation when connected directly to the QM, when client channels come into play, the maximum size message you could ever hope to put would be 100MB, and turning on compression won't solve anything for you.
Furthermore sending large messages across channels can cause serious problems requiring careful planning of channels and xmit queues to work round.
You should seriously consider the use of manual segmentation into lots of small chunks and manual reassembly, or using message grouping and manual reassembly.
If you are trying to transfer large files over MQ, there are loads of "off the shelf" solutions that will do this for you without pain, whereas trying to roll your own solution will cause you a lot of pain.
_________________
Cheers,
David C. Partridge |
|
| Back to top |
|
|
| skytrain |
| Posted: Fri Jul 25, 2008 6:33 am Post subject: |
|
|
Newbie
Joined: 25 Jul 2008
Posts: 4
|
Hello David, and thx for replying,
u r right about the direkt connection to the MQ it is working very well..but as u said i am getting the problem when using my connection channel SYSTEM.ADMIN.SVRCONN i thought maybe there is some function to allow this kind of transfer but it seems there is nothing like this...i find it a bit weird, how mq can let very big messages be segmeted when u connect directly and doesnt let big messages be transfered through the channel hmmm... i think i have to begin thinking to implement some new methods to use manuell segmentation or manuell message grouping
loads "off the shelf" i think the costs would be very high no?? or do u think i can find something else as open source???
thx alot... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 25, 2008 8:08 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
| skytrain wrote: |
Hello David, and thx for replying,
u r right about the direkt connection to the MQ it is working very well..but as u said i am getting the problem when using my connection channel SYSTEM.ADMIN.SVRCONN i thought maybe there is some function to allow this kind of transfer but it seems there is nothing like this...i find it a bit weird, how mq can let very big messages be segmeted when u connect directly and doesnt let big messages be transfered through the channel hmmm... i think i have to begin thinking to implement some new methods to use manuell segmentation or manuell message grouping
loads "off the shelf" i think the costs would be very high no?? or do u think i can find something else as open source???
thx alot... |
Remember that as by default any of your svrconn channels are limited to 4MB regardless of what you set.
If you want to break through that threshold you need to explicitely define as well a client channel and set the bigger size limitation on both svrconn and corresponding clntconn channel. Then use the channel table to connect.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Jul 27, 2008 11:39 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
| Why don't use you use the standard MQ channel compression? |
|
| Back to top |
|
|
|
|
