ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » Large message segmentation

Post new topic  Reply to topic Goto page Previous  1, 2
 Large message segmentation « View previous topic :: View next topic » 
Author Message
bruce2359
PostPosted: Wed Jan 05, 2022 10:32 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

sl12345678 wrote:
bruce2359 wrote:
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.

I don’t have a dev environment available to duplicate Morag’s testing.


Yes, I am specifically trying to segment messages larger than 100MB.

Break your >100MB app data into multiple <100MB segments, each with its own MQMD, then do individual MQPUTs for each segment.

The consuming app will need to do individual MQGETs to combine the segments into the original the app data.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
ashanks6
PostPosted: Wed Jan 05, 2022 11:01 am    Post subject: Reply with quote

Newbie

Joined: 03 Jan 2022
Posts: 5

bruce2359 wrote:
sl12345678 wrote:
bruce2359 wrote:
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.

I don’t have a dev environment available to duplicate Morag’s testing.


Yes, I am specifically trying to segment messages larger than 100MB.

Break your >100MB app data into multiple <100MB segments, each with its own MQMD, then do individual MQPUTs for each segment.

The consuming app will need to do individual MQGETs to combine the segments into the original the app data.


Hi Bruce,

Yes, that is a workaround we could use, but I'd rather have our folks use the functionality that MQ provides. I have opened a ticket with IBM support, and will post back the results of that submission. There is a lot of conflicting information on this board about this issue, and it is my hope to provide absolute clarity.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Jan 05, 2022 11:34 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

https://www.ibm.com/docs/en/ibm-mq/9.2?topic=segmentation-reassembly-by-queue-manager

Yes, you can put and get messages > 100MB using this method without resorting to using multiple MQPUTs and MQGETs.

https://www.ibm.com/docs/en/ibm-mq/9.2?topic=long-message-segmentation
Quote:
Use this information to learn about segmenting messages. This feature is not supported on IBM® MQ for z/OS® or by applications using IBM MQ classes for JMS.

_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 11:40 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jan 05, 2022 12:20 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

sl12345678 wrote:
Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error.

sl12345678 wrote:
MessageSize = message.MessageLength; //124059223

Message length must be <100MB - the maximum size of an MQ physical message.

Reduce message length, and try again.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
ashanks6
PostPosted: Wed Jan 05, 2022 12:27 pm    Post subject: Reply with quote

Newbie

Joined: 03 Jan 2022
Posts: 5

PeterPotkay wrote:
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=segmentation-reassembly-by-queue-manager

Yes, you can put and get messages > 100MB using this method without resorting to using multiple MQPUTs and MQGETs.

https://www.ibm.com/docs/en/ibm-mq/9.2?topic=long-message-segmentation
Quote:
Use this information to learn about segmenting messages. This feature is not supported on IBM® MQ for z/OS® or by applications using IBM MQ classes for JMS.


Yes! Exactly. Our queue manager is on AIX btw.
And the OP is using .net.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 2:34 pm    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

bruce2359 wrote:
sl12345678 wrote:
Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error.

sl12345678 wrote:
MessageSize = message.MessageLength; //124059223

Message length must be <100MB - the maximum size of an MQ physical message.

Reduce message length, and try again.


Thanks, Bruce - I am already able to send smaller messages successfully with the same code. It is my understanding that messages over 100mb are automatically segmented if MQMF_SEGMENTATION_ALLOWED flag is turned on.

We are trying to figure out what is missing that is causing the data length error instead of automatically segmenting the large message.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jan 05, 2022 2:44 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Are you using a client channel to retrieve the message and what is the max msg length on your client channel / svrconn channel defined as?
If the queue manager assembles the message, does that mean it has to fit the max size of the client/svrconn channel?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
hughson
PostPosted: Wed Jan 05, 2022 3:07 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

I did my original testing using the Q Utility so I can't easily post the source. So I have quickly created a small application that illustrates the same. I don't think it is necessary to post the whole source, as you can manage to do an MQCONN and an MQOPEN I'm sure, but here is the source of the MQPUT part of the simple C part.

Code:
  MQMD       MsgDesc   = {MQMD_DEFAULT};
  MQPMO      pmo       = {MQPMO_DEFAULT};
  MQLONG     MsgLength = 124059223;
:
:
 /********************************************************************/
  /* Put large message segmented                                      */
  /********************************************************************/
  if (Verbose) printf("Putting %d bytes message to Queue '%s'\n",
                      MsgLength, pQName);
  pBuffer = malloc(MsgLength);
  if (!pBuffer)
  {
    printf("Unable to allocate %d bytes\n", MsgLength);
    goto MOD_EXIT;
  }
  memset(pBuffer, '*', MsgLength);
  MsgDesc.Version  = MQMD_VERSION_2;
  MsgDesc.MsgFlags = MQMF_SEGMENTATION_ALLOWED;
  MQPUT(hConn,
        hObj,
        &MsgDesc,
        &pmo,
        MsgLength,
        pBuffer,
        &CompCode,
        &Reason);
  if (Reason)
  {
    printf("MQPUT to Queue '%s' failed with reason %d\n",
           pQName, Reason);
    goto MOD_EXIT;
  }


As I've said before, what you are seeing is not a restriction the queue manager puts on you, as demonstrated by this code, but might be a restriction the environment or language bindings you are making use of places on you.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software


Last edited by hughson on Wed Jan 05, 2022 3:33 pm; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website
hughson
PostPosted: Wed Jan 05, 2022 3:23 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

I suspect that the problem is around the client channels. Try running your application locally bound to see if that works in the interim.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
hughson
PostPosted: Wed Jan 05, 2022 3:32 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.

So, you can do this segmentation of >100MB messages when locally bound but not when client bound.

When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.

Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
ashanks6
PostPosted: Wed Jan 05, 2022 3:38 pm    Post subject: Reply with quote

Newbie

Joined: 03 Jan 2022
Posts: 5

hughson wrote:
I did my original testing using the Q Utility so I can't easily post the source. So I have quickly created a small application that illustrates the same. I don't think it is necessary to post the whole source, as you can manage to do an MQCONN and an MQOPEN I'm sure, but here is the source of the MQPUT part of the simple C part.

Code:
  MQMD       MsgDesc   = {MQMD_DEFAULT};
  MQPMO      pmo       = {MQPMO_DEFAULT};
  MQLONG     MsgLength = 124059223;
:
:
 /********************************************************************/
  /* Put large message segmented                                      */
  /********************************************************************/
  if (Verbose) printf("Putting %d bytes message to Queue '%s'\n",
                      MsgLength, pQName);
  pBuffer = malloc(MsgLength);
  if (!pBuffer)
  {
    printf("Unable to allocate %d bytes\n", MsgLength);
    goto MOD_EXIT;
  }
  memset(pBuffer, '*', MsgLength);
  MsgDesc.Version  = MQMD_VERSION_2;
  MsgDesc.MsgFlags = MQMF_SEGMENTATION_ALLOWED;
  MQPUT(hConn,
        hObj,
        &MsgDesc,
        &pmo,
        MsgLength,
        pBuffer,
        &CompCode,
        &Reason);
  if (Reason)
  {
    printf("MQPUT to Queue '%s' failed with reason %d\n",
           pQName, Reason);
    goto MOD_EXIT;
  }


As I've said before, what you are seeing is not a restriction the queue manager puts on you, as demonstrated by this code, but might be a restriction the environment or language bindings you are making use of places on you.

I will reiterate for everyone who has said otherwise. You **CAN** do an MQPUT with a buffer longer than 100MB if you use segmentation. Stop telling the OP to reduce his message length!

Cheers,
Morag



Interesting, our code is VERY similar (as you might expect). The only difference was in how we loaded our message (I used 'Z' instead of '*') and sizeof to load the messlen. I changed my code to match yours. I still fail with a 2010 return code. I'm afraid I'm now forced to conclude that the queue manager itself is configured differently than your queue manager.

We are able yo send 102457600 byte messages, so I can not for the life of me think of what could be different.
Back to top
View user's profile Send private message
ashanks6
PostPosted: Wed Jan 05, 2022 3:46 pm    Post subject: Reply with quote

Newbie

Joined: 03 Jan 2022
Posts: 5

hughson wrote:
The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.

So, you can do this segmentation of >100MB messages when locally bound but not when client bound.

When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.

Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.

Cheers,
Morag


Looks like we crossed streams (never cross streams).

This explains everything, and I thank you for leaning into this.
Seems the documentation for server auto segmentation should note this.
Perhaps some future version of the client will allow this.

In the meantime, it looks like we will have to split the message ourselves.

If you ever get to Phoenix, please send me a note and redeem a free drink on me
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 3:55 pm    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

hughson wrote:
The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.

So, you can do this segmentation of >100MB messages when locally bound but not when client bound.

When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.

Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.

Cheers,
Morag


Thank you for your responses! Is it possible to set the channel limit programmatically? I assumed it was being set on the server.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jan 05, 2022 4:42 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

sl12345678 wrote:
Is it possible to set the channel limit programmatically? I assumed it was being set on the server.

Channel attributes here https://www.ibm.com/docs/en/ibm-mq/8.0?topic=attributes-channel-channel-types
SVRCONN channels answer-up to inbound client requests.

Programmatically, MQSET can set/alter a limited set of queue attributes only. What do you want to set the limit to - keeping in mind that 100MB is the maximum?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2 Page 2 of 2

MQSeries.net Forum Index » IBM MQ API Support » Large message segmentation
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.