ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » Large message segmentation

Post new topic  Reply to topic Goto page 1, 2  Next
 Large message segmentation « View previous topic :: View next topic » 
Author Message
sl12345678
PostPosted: Tue Dec 28, 2021 1:50 pm    Post subject: Large message segmentation Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

Hi, I am trying to test sending large messages over 100MB over MQ using message segmentation and keep receiving MQRC_DATA_LENGTH_ERROR.

Message size= 124059223
QMGR max = 104857600
Queue max = 104857600


Here's a snippet of the code:

message.MessageFlags += MQC.MQMF_SEGMENTATION_ALLOWED;
message.GroupId = MQC.MQGI_NONE;
message.Version = MQC.MQMD_VERSION_2;

var putMessageOptions = new MQPutMessageOptions();
putMessageOptions.Options = MQC.MQPMO_SET_IDENTITY_CONTEXT | MQC.MQPMO_LOGICAL_ORDER | MQC.MQPMO_SYNCPOINT;

queue.Put(message, putMessageOptions);

Are there other properties that need to be set in order to segment this message into smaller messages/a message group?

Any insight would be appreciated - thank you!
Back to top
View user's profile Send private message
hughson
PostPosted: Tue Dec 28, 2021 8:07 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

Are you using a client connection? Have you set the MAXMSGL of the client and svrconn channels as well? What platform is your queue manager on and what version of MQ is it?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
bruce2359
PostPosted: Wed Dec 29, 2021 7:33 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Please read about segmentation and message groups here
https://www.ibm.com/docs/en/ibm-mq/7.5?topic=messages-message-groups
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Dec 29, 2021 7:49 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

The channels are set to the max length as well. I am using MQ 9.2 and .Net Full Framework 4.8.

I am trying to determine if the segmentation, or breaking the message into a smaller message/message group is done automatically by the program by turning on the segmentation flag in order to handle messages over 100MB.

I am able to put a 104MB message into the queue successfully. I am trying to use Segmentation and reassembly by queue manager as discussed here:
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=segmentation-reassembly-by-queue-manager
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Dec 29, 2021 10:38 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The maximum physical message size MQPUT is 100MB. If your payload is bigger than 100MB, then your app must create (MQPUT) segments < 100MB. Reassembly of a segmented message is possible by qmgr, channel or your consuming app MQGET.

Search google for C samples of message groups and segmentation.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Dec 29, 2021 10:43 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

Ok, so the segmentation is not done automatically for messages larger than 100MB - I need to programmatically break them into smaller messages and then the QueueManager will assemble them with the MQGET?
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Dec 29, 2021 6:35 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

sl12345678 wrote:
Ok, so the segmentation is not done automatically for messages larger than 100MB - I need to programmatically break them into smaller messages and then the Queue Manager will assemble them with the MQGET?

If you have to segment the message in your code, you will have to reassemble it the same way.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
sl12345678
PostPosted: Thu Dec 30, 2021 7:27 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

fjp_saper wrote:

If you have to segment the message in your code, you will have to reassemble it the same way.


Yes, I am trying to avoid manually segmenting the message, but I do not see another way to do so for messages over 100MB?

bruce2359 wrote:
The maximum physical message size MQPUT is 100MB. If your payload is bigger than 100MB, then your app must create (MQPUT) segments < 100MB. Reassembly of a segmented message is possible by qmgr, channel or your consuming app MQGET.

Search google for C samples of message groups and segmentation.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Dec 30, 2021 6:13 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

There seems to be some confusing, and inaccurate information in this thread.

I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.

This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.

There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.

I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
ashanks6
PostPosted: Mon Jan 03, 2022 3:24 pm    Post subject: Reply with quote

Newbie

Joined: 03 Jan 2022
Posts: 5

hughson wrote:
There seems to be some confusing, and inaccurate information in this thread.

I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.

This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.

There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.

I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.

Cheers,
Morag



Hello Ms Hughson. I am an associate of the OP. I tried to replicate your results in 'C', but I seem to have failed. Any messlen setting above 104857600 returns an mqrc of 2010. Could you share what you set the messlen to?

Thank you.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 8:28 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

hughson wrote:
There seems to be some confusing, and inaccurate information in this thread.

I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.

This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.

There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.

I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.

Cheers,
Morag



Thank you for your reply. Can you please post the C code you used as well as the setup for the program so that we can compare it to ours?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jan 05, 2022 8:37 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Rather than posting the snippet you offered, Please post your C code here, so we can take a closer look at it.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 8:41 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

Here is the Put code:

// .Net Framework 4.8
// MQ Client 9.2.4.0



private void PutMessageInMQ(string requestMessage, string queueName, byte[] correlationId, string courtId)
{

MQMessage message;
_properties.Clear();

_properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
_properties.Add(MQC.HOST_NAME_PROPERTY, ConfigurationManager.AppSettings["PutHost"]);
_properties.Add(MQC.CHANNEL_PROPERTY, ConfigurationManager.AppSettings["PutChannel"]);
_properties.Add(MQC.PORT_PROPERTY, ConfigurationManager.AppSettings["PutPort"]);

using (var queueManager = new MQQueueManager(ConfigurationManager.AppSettings["PutQueueManager"], _properties))
{
try
{
using (var queue = queueManager.AccessQueue(ConfigurationManager.AppSettings["PutMQ"],
MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING | MQC.MQOO_SET_IDENTITY_CONTEXT | MQC.MQOO_INQUIRE))
{
int qmmml = queueManager.MaximumMessageLength; //104857600
int qmax = queue.MaximumMessageLength; //104857600

message = new MQMessage();

message.PutApplicationType = MQC.MQAT_WINDOWS;
message.Format = MQC.MQFMT_STRING;
message.MessageId = MQC.MQMI_NONE;
message.CorrelationId = MQC.MQCI_NONE;
message.Encoding = MQC.MQENC_NATIVE;
message.Persistence = MQC.MQPER_PERSISTENT;
message.MessageType = MQC.MQMT_DATAGRAM;

message.MessageFlags += MQC.MQMF_SEGMENTATION_ALLOWED;
message.GroupId = MQC.MQGI_NONE;
message.Version = MQC.MQMD_VERSION_2;

message.ApplicationIdData = string.Format("C{0}{1}{2}", courtId, ConfigurationManager.AppSettings["QInterfaceNumber"], ConfigurationManager.AppSettings["QOriginatorCode"]);
message.CorrelationId = correlationId;

message.Write(ASCIIEncoding.ASCII.GetBytes(requestMessage));

var putMessageOptions = new MQPutMessageOptions();
putMessageOptions.Options = MQC.MQPMO_SET_IDENTITY_CONTEXT | MQC.MQPMO_LOGICAL_ORDER | MQC.MQPMO_SYNCPOINT;

int MessageSize = 0;
MessageSize = message.MessageLength; //124059223

queue.Put(message, putMessageOptions); //Messages put on the queue and sent


message.ClearMessage();
queue.Close();

}
}
catch (Exception ex)
{
_logger.Error(ex);
}


finally
{
queueManager.Close();
queueManager.Disconnect();
}

}
}
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jan 05, 2022 9:14 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.

I don’t have a dev environment available to duplicate Morag’s testing.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sl12345678
PostPosted: Wed Jan 05, 2022 9:18 am    Post subject: Reply with quote

Novice

Joined: 28 Dec 2021
Posts: 10

bruce2359 wrote:
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.

I don’t have a dev environment available to duplicate Morag’s testing.


Yes, I am specifically trying to segment messages larger than 100MB.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » IBM MQ API Support » Large message segmentation
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.