| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| DataPower to IMS via the OTMA Bridge - MQMD Encoding issue |
View previous topic :: View next topic |
| Author |
Message
|
| PeterPotkay |
 Posted: Wed Jun 19, 2013 8:14 am Post subject: |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Message A
Put by DataPower (MQ Client 7.0.1.1) to a Red Hat Linux QM (MQ 7.0.1.6) without specifying the MQMD Encoding on the MQPUT. The message ends up with an Encoding of 546, the CCSID is 819. The app data payload is built with a WTX map, the MQIIH Header is produced by the style sheet described here:
http://publib.boulder.ibm.com/infocenter/wsdatap/v4r0m2/topic/com.ibm.dp.xb.doc/integratingwithmq39.htm?path=5_3_1_3_2_1#mqint_imsinfoheader_concept
The first line of the MQIIH is as follows. This is a copy and paste from BMTM looking at the data with the Text and Hex columns turned on.
| Code: |
| I I H T 49494820 01000000 54000000 00000000 |
The first line of the message payload, after the MQIIH, is as follows:
| Code: |
| A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E |
Message B
Put by DataPower (MQ Client 7.0.1.1) to a Red Hat Linux QM (MQ 7.0.1.6) specifying the MQMD Encoding as 273 on the MQPUT. The message ends up with an Encoding of 273, the CCSID is 819.
The first line of the MQIIH is as follows, first the text and then in hex. Notice the difference in endianess of the 2nd and 3rd block of data when compared to Message A.
| Code: |
| I I H T 49494820 00000001 00000054 00000000 |
The first line of the message payload, after the MQIIH, is as follows. Notice the bytes are exactly the same when compared to Message A, but the MQMD has a different Encoding describing these exact bytes. Uh oh, you can see there is going to be a conversion issue.
| Code: |
| A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E |
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.csqzak.doc/fr12840_.htm
This says the Encoding value of the MQIIH is taken from the preceding header, in this case the MQMD.
It seems the MQMD Encoding value on the MQPUT is affecting the way the bytes are produced in the MQIIH Header, but it is not having an effect on the bytes in the message payload?
Repeating the test but instead having both Message A and B go from the Linux QM to the mainframe QM over a SNDR channel with convert set to NO has the messages looking exactly the same as when they were at rest in a local queue on that Linux QM.
I change the SNDR channel on that Linux QM to have convert set to Yes and repeat the test.
Message A now has an Encoding of 785 and a CCSID of 500. The first line of the MQIIH is as follows:
| Code: |
| I I H è C9C9C840 00000001 00000054 00000000 |
The first line of the message payload, after the MQIIH, is as follows:
| Code: |
| A 3 D H 8 7 0 1 C O N 0D400000 C1F3C4C8 F8F7F0F1 40C3D6D5 |
Message B now also has an Encoding of 785 and a CCSID of 500. The first line of the MQIIH is as follows:
| Code: |
| I I H è C9C9C840 00000001 00000054 00000000 |
The first line of the message payload, after the MQIIH, is as follows:
| Code: |
| ‡ A 3 D H 8 7 0 1 C O N 00170000 C1F3C4C8 F8F7F0F1 40C3D6D5 |
After conversion the bytes now differ between Message A and Message B in the data payload, but the MQIIH looks the same. So the MQIIH header is getting described properly by the MQMD Encoding and thus being converted properly. The message payload beyond the MQIIH is potentially not being described properly in one of these cases.
OK, lets see how the IMS application deals with this. Datapower connects in client mode to that Linux Queue Manager putting to a remote q that aims at the mainframe over a SNDR channel with Convert set to NO.
Message A causes an Abend and 3 MQ replies come back from the OTMA bridge. Bunch of mainframe gibberish, put the phrase “NO SEGMENT RECEIVED AFTER TCODE SEGMENT” is in one of the 3 reply messages.
Message B causes a successful reply message to come back.
I set the SND channel to convert, restart the channel and repeat the test. No change – Message A causes an Abend with 3 replies and Message B works correctly.
So, I’m open to opinions, but here’s where I’m at. If the bytes in the payload of the app data look exactly the same coming out of Datapower regardless of the Encoding, but after translation they look different when compared to each other, that says to me one of the Encodings is not describing the data correctly on output. Based on testing it would seem that the 546 Encoding does not correctly describe the data that DataPower outputs. At this link:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.csqzaq.doc/fc_MQENC_.htm
It describes the systems for the various encodings. It shows 546 for Linux and Windows, and 273 for Unix. My testing seems to indicate that 273 is correct for DataPower and 546 is incorrect. If true, is this a problem in the DataPower layer, or perhaps in the MQ Client version (7.1.0.1) loaded into this firmware version of DataPower? Maybe its picking up a Linux Encoding value, maybe from the Linux QM its connected to? Picking up the queue manager’s encoding is contrary to how MQ Clients should set an MQMD Encoding when asking for the default. They should get the encoding of THEIR platform (not the QM’s) because THEY are producing the bytes in the message. Or maybe the WTX map is getting in the way and producing the data in a 273 encoding when it should be a 546 encoding?
As a test we connected Datapower directly to a z/OS Queue manager that has an Encoding of 785. DataPower produced a message without setting the MQMD Encoding and the message landed in the queue with an MQMD encoding of 546 again. It did not (incorrectly) pick up the 785 from the z/OS Queue Manager. Putting the message while connected to the z/OS QM and specifying the 273 encoding also produced the same message and MQIIH as Message B above.
I guess this means that the MQ Client 7.0.1.1 on a DataPower appliance is populating the MQMD.Encoding with 546 if allowed to default. The encoding of the MQIIH header is produced to match this 546 encoding, but the message payload produced needs an encoding of 273. Shouldn’t the encoding in the MQMD default to match the encoding of the data being produced by the platform? If yes, is this a problem worthy of a PMR? If yes, do you think it needs to go to IBM MQ or IBM DataPower or IBM WTX? I want to make sure that by setting the MQMD.Encoding to 273 on the put in DataPower is a legit fix and not some hack work around that will byte us in the future.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Jun 19, 2013 8:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The encoding and CCSID of any given piece of an MQ message describes the contents of the NEXT piece of the message.
So the MQMD describes the contents of the MQIIH, and the MQIIH describes the contents of the message body.
So your MQIIH needs to indicate that the message data is in the relevant CCSID and encoding.
This is regardless of channel conversion or not. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jun 19, 2013 8:35 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9399
Location: US: west coast, almost. Otherwise, enroute.
|
| PeterPotkay wrote: |
| ...Bunch of mainframe gibberish, put the phrase “NO SEGMENT RECEIVED AFTER TCODE SEGMENT” is in one of the 3 reply messages. |
What gibberish?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 19, 2013 8:42 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| mqjeff wrote: |
The encoding and CCSID of any given piece of an MQ message describes the contents of the NEXT piece of the message.
So the MQMD describes the contents of the MQIIH, and the MQIIH describes the contents of the message body.
So your MQIIH needs to indicate that the message data is in the relevant CCSID and encoding.
This is regardless of channel conversion or not. |
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzak.doc%2Ffr12840_.htm
| Quote: |
Encoding (MQLONG)
This is a reserved field; its value is not significant. The initial value of this field is 0.
The Encoding for supported structures which follow a MQIIH structure is the same as that of the MQIIH structure itself and taken from any preceding MQ header.
|
The same thing is said about the CCSID field in the MQIIH header.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 19, 2013 8:46 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| bruce2359 wrote: |
| PeterPotkay wrote: |
| ...Bunch of mainframe gibberish, put the phrase “NO SEGMENT RECEIVED AFTER TCODE SEGMENT” is in one of the 3 reply messages. |
What gibberish? |
Gibberish was the wrong word. Its readable, its just mainframe specific stuff. A bunch of data specific to our Abend routines. Calling this routine and this paragraph in the abend routine via this abend module, things like that.
When the MQIIH header had issues the OTMA bridge would reject the message. Now the MQIIH is being intepretted correctly regardless of the original encoding, so we are at the app data portion of the message causing issues when its encoding is not an accurate match with what's in the MQMD.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 19, 2013 9:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| When the MQIIH header had issues the OTMA bridge would reject the message. Now the MQIIH is being intepretted correctly regardless of the original encoding, so we are at the app data portion of the message causing issues when its encoding is not an accurate match with what's in the MQMD. |
Okay, but again that's consistent with what you said,
| Quote: |
| The Encoding for supported structures which follow a MQIIH structure is the same as that of the MQIIH structure itself and taken from any preceding MQ header. |
To whit, the MQIIH Encoding and CCSID are ignored. The segment following the MQIIH is expected to be in the CCSID and Encoding of the segment that precedes the MQIIH. In your case, and presumably always, this is the MQMD. (I doubt the IMS Bridge would appreciate receiving an MQRFH2).
So DP/WTX needs to produce the message body in the same Encoding/CCSID as it produces the MQIIH, and these need to be set properly in the MQMD. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 19, 2013 9:08 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| mqjeff wrote: |
| So DP/WTX needs to produce the message body in the same Encoding/CCSID as it produces the MQIIH, and these need to be set properly in the MQMD. |
Does Encoding work the same as CCSID for MQ Clients in that the MQ Client has the option to NOT specify these 2 parameters on the put with confidence that the MQ Client libraries will default to values appropriate for the platform where the MQ Client is executing?
Or is Encoding a unique animal compared to CCSID and an MQ Client cannot let the MQMD Encoding default?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 19, 2013 9:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Encoding is the same as CCSID, in that both need to be set by the client to values that properly reflect the contents of the message segment that follows.
If the app is populating the segment with data in the encoding that is not the default encoding, then the app needs to set the Encoding to the correct value.
If the app is populating the segment with data in the default encoding, then the app can allow the Encoding to be set by the MQ libraries.
In the case of the MQMD, this can be set specifically or taking from the MQMD Default values.
Notice I don't comment on what DP or WTX can or cannot do. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 19, 2013 9:32 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
And I think this has brought us to the question: When DataPower produces data, is it correctly described by 546 or 273, or, what is the proper encoding for the DataPower platform?
If its 273, then the default value is wrong (MQ Client bug?)
If the default value is 546, then why does it appear to need 237 as a descriptor? I just asked my DataPower guy "Does that WTX map you are using have an encoding value, perhaps 273???"
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 19, 2013 9:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| And I think this has brought us to the question: When DataPower produces data, is it correctly described by 546 or 273, or, what is the proper encoding for the DataPower platform? |
Is the byte value you see for, say, the MQIIH.Version, a big-endian integer that has the value '1', or is it a little-endian integer that has the value 1? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 19, 2013 10:35 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| mqjeff wrote: |
| PeterPotkay wrote: |
| And I think this has brought us to the question: When DataPower produces data, is it correctly described by 546 or 273, or, what is the proper encoding for the DataPower platform? |
Is the byte value you see for, say, the MQIIH.Version, a big-endian integer that has the value '1', or is it a little-endian integer that has the value 1? |
Either one. If we set the MQMD.Encoding to 546, or let it default to 546, the MQIIH bytes are little endian and thus properly described. If we set the MQMD.Encoding to 273 the MQIIH bytes are big endian and again thus properly described.
But in either case the data that follows the MQIIH is big endian and only properly described if the MQMD.Encoding is set to 273 with the MQIIH ending up as 273 to match.
Hmm, I wonder if this WTX map isn't that issue. Maybe its producing the data as 273 because the WTX map knows its building something destined for the mainframe, but the MQ Client on DataPower has no way to know the data portion of the message was produced in a non native encoding so when asked to default it does default to the proper 546 - proper for native DataPower, but not proper for the custom data? This sure sounds plausible.
Waiting for our WTX / DP guy to confirm what they are coding for encoding at the WTX layer.
I would still like confirmation of what the native encoding is for DataPower - is it 546 or 273. Might be PMR time for that bit of info.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 19, 2013 9:04 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
I'd expect it to be like for Linux for that processor type... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Thu Jun 20, 2013 3:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
Is that exact stylesheet producing the MQIIH header, or is it a similar stylesheet, that sets the MQIIH Format and other elements to different values?
| PeterPotkay wrote: |
| This is a copy and paste from BMTM looking at the data with the Text and Hex columns turned on. |
The spacing seems a bit inconsistent between messages A and B below. Is that due to BMTM, or due to copy/pasting?
| PeterPotkay wrote: |
Message A [Linux QM, ccsid=819, encoding=546]
| Code: |
I I H T 49494820 01000000 54000000 00000000
A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E |
Message B [Linux QM, ccsid=819, encoding=273]
| Code: |
I I H T 49494820 00000001 00000054 00000000
A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E |
Notice the difference in endianess of the 2nd and 3rd block of data [in MQIIH] when compared to Message A. ... Notice the bytes are exactly the same [in message payload] when compared to Message A, but the MQMD has a different Encoding describing these exact bytes. Uh oh, you can see there is going to be a conversion issue. ... It seems the MQMD Encoding value on the MQPUT is affecting the way the bytes are produced in the MQIIH Header, but it is not having an effect on the bytes in the message payload? |
When writing the message, the DataPower rule converts the MQ headers from text to binary, and can apply the given MQMD ccsid/encoding to the MQIIH header elements. But it can't do the same for the message payload, because the WTX Map has already converted the payload to bytes. The DataPower rule is responsible for making sure that the MQ headers correctly describe the payload, and it should not rely on default values for ccsid/encoding, unless the WTX map does.
| PeterPotkay wrote: |
Message A [mainframe QM, ccsid=500, encoding=785]
| Code: |
I I H è C9C9C840 00000001 00000054 00000000
A 3 D H 8 7 0 1 C O N 0D400000 C1F3C4C8 F8F7F0F1 40C3D6D5 |
Message B [mainframe QM, ccsid=500, encoding=785]
| Code: |
I I H è C9C9C840 00000001 00000054 00000000
‡ A 3 D H 8 7 0 1 C O N 00170000 C1F3C4C8 F8F7F0F1 40C3D6D5 |
The message payload beyond the MQIIH is potentially not being described properly in one of these cases. |
The 'è' and '‡' should probably be 'è' and '' respectively; hopefully that's just an issue with the display ccsid, rather than an issue with the data itself. The X'0D40' at the start of the message payload is definitely off, and it looks like the problem is more than just the Encoding. What was the MQIIH.Format again?
| PeterPotkay wrote: |
| My testing seems to indicate that 273 is correct for DataPower and 546 is incorrect. |
DataPower generally deals with character data, and relies on extensions such as WTX to manage binary data, so if there is a "native encoding", it would be for the WTX extension, not for DataPower itself. And, unless the message payload contains floating point or packed decimal data, only the last 4 bits of the Encoding are relevant, so there's no real difference between 273 (X'111'), 785 (X'311') or even 1 (X'001').
| PeterPotkay wrote: |
| Or maybe the WTX map is getting in the way and producing the data in a 273 encoding when it should be a 546 encoding? |
Not so much getting in the way. It's the WTX map that decides what the ccsid and encoding should be; the MQ headers need to match that, not the other way around. Either find out what the WTX map uses by default, or else make the WTX map use the desired ccsid/encoding to begin with (e.g 500/785), and then construct the MQ headers to match. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jun 21, 2013 7:08 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Here is the end result for this problem and solution.
Problem Summary:
Messages sent from Datapower via MQ to an OTMA Bridge queue on the mainframe were causing the IMS transaction to abend.
Solution:
By populating the MQMD.Encoding field in the MQ message to 273 the abends stopped and a good reply came back from IMS. We figured this out last Friday, the question was why did this work. Was it a proper solution, or a hack to get around a bug and the hack solution might itself break in the future? We now know this is the right solution.
1. WTX Developer A was using a WTX Map to create the application specific message payload in the MQ message. He was referring to how another app first did it in their Java code years ago to understand how to lay the data out.
2. Datapower Developer B was creating the MQMD header and the MQIIH header, appending WTX Developer A’s app data and sending it to the MQ queue.
3. Developer B was not specifying the MQMD Encoding or the MQMD CCSID fields when creating the message. I consider this is normal. In general you want to allow these fields to default because by default the MQ Client libraries will correctly identify the platform they are executing on and will fill in the Encoding and CCSID appropriate for the client platform. In this case the client platform is Datapower and the default encoding value is 546 (little endian based on most likely x86 chips) and the default encoding is 819 (a Unix variant), so that’s what the default values ended up being.
4. The encoding value is meant to describe the non character data like integer bytes, and the CCSID describes the character data.
5. We now had an MQ message where the MQMD header had MQMD.Encoding 546 and MQMD.CCSID 819.
6. The MQIIH header that follows the MQMD was built to match whatever the MQMD said. So the MQIIH’s character data was in the 819 code page. The MQIIH also happens to have some integer byte fields, so the encoding value in the header is relevant. The bytes were created to match the MQMD.Encoding value of 546, so they were in little endian format. So far so, so good.
7. While the MQIIH has its own encoding and CCSID fields, they are not used for anything. This is documented in the MQ manuals. This is contrary to other types of headers like an RFH2 header, where the MQMD describes the format of the RFH2, and then the RFH2 fields describe what follows it, usually the app data. So the MQIIH and the app data that follows it both need to be in the same encoding and CCSID.
8. In the application data created by the WTX Map, there was character data. The WTX Map asked for “Native” data character set, so the character data was produced in a character set appropriate for the executing platform (DataPower) and this matched what the MQMD.CCSID field was defaulting to: 819. Still good.
9. In the application data created by the WTX Map, there were also some binary integer fields in the data. This is not common at all but these types of fields are required when sending to IMS via the OTMA bridge. The WTX Map asked for the Byte Order of these fields to be “Big Endian” (based on the other app example in step 1) and so the data was produced in a big endian format. We now a message payload where some of the fields are in big endian, but the MQMD.Encoding value is little endian (546). This is what causes the abends on the mainframe. Some of the data in the payload is not described correctly by the header in this scenario and the next app that tries to MQ convert this data (the RCVR MCA on the mainframe QM in this case) ends up with some of the fields completely garbled.
10. When Developer B hard coded the MQMD.Encoding field to 273 (big endian) two things occurred. The MQIIH header was created with its integer bytes in big endian, so it was still correctly described, but now the message payload that was already in big endian was properly described by the header as well. The conversion routine was now doing its worked based on good information and the converted message was good. IMS was able to process it and send back a reply.
The lesson here is the vast majority of the time the data produced by the MQ Client is in the CCSID and Encoding for the platform that the MQ Client is executing on, and letting the MQMD.CCSID and MQMD.Encoding default is just fine. But if your app has the capability to produce data in a different encoding or CCSID you need to identify that you are doing that and then set the MQMD fields so that the data is described correctly.
It looks like the WTX Map could set those integer bytes to the native byte order which would give you little endian. If that was done the default MQMD.Encoding of 546 would correctly describe the data and that would work.
A. The map sets the bytes to little endian and the MQMD.Encoding is allowed to default to 546, or can be explicitly set to 546 for clarity’s sake
B. The map set the bytes to big endian and the MQMD.Encoding must be set to 273. This is how things are now.
I am not aware of there being a strong reason to go with A or B. If writing this thing from the ground up I would probably vote for A. If you are writing the request on a little endian platform, and the receiver can do a standard convert, then produce the data in the encoding that matches where the producer is executing. But B will work fine too.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Wed Sep 17, 2014 11:28 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1231
Location: Gold Coast of Florida, USA
|
Peter,
I will commit the sin and bring up this old thread but that is because you referenced it in a new thread I posted. 
I get the part about using WTX in DP to build the message. I get your part about encoding (funny that encoding for XML means CCSID, but I digress).
I don't get the part about the message conversion. DP connects to MQ via client channels. Did you connect to a intermediate Qmgr with a sender channel that did the convert? Was there an Exit? How would convert on MQGET work when there is an MQIIH header? Say what? 
You seem to be intimating that something like WTX maps are needed to construct msgs to go to IMS OTMA for DP. I'm not too familar with DP, so I don't know how flexible it is about building message types other than XML.
-Joseph |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
