| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Here we go again - ! on the mainframe |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Thu Aug 22, 2013 6:09 am Post subject: Here we go again - ! on the mainframe |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
The exclamation point is one of those well known characters that does not get converted correctly in some cases when going from ASCII to EBCIDIC
Sending system is Windows, puts a message with CCSID 437, the ! character is represented in hex in this message as 21.
Receiving system is z/OS, QM CCSID on z/OS is set to 500, and the mainframe app sees not an ! but a |.
Why? Why did hexadecimal 21 get converted to a vertical bar? The veritical bar in CCSID 500 is hexadecimal BB, not 21.
I've been Googling trying to find what CCSID 500 thinks hexadecimal 21 is, but to no avail. None of the charts for CCSID 500 go that low. So why does hexadecimal 21 labeled as MQMD.CCSID 437 get seen as a vertical bar after conversion and not some other random character?
Adding to the confusion: When I do the reverse test, using our QPASA Agent on the mainframe to put a message directly to a local queue on the mainframe, the message sits with an MQMD CCSID of 500, and the ! is represented by hexadecimal 4F. If I then browse this message, MQExplorer, QPASA, and rfhutilc all display that 4F as an !, but MO71 displays that 4F as |. Why is MO71 showing a vertical bar for hexadecimal 4F labeled as MQMD.CCSID 500, but those other tools see 4F labeled as MQMD.CCSID 500 as the exclamation point?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| fatherjack |
| Posted: Thu Aug 22, 2013 6:53 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| PeterPotkay wrote: |
| The veritical bar in CCSID 500 is hexadecimal BB, not 21. |
You seem to be saying the ! on your Windows machine is x'21' and when it gets to your mainframe app the hex value of your character is still 21? In which case it suggests no conversion has happened. If it has, what is the hex value of the character on the mainframe? And what's doing the conversion? And what are you using to look at the message on the mainframe?
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Aug 22, 2013 8:49 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
We got a discussion on the MQ list server on this very topic.
Here's what I think is happening.
Sending app puts a !, represented as x21, labeled as MQMD.CCSID 437. 100% correct.
Mainframe QM running as CCSID 500 gets an MQGET with convert call, and the mainframe QM converts the x21 to a x4F, knowing that x4F does represent the explanation point in CCSID 500. Still 100% correct.
The QM gives the message to the app with a x4F. But the app on the mainframe is actually executing in another CCSID, perhaps 037. So that app ingests that x4F character and ‘sees’ it as the vertical bar, because in its code page x4F is a vertical bar.
An easy answer would be for the consuming app on the mainframe to specify MQMD CCSID of 037 on its MQGET with convert. Then the QM would know to convert from 437 to 037 and that x21 would change to x5A and the app would see it as !.
But the consuming app in this case is the IMS OTMA Bridge, and there is no way to tell it to selectivly choose 037 as the target CCSID for conversions. Its forced to use the QM's CCSID if 500. And yes, it can be argued that the QM's CCSID is wrong at 500 on this sysplex and it should be changed to 037 and all will be well. But as explained before, that is a big risky change on a system hosting dozends of apps running for 15 years plus.
We could hack it, but ugly.
A. Figure out what in CCSID 437 gets converted to x5A in 500, and place that instead of an !. 
-or-
B. Use WMB or DP to convert the 437 message to 037, leaving the ! represented correctly as x5A. And then mislabel the message as CCSID 500 to trick the OTMA bridge to thinking no conversion needed, leaving that x5A alone.
-or-
C. Stop sending the ! and avoid the problem. 
_________________
Peter Potkay
Keep Calm and MQ On
Last edited by PeterPotkay on Fri Aug 23, 2013 3:06 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 22, 2013 8:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| You forgot "Stop sending in 437 and switch to something like UTF-8 that may convert differently". |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Aug 22, 2013 9:09 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
I don't think that would change the situation, Jeff. The conversion is technically happening correctly from 437 to 500, and would continue I assume to be correctly converted from UTF-8 to 500. In both cases the ! ends up represented as x4F (no issue there), and the problem only really happens when you hand an app running as 037 a message converted to and described as 500.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Aug 23, 2013 1:01 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| PeterPotkay wrote: |
| I've been Googling trying to find what CCSID 500 thinks hexadecimal 21 is, but to no avail. None of the charts for CCSID 500 go that low. |
Code page charts generally don't include control characters, (which for EBCDIC, is hex 00-3F). The hex 21 ('!') is ASCII here, so it's not relevant what EBCDIC might think it is. But for the curious, it would represent the EBCDIC control character <SOS.> ("Start of Significance").
| PeterPotkay wrote: |
Here's what I think is happening.
Sending app puts a !, represented as x21, labeled as MQMD.CCSID 500. 100% correct. |
Not quite. Sending app puts a !, represented as x21, labeled as MQMD.CCSID 437.
| PeterPotkay wrote: |
| Mainframe QM running as CCSID 500 gets an MQGET with convert call ... But the app on the mainframe is actually executing in another CCSID, perhaps 037. |
Yes, that's the most likely explanation. The message is valid, but the recieving app misinterprets it.
| PeterPotkay wrote: |
| We could hack it, but ugly. |
How about converting the message to ccsid 037 beforehand, (using WMB, DP, etc.), but setting the Format to MQFMT_NONE? That should prevent any further conversion attempts downstream. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Aug 23, 2013 2:25 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| rekarm01 wrote: |
| PeterPotkay wrote: |
| Mainframe QM running as CCSID 500 gets an MQGET with convert call ... But the app on the mainframe is actually executing in another CCSID, perhaps 037. |
Yes, that's the most likely explanation. The message is valid, but the recieving app misinterprets it. |
Yes I agree. It looks like maybe the data is getting stored in DB2? I think 037 is the default CCSID for DB2 on the mainframe.
But this problem is not restricted to mainframes. If you send data in one CCSID to somewhere that is using a different CCSID to read it you'll get problems.
| rekarm01 wrote: |
| How about converting the message to ccsid 037 beforehand, (using WMB, DP, etc.), but setting the Format to MQFMT_NONE? That should prevent any further conversion attempts downstream. |
Until someone uses something else using a different CCSID to look at the data! Thats why I like this:
| PeterPotkay wrote: |
| C. Stop sending the ! and avoid the problem. |
Why do you need to send ! or any special character for that matter. A-Z, a-z and 0-9 always seem to work fine no matter what CCSID.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 23, 2013 3:05 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| rekarm01 wrote: |
| PeterPotkay wrote: |
Here's what I think is happening.
Sending app puts a !, represented as x21, labeled as MQMD.CCSID 500. 100% correct. |
Not quite. Sending app puts a !, represented as x21, labeled as MQMD.CCSID 437.
|
I corrected my original post, thanks for catching that.
| rekarm01 wrote: |
| PeterPotkay wrote: |
| We could hack it, but ugly. |
How about converting the message to ccsid 037 beforehand, (using WMB, DP, etc.), but setting the Format to MQFMT_NONE? That should prevent any further conversion attempts downstream. |
Interesting....I wonder if we can do this with the MQIIH header and the OTMA bridge in play. I think so, just leave the MQIIH.Format field in its initial value of MQFMT_NONE. Hopefully there are no other negative consequences. If they insist on sending the !, this may be the best option, although it will involve a detour thru WMB or DataPower to make that conversion (437 --> 037) happen prior to be sent down the channel to the mainframe QM.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 23, 2013 3:07 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| fatherjack wrote: |
| PeterPotkay wrote: |
| ]C. Stop sending the ! and avoid the problem. |
Why do you need to send ! or any special character for that matter. A-Z, a-z and 0-9 always seem to work fine no matter what CCSID. |
This will be my proposed best choice all things considered. Hopefully the business case will allow it.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 23, 2013 6:58 am Post subject: Re: Here we go again - ! on the mainframe |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fatherjack wrote: |
| PeterPotkay wrote: |
| C. Stop sending the ! and avoid the problem. |
Why do you need to send ! or any special character for that matter. A-Z, a-z and 0-9 always seem to work fine no matter what CCSID. |
Because you have two variable length string fields next to each other, and you need to indicate when one has stopped and the next has begun?
Granted, a comma is as good as ! for that, but. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 23, 2013 7:59 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Because Kia decided that an exclamation point should be used to define a a trim level for their Kia Soul model.
http://www.kia.com/us/#/soul/build/?cid=sem&ppc=y
Bet somebody in their marketing dept was feeling real clever when they came up with that one.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Aug 25, 2013 12:38 pm Post subject: Re: Here we go again - ! on the mainframe |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| fatherjack wrote: |
| But this problem is not restricted to mainframes. |
If this problem occurs with ASCII-based ccsids, it's typically limited to just non-ASCII characters, so it wouldn't affect applications that only use the ASCII character subset. For EBCDIC-based ccsids however, this problem is not limited to just the non-ASCII characters.
Non-mainframe MQ admins may also tend to configure their queue managers with a locale-specific QMgr.ccsid, more suitable for applications that neglect to specify their own MQMD.ccsid, while mainframe MQ admins may tend to configure their queue managers with the default multi-lingual QMgr.ccsid=500, less suitable for applications that neglect to specify their own MQMD.ccsid. If that's the case, then mainframe applications would be more likely to encounter this problem.
| fatherjack wrote: |
| Why do you need to send ! or any special character for that matter. A-Z, a-z and 0-9 always seem to work fine no matter what CCSID. |
That's fine for telegrams, but some applications may require a broader character set.
| PeterPotkay wrote: |
| I think so, just leave the MQIIH.Format field in its initial value of MQFMT_NONE. Hopefully there are no other negative consequences. If they insist on sending the !, this may be the best option, although it will involve a detour thru WMB or DataPower to make that conversion (437 --> 037) happen prior to be sent down the channel to the mainframe QM. |
MQGET would return an MQCC_WARNING reason code MQRC_FORMAT_ERROR, but would successfully get the uncoverted message. Whether this works for the OTMA bridge remains to be seen. Is it possible for the sending application to perform the conversion directly, without involving an extra WMB or DataPower component? |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
