| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| DFDL Numbers Parsing Error |
« View previous topic :: View next topic » |
| Author |
Message
|
| abooddy |
 Posted: Fri Jun 14, 2019 12:05 pm Post subject: |
 |
|
Novice
Joined: 28 May 2019
Posts: 20
Location: Baghdad, Iraq
|
Hi,
This is the TraceNode's ${Root}:
| Code: |
( ['GENERICROOT' : 0x291ef443b40]
(0x01000000:Name):Properties = ( ['MQPROPERTYPARSER' : 0x291f5897e30]
(0x03000000:NameValue):MessageSet = NULL
(0x03000000:NameValue):MessageType = '{}:ISO8583_1987_Unpacked' (CHARACTER)
(0x03000000:NameValue):MessageFormat = NULL
(0x03000000:NameValue):Encoding = NULL
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Transactional = NULL
(0x03000000:NameValue):Persistence = NULL
(0x03000000:NameValue):CreationTime = NULL
(0x03000000:NameValue):ExpirationTime = NULL
(0x03000000:NameValue):Priority = NULL
(0x03000000:NameValue):ReplyIdentifier = NULL
(0x03000000:NameValue):ReplyProtocol = 'TCPIP' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = NULL
(0x03000000:NameValue):IdentitySourceType = NULL
(0x03000000:NameValue):IdentitySourceToken = NULL
(0x03000000:NameValue):IdentitySourcePassword = NULL
(0x03000000:NameValue):IdentitySourceIssuedBy = NULL
(0x03000000:NameValue):IdentityMappedType = NULL
(0x03000000:NameValue):IdentityMappedToken = NULL
(0x03000000:NameValue):IdentityMappedPassword = NULL
(0x03000000:NameValue):IdentityMappedIssuedBy = NULL
)
(0x01000000:Name):DFDL = ( ['dfdl' : 0x291f8790d60]
(0x01000000:Name)http://www.ibm.com/dfdl/ISO8583-1993:ISO8583_1993_Unpacked = (
(0x03000000:NameValue):MTI_Version = 1 (DECIMAL)
(0x03000000:NameValue):MTI_MessageClass = 1 (DECIMAL)
(0x03000000:NameValue):MTI_MessageFunction = '1' (CHARACTER)
(0x03000000:NameValue):MTI_MessageOrigin = 0 (DECIMAL)
(0x01000000:Name ):PrimaryBitmap = (
(0x03000000:NameValue):Bits001to004 = '15' (CHARACTER)
(0x03000000:NameValue):Bits005to008 = '8' (CHARACTER)
(0x03000000:NameValue):Bits009to012 = '2' (CHARACTER)
(0x03000000:NameValue):Bits013to016 = '0' (CHARACTER)
(0x03000000:NameValue):Bits017to020 = '0' (CHARACTER)
(0x03000000:NameValue):Bits021to024 = '0' (CHARACTER)
(0x03000000:NameValue):Bits025to028 = '0' (CHARACTER)
(0x03000000:NameValue):Bits029to032 = '0' (CHARACTER)
(0x03000000:NameValue):Bits033to036 = '0' (CHARACTER)
(0x03000000:NameValue):Bits037to040 = '14' (CHARACTER)
(0x03000000:NameValue):Bits041to044 = '8' (CHARACTER)
(0x03000000:NameValue):Bits045to048 = '1' (CHARACTER)
(0x03000000:NameValue):Bits049to052 = '4' (CHARACTER)
(0x03000000:NameValue):Bits053to056 = '4' (CHARACTER)
(0x03000000:NameValue):Bits057to060 = '0' (CHARACTER)
(0x03000000:NameValue):Bits061to064 = '0' (CHARACTER)
)
(0x01000000:Name ):SecondaryBitmap = (
(0x03000000:NameValue):Bits065to068 = '0' (CHARACTER)
(0x03000000:NameValue):Bits069to072 = '0' (CHARACTER)
(0x03000000:NameValue):Bits073to076 = '0' (CHARACTER)
(0x03000000:NameValue):Bits077to080 = '0' (CHARACTER)
(0x03000000:NameValue):Bits081to084 = '0' (CHARACTER)
(0x03000000:NameValue):Bits085to088 = '0' (CHARACTER)
(0x03000000:NameValue):Bits089to092 = '0' (CHARACTER)
(0x03000000:NameValue):Bits093to096 = '0' (CHARACTER)
(0x03000000:NameValue):Bits097to100 = '0' (CHARACTER)
(0x03000000:NameValue):Bits101to104 = '4' (CHARACTER)
(0x03000000:NameValue):Bits105to108 = '0' (CHARACTER)
(0x03000000:NameValue):Bits109to112 = '0' (CHARACTER)
(0x03000000:NameValue):Bits113to116 = '0' (CHARACTER)
(0x03000000:NameValue):Bits117to120 = '0' (CHARACTER)
(0x03000000:NameValue):Bits121to124 = '0' (CHARACTER)
(0x03000000:NameValue):Bits125to128 = '0' (CHARACTER)
)
(0x03000000:NameValue):PrimaryAccountNumber_002 = '5210972000000328' (CHARACTER)
(0x03000000:NameValue):ProcessingCode_003 = '182000' (CHARACTER)
(0x03000000:NameValue):AmountTransaction_004 = '000000020000' (CHARACTER)
(0x03000000:NameValue):AmountReconciliation_005 = '000000020200' (CHARACTER)
(0x03000000:NameValue):SystemsTraceAuditNumber_011 = '279135' (CHARACTER)
(0x03000000:NameValue):RetrievalReferenceNumber_037 = '000004279135' (CHARACTER)
(0x03000000:NameValue):ApprovalCode_038 = '129601' (CHARACTER)
(0x03000000:NameValue):ResponseCode_039 = '988' (CHARACTER)
(0x03000000:NameValue):CardAcceptorTerminalIdentification_041 = '99900099' (CHARACTER)
(0x03000000:NameValue):AdditionalDataPrivate_048 = '001001100200373602500120260015027000028012000004279135' (CHARACTER)
(0x03000000:NameValue):CurrencyCodeReconciliation_050 = '368' (CHARACTER)
(0x03000000:NameValue):AmountsAdditional_054 = '007016C000000000000368' (CHARACTER)
(0x03000000:NameValue):AccountIdentification1_102 = '00136821010100100044000' (CHARACTER)
)
)
) |
The user trace wrote a lot of logs, I'll assume that this is the interesting stuff:
| Code: |
UserTrace BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement ''BEGIN ... END;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '2.2').
UserTrace BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement ''SET OutputRoot = InputRoot;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '6.7').
UserTrace BIP2539I: Node '': Evaluating expression ''InputRoot'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '6.24'). This resolved to ''InputRoot''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
UserTrace BIP2568I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Copying sub-tree from ''InputRoot'' to ''OutputRoot''.
UserTrace BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement ''SET OutputRoot.Properties.CodedCharSetId = 1208;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '11.6').
UserTrace BIP2566I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Assigning value ''1208'' to field / variable ''OutputRoot.Properties.CodedCharSetId''.
UserTrace BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement ''PROPAGATE TO TERMINAL 'out1' FINALIZE DEFAULT DELETE DEFAULT;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '12.6').
UserTrace BIP4016I: Message propagated to terminal ''out1'' of node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB' with the following message trees: 'InputLocalEnvironment, OutputRoot, InputExceptionList'.
UserTrace BIP2539I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace': Evaluating expression ''Root'' at ('', '1.3'). This resolved to ''Root''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
UserTrace BIP4067I: Message propagated to output terminal for trace node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace'.
The trace node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace' has received a message and is propagating it to any nodes connected to its output terminal.
No user action required.
UserTrace BIP7944I: A message is being processed in node ''msg_TCPISO8583.Rebuild_ISO_Msg.MQ Output'', with the following attributes derived from the policy at '''': ''connection: SERVER, destinationQueueManagerName: APSQMGR, logLabel: msg_TCPISO8583.Rebuild_ISO_Msg.MQ Output''.
Each message that is processed by the node might use different attributes derived from a policy. This message records the attribute values that are used for a specific message.
No user action required.
UserTrace BIP5841I: ''Offset: 0. Starting to process the DFDL infoset.''
UserTrace BIP5841I: ''Offset: 0. Element 'ISO8583_1993_Unpacked' was found in the infoset.''
UserTrace BIP5841I: ''Offset: 0. Starting to write root element 'ISO8583_1993_Unpacked'.''
UserTrace BIP5841I: ''Offset: 0. Starting to process element 'ISO8583_1993_Unpacked'.''
UserTrace BIP5841I: ''Offset: 0. Element 'MTI_Version' was found in the infoset.''
UserTrace BIP5841I: ''Offset: 0. Starting to process element 'MTI_Version'.''
UserTrace BIP5841I: ''Offset: 0. Value '1' of type xs:decimal received for element 'MTI_Version' .''
UserTrace BIP5843E: ''The DFDL serializer cannot output the character '' in encoding 'US-ASCII' for the value of element 'MTI_Version'.'' |
Adding the type clause to the ASBITSTREAM function did not help, the same error occured. |
|
| Back to top |
|
 |
| abooddy |
| Posted: Fri Jun 14, 2019 12:09 pm Post subject: |
|
|
Novice
Joined: 28 May 2019
Posts: 20
Location: Baghdad, Iraq
|
This is how it appears in Notepad++:
 |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 15, 2019 7:38 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
I believe there is confusion here between what should be a character representation of a digit, and the numeric representation of the digit.
Should that part not serialize as a BLOB and not as character?
At best a hex representation of the BLOB... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Jun 16, 2019 3:13 pm Post subject: Re: DFDL Numbers Parsing Error |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| abooddy wrote: |
I'll assume that this is the interesting stuff:
| Code: |
...
UserTrace BIP5841I: ''Offset: 0. Value '1' of type xs:decimal received for element 'MTI_Version' .''
UserTrace BIP5843E: ''The DFDL serializer cannot output the character '١' in encoding 'US-ASCII' for the value of element 'MTI_Version'.'' |
|
The interesting stuff would probably be happening between those two statements. The initial post indicated that changing the Windows system locale didn't help, and according to timber, "DFDL never uses platform/OS defaults" anyway.
But, at the very least, changing the user locale (or codepage) may fix the substituting of arabic characters with the '<SUB>' control character, in the user trace output.
| abooddy wrote: |
| Code: |
| <xs:element dfdl:length="1" dfdl:textNumberPattern="#0" ibmDfdlExtn:sampleValue="1" name="MTI_Version" type="xs:integer"/> |
|
How does the DFDL parser convert this element to "xs:decimal", when the posted schema defines the element type as "xs:integer"? Is that from the right XSD?
It might help to provide some more detail, clarifying the order of transformations, and the schemas involved, for InputRoot ({}:TCPGlobalMsg), BLOB B, and ROW R.
Here is another (possibly irrelevant) example, using iconv on Linux to convert the Arabic characters from Unicode to IBM-9448:
| Code: |
$ echo -n "١٧" | iconv -f UTF-8 -t IBM-9448 | od -c -t xC
0000000 1 7
31 37 |
It substitutes the given arabic digits with the corresponding ASCII digits ('1' and '7'). That works for 'IBM-9448', but not for 'WINDOWS-1256'.
However, it is not clear what that might mean, for the DFDL parser, in the broker, on a Windows platform. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Jun 17, 2019 1:31 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Thanks - that's very helpful. It doesn't look like correct behaviour to me, and I wonder whether DFDL is strugging to output the DECIMAL value. What happens if you CAST MtiVersion to INTEGER before sending the message tree to DFDL? |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Mon Jun 17, 2019 2:34 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| fjb_saper wrote: |
I believe there is confusion here between what should be a character representation of a digit, and the numeric representation of the digit.
Should that part not serialize as a BLOB and not as character? |
Technically, everything serializes as a BLOB. But the DFDL schema governs the exact format of the resulting BLOB:
| Code: |
...
<dfdl:format encoding="US-ASCII" ...>
...
<xs:element dfdl:length="1" dfdl:textNumberPattern="#0" ibmDfdlExtn:sampleValue="1" name="MTI_Version" type="xs:integer"/>
... |
In this case, the parsed element is an INTEGER, and the unparsed BLOB is a "US-ASCII" byte, representing the INTEGER value.
Someone else can probably much better explain the internal workings of the DFDL parser, but there seems to be an intermediate conversion while unparsing, from INTEGER to CHARACTER, (before converting from CHARACTER to BLOB), where DFDL converts the INTEGER value to an Arabic CHARACTER, instead of an ASCII character.
A separate question is: Why are the parsed elements DECIMAL, when the posted schema defines them as INTEGER? |
|
| Back to top |
|
|
| abooddy |
| Posted: Thu Jun 20, 2019 10:24 am Post subject: |
|
|
Novice
Joined: 28 May 2019
Posts: 20
Location: Baghdad, Iraq
|
Hi,
Unfortunately, I can only work on this issue on weekends, so I will available for the next two days very effectively and hopefully we manage to fix the issue by then.
| Quote: |
| But, at the very least, changing the user locale (or codepage) may fix the substituting of arabic characters with the '<SUB>' control character, in the user trace output. |
My system locale is "English (United Kingdom)" and my active code page according to chcp is 850. I see that code page 850 does not contain the Arabic numbers.
| Quote: |
| How does the DFDL parser convert this element to "xs:decimal", when the posted schema defines the element type as "xs:integer"? Is that from the right XSD? |
Yes it is from the correct and only XSD. The line:
| Code: |
CREATE LASTCHILD OF R DOMAIN('DFDL')
PARSE(B CCSID InputRoot.Properties.CodedCharSetId TYPE
'{http://www.ibm.com/dfdl/ISO8583-1993}:ISO8583_1993_Unpacked' OPTIONS RootBitStream); |
Creates the element MTI_Version as a DECIMAL, not an INTEGER. This should be really weird, but I don't think it is the issue because when I tried:
| Quote: |
| What happens if you CAST MtiVersion to INTEGER before sending the message tree to DFDL? |
The same issue happened. |
|
| Back to top |
|
|
| timber |
| Posted: Thu Jun 20, 2019 11:28 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| Why are the parsed elements DECIMAL, when the posted schema defines them as INTEGER? |
That's a fair question, and one that I do know the answer to. The xsd:integer type is unconstrained (has infinite range) so it would be dangerous to convert that type to the native 64-bit integer type of ESQL. Early versions of IBM DFDL did, and the defect was caught and fixed.
If the CAST to INTEGER had fixed the problem, then I would have recommended a change from xsd:integer to xsd:long (which is defined to be 64-bit). But it didn't, so no point in making the change. |
|
| Back to top |
|
|
| timber |
| Posted: Thu Jun 20, 2019 11:40 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I have to admit that I'm almost out of ideas. The field is a text representation of an integer in US-ASCII. The output should be a single byte with value 0x31.
There is no good reason why this should work...but maybe it's worth explicitly setting the 'encoding' property on MtiVersion to 'ISO-8859-1' (which is an alias for 7-bit US-ASCII). You can do that in the DFDL editor. It should add an attribute dfdl:encoding="ISO-8859-1" to the element definition for MtiVersion, which will override the default value.
If that does not change anything (and it probably will not) then I think it's time to ask IBM for some help. If you do open a PMR, I would advise you to reference this thread in your problem description. |
|
| Back to top |
|
|
| abooddy |
| Posted: Fri Jun 21, 2019 2:57 am Post subject: |
|
|
Novice
Joined: 28 May 2019
Posts: 20
Location: Baghdad, Iraq
|
Hi,
| Quote: |
| maybe it's worth explicitly setting the 'encoding' property on MtiVersion to 'ISO-8859-1' |
Same issue occurred:
| Code: |
| The DFDL serializer cannot output the character '١' in encoding 'ISO-8859-1' for the value of element 'MTI_Version'. |
Thank you all for you support. Seems I'm forced to use the Linux brokers, which is not that bad. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Jun 21, 2019 10:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| abooddy wrote: |
| My system locale is "English (United Kingdom)" and my active code page according to chcp is 850. I see that code page 850 does not contain the Arabic numbers. |
Keep in mind that this would just fix the mqsiformatlog output file; it doesn't address the underlying issue. The Windows codepage for utf-8 is 65001. Try that. If that doesn't work, then another option is to look through the mqsireadlog output xml file for non-ASCII characters instead; mqsireadlog encodes its output as utf-8.
| timber wrote: |
| The xsd:integer type is unconstrained (has infinite range) so it would be dangerous to convert that type to the native 64-bit integer type of ESQL. |
That's good to know. Working as designed. ESQL DECIMAL is also finite, but that's probably less of an issue. ;-)
| timber wrote: |
| ... it's time to ask IBM for some help. If you do open a PMR, ... |
IBM calls them "cases" now.
| abooddy wrote: |
| Seems I'm forced to use the Linux brokers, which is not that bad. |
Some might call that an upgrade. Still, it's probably worthwhile to open up a support case with IBM. If there's a defect with the DFDL parser, then fixing it could help future users too. |
|
| Back to top |
|
|
| abooddy |
| Posted: Mon Jan 30, 2023 3:24 am Post subject: |
|
|
Novice
Joined: 28 May 2019
Posts: 20
Location: Baghdad, Iraq
|
Hi,
Four years later. 
We encountered this issue on a Linux machine and we were able to identify the issue and fix it.
The issue was that IIB (ASBITSTREAM? DFDL? Something else?) was falling back to OS locale while serializing.
We've changed the OS locale to Arabic and this issue started happening, once we set it back to English, the serializer worked as expected.
| Code: |
| localectl set-locale LANG=en_US |
I'm posting this because it could help others facing the same issue. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
