| Author |
Message
|
| pomeroy8 |
 Posted: Tue May 13, 2008 11:09 am Post subject: FileInput Parsed Record Sequence & failure |
 |
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
I would like to use the FileInput node with "Parsed Record Sequence". I have created my msgset and msgflow. I have set the FileInput node validation to "Content and Value".
I want all successfully parsed messages to flow into my compute node etc. I want all unsuccessfully parsed messages to go to a "reject" process and to continue processing next parsed message. I have set the failure action on the validation to be "Local Error Log" which according to help is supposed to "continue regardless of validation failures". I have wired the failure terminal of the FileInput node to a "reject" process that currently does nothing (Compute node that returns FALSE). When I pass bad data into the FileInput node, the messages up to the bad data are output to my MOOutput node, however, the processing stops and the file is written to the mqsibackout folder.
How do I achieve a flow that will process the entire filw with all the valid parsed messages and reject the invalid parsed messages?
Thanks in advance. |
|
| Back to top |
|
 |
| kimbert |
| Posted: Tue May 13, 2008 1:53 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Quote: |
| How do I achieve a flow that will process the entire filw with all the valid parsed messages and reject the invalid parsed messages? |
First, you need to define 'valid' and 'invalid'.
Bear in mind that some forms of 'invalid' message could cause the parsing of more than one message to fail ( because the parser might run on into the next message, or even to the end of the file ).
| Quote: |
| When I pass bad data into the FileInput node, the messages up to the bad data are output to my MOOutput node, however, the processing stops and the file is written to the mqsibackout folder. |
Take a user trace and look at it carefully. It might contain a clue about why the parser is acting strangely. I would guess that you are getting parsing errors, not validation errors. You can't suppress parsing errors. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Wed May 14, 2008 4:51 am Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
Interest. You just made me realize that I don't really have a clear understanding of how WMB or I interpret "valid" or "invalid". Is there a concise definition for WMB?
To your first point, the "bad" data is actually near the beginning of the message not the end. It's unlikely that the error is due to runnning into the next message. The message set is TaggedFixedLength. The "bad" data is in the first of the tagged records.
The "bad" data consists of having a string where an integer was expected.
Thanks, I'll look at the trace. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed May 14, 2008 10:39 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Quote: |
| I don't really have a clear understanding of how WMB or I interpret "valid" or "invalid". Is there a concise definition for WMB? |
The MRM parser issues a parsing exception when one of the following occurs:
- It runs out of bitstream before encountering the end of the model
- It runs out of model before encountering the end of the bitstream
- It cannot convert the physical data ( bytes/characters) to the logical type defined in the message definition. ( e.g. 'xyz' cannot be converted to a float )
- It encounters a character conversion exception while transoding from the input code page to the broker's native UTF16.
etc
A validation exception is issed if validation is enabled and
- minOccurs or maxOccurs is violated
- A type or group contains undeclared members ( unless Content Validation is set to Open or Open Defined ).
- A simple value does not obey the Value Constraints ( schema facets ) on its simple type.
etc
Most XML validating parsers don't make that distinction, so a lot of people don't expect the MRM parser to make it either. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Thu May 15, 2008 10:33 am Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
Thanks. Based on your definitions I have a parser error. (had to do "real" work yesterday sorry for the delay responding).
I change the FileInput node to parse "onDemand". However, I left validation to "Content and Value".
Now when I send the "bad" data. The initial "good" messages are processed to the output queue. on the "bad" message initially the flow just seemed to end. The file was deleted. I changed the "Action on successful processing" to "Add timestamp ..." Now the file goes to the mqsiarchive folder.
This seems to indicate that the file was processed correctly even though it was not.
| Quote: |
| You can't suppress parsing errors. |
This seems to me to be a FUNDAMENTAL deficiency with FileInput node. A simple scenario is that I receive files from different suppliers with varying levels of data quality/integrity. I still need to process all the "good" messages a supplier sends me (perhaps so that they can get paid) and I need to reject the "bad" messages.
If I can't suppress parsing errors then:
how useful is the failure terminal on fileInput node? this is how we would handle parser errors for MQInput
how am I supposed to achieve my business requirement to reject "bad" data?
Thank you for your input/time thus far. We really had high expectations that WMB v6.1 file nodes would be the answer here. I don't want to have to go back and say, "No, we need to buy/build our own adapter". |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu May 15, 2008 10:44 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Suppose you receive a file that starts off with 100 good records.
It then includes a single, incorrect, file termination character, and then a long string of other characters that may or may not represent "good records".
In what manner is the FileInput node supposed to be able to determine what to do, after reading the file termination character?
Or indeed, any other character that does not actually match the model you've told it to use.
Or suppose that you receive a file that has a single extra character (perhaps the letter 'a') as the very first character. Every other byte of the file is absolutely correct and consistent with your model. How many bad records are there in this file? how many good records?
These are not random, abstract, questions. These are questions that you need to consider when dealing with your specific problem, and determine what your specific exception handling requirements and procedures are. Then you can look at how to use the FileInput node to implement these.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Thu May 15, 2008 12:30 pm Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
Thanks for your input. You make some valid points, however, as is the FileInput node provides NO OPTION to deal with these in my flow.
Of course, it is impossible to guard against all possible ways a file can be incorrect or corrupted.
The assumption is that the FileInput node, with ParedRecordSequence, was able to find a message. I want to determine if the message is "good" or "bad". If its "good" then fine. If its "bad" what can I do? MQInput nodes have a Failure terminal that allows me to do something useful with a "bad" message and keep processing the queue. How can I keep processing the file given my assumption that ParsedRecordSequence will find a message? The Failure terminal of the FileInput node seems the logical choice, however, apparently you can't suppress parser errors which makes its usefulness suspect/misleading.
I could read the file a record at a time and build the message in a Compute node which would PROPAGATE a complete message. I would have the parsing to a later step. This approach requires more code on my part (always bad) compared to using the ParsedRecordSequence approach.
Finally, I need a new (custom) file adapter. Why should I buy/build one when I already have one that is supported by my vendor? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu May 15, 2008 2:46 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You can use the FileInput node in several modes, at least some of which, as is, will let you deal with your file in the way you want to.
For example, in a fixed length way to dump unparsed "records" onto a queue, where you can then parse, validate, backout, deal with them, etc.
Or for example as a single output message representing the entire file, where you can then use SUBSTRING and CREATE FIELD...PARSE to deal with your structures, and deal with your errors as you like.
It may not be that the FileInput node, as delivered in 6.1.0.1, does exactly and explicitly what you want it to out of the box. That doesn't mean that it won't in 6.1.0.2, or in later time frames either. Nor does it mean that, out of the box at 6.1.0.1 there is not a solution for your problem with Broker - that may require more than just a single FileInput node.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri May 16, 2008 12:46 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Quote: |
| Based on your definitions I have a parser error |
Can you quote the BIP number please. I would like to understand the problem that you are facing. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Fri May 16, 2008 6:25 am Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
@ kimbert
there were NO ERRORS in the Windows Event Application log!! After the failure, the file went into the mqsiarchive folder (hmm....?).
I did, however, grab the following from the debugger.
| Code: |
Message
Properties
MessageSet IA106_TAGGEDDELIMITED_MSP
MessageType {}:InvoiceGroups
MessageFormat TDS1
Encoding 546
CodedCharSetId 437
Transactional false
Persistence false
CreationTime 2008-05-16 14:10:26.875
ExpirationTime -1
Priority 0
ReplyIdentifier
ReplyProtocol
Topic
ContentType
IdentitySourceType
IdentitySourceToken
IdentitySourcePassword
IdentitySourceIssuedBy
IdentityMappedType
IdentityMappedToken
IdentityMappedPassword
IdentityMappedIssuedBy
MRM
ECHeaderRecord
SenderId 5195795736
ReceiverId 201613668
ANSITransactionType 810
LocalEnvironment
File
Wildcard
Environment
ExceptionList
RecoverableException
File F:\build\S000_P\src\DataFlowEngine\ImbDataFlowNode.cpp
Line 739
Function ImbDataFlowNode::createExceptionList
Type ComIbmTryCatchNode
Name IA106_FILE_IN_XML_OUT#FCMComposite_1_3
Label IA106_FILE_IN_XML_OUT.TryCatch
Catalog BIPv610
Severity 3
Number 2230
Text Node throwing exception
RecoverableException
File F:\build\S000_P\src\DataFlowEngine\ImbMqOutputNode.cpp
Line 894
Function ImbMqOutputNode::evaluate
Type ComIbmMQOutputNode
Name IA106_FILE_IN_XML_OUT#FCMComposite_1_5
Label IA106_FILE_IN_XML_OUT.IA106_XML_OUT
Catalog BIPv610
Severity 3
Number 2230
Text Caught exception and rethrowing
ParserException
File F:\build\S000_P\src\MTI\MTIforBroker\MtiImbParser2\MtiImbParser.cpp
Line 1865
Function MtiImbParser::refreshBitStreamFromElements - 9 par
Type ComIbmFileInputNode
Name IA106_FILE_IN_XML_OUT#FCMComposite_1_2
Label IA106_FILE_IN_XML_OUT.IA106_PARSE_FILE_IN
Catalog BIPv610
Severity 3
Number 5286
Text ImbRecoverableException caught from worker when attempting to write out the bitstream.
Insert
Type 5
Text IA106_TAGGEDDELIMITED_MSP
Insert
Type 2
Text 1
Insert
Type 5
Text XML1
Insert
Type 5
Text /InvoiceGroups
ParserException
File F:\build\S000_P\src\MTI\MTIforBroker\MtiImbParser2\MtiImbParser.cpp
Line 4061
Function MtiImbParser::checkSeqForMissingMandatories
Type ComIbmFileInputNode
Name IA106_FILE_IN_XML_OUT#FCMComposite_1_2
Label IA106_FILE_IN_XML_OUT.IA106_PARSE_FILE_IN
Catalog BIPv610
Severity 3
Number 5374
Text An element has been deemed complete occurring less than specified minOccurs
Insert
Type 5
Text InvoiceHeaderRecord
Insert
Type 2
Text 0
Insert
Type 2
Text 1
Insert
Type 5
Text InvoiceGroups
Insert
Type 2
Text 1 |
So its a BIP2230. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Fri May 16, 2008 6:36 am Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
@ jefflowrey
| Quote: |
| For example, in a fixed length way to dump unparsed "records" onto a queue, where you can then parse, validate, backout, deal with them, etc. |
I had already written something similar to this in order to write the contents of a FileInput node into an MQ group of messages. This was my scenario where all the messages had to be "good" in order to continue processing. I was disappointed that the FileInput node as is did not support MQ grouping, however, I can see that the file parser is not "looking ahead". In this case, I did not write any code to "backout, deal with them, etc." as this was not the purpose of the test. I was satisfied that the incomplete group was sufficient to prevent down stream processing.
Given my recent problems, I will have to revisit this, to validate that the files are going into the correct folders after processing. If they are going to the mqsiarchive folder, then, I would say we have a problem. |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Fri May 16, 2008 7:15 am Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
@ kimbert
More testing ...
I had a TryCatch node between the FileInput node and a Compute node. The catch terminal was wired to a Compute node that did nothing (IEFBR14 for us old guys) (RETURN FALSE). When I removed the TryCatch node, now the file goes into the mqsibackout folder. There are messages in the Application Event Log
If I restore the TryCatch node and ADD a Throw after my IEFBR14 the file goes to the mqsiarchive folder. NO messages in the Application Event Log.
In neither case have I wired the failure terminal of the FileInput node. |
|
| Back to top |
|
|
|
|
