| Author |
Message
|
| somsayan |
 Posted: Mon Dec 30, 2013 8:13 am Post subject: Ignoring line feed in fixed length message [SOLVED] |
 |
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
Hi,
We are having a scenario, where the source system is sending us messages in fixed length format. There is a description field that has a length of 200. This field can include <LF>. The length of this field is 200 if we ignore the <LF> in this field.
Using DFDL, is there a way to ignore a particular character in one of the fields. I can't replace the LF in BLOB format as the lines are delimited using LF.
Can anyone help please. 
Last edited by somsayan on Tue Jan 07, 2014 4:23 am; edited 1 time in total |
|
| Back to top |
|
 |
| dogorsy |
| Posted: Mon Dec 30, 2013 10:07 am Post subject: Re: Ignoring line feed in fixed length message |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| somsayan wrote: |
Hi,
We are having a scenario, where the source system is sending us messages in fixed length format. There is a description field that has a length of 200. This field can include <LF>. The length of this field is 200 if we ignore the <LF> in this field.
Using DFDL, is there a way to ignore a particular character in one of the fields. I can't replace the LF in BLOB format as the lines are delimited using LF.
Can anyone help please. |
Yes, set %LF as the record terminator |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Dec 30, 2013 4:25 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So the length of the Description field is always exactly 200 characters, but some of those characters might be LFs? In that case, the solution is to alter the definition of the Description field as follows:
- change LengthKind from 'Delimited' to 'Explicit'
- set Length to 200
Note that DFDL is more flexible than MRM TDS in this respect. With TDS, if Data Element Separation is 'Delimited' then all members of the group are controlled by the delimiter or terminator.
In DFDL a delimiter ( separator or terminator ) is only used when LengthKind is 'delimited' or 'end of parent'.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| somsayan |
| Posted: Tue Dec 31, 2013 1:54 am Post subject: |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| kimbert wrote: |
| So the length of the Description field is always exactly 200 characters, but some of those characters might be LFs? |
Hi Kimbert,
The length of the field is 200 if we ignore all the LFs in the field. If there are 3 LFs, the field length is 203.
This is one field in a string having a total length of 2042 characters. So based on the number of LFs in this field (Description) the total length might be more that 2042.
Regards
Som |
|
| Back to top |
|
|
| somsayan |
| Posted: Tue Dec 31, 2013 1:57 am Post subject: Re: Ignoring line feed in fixed length message |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| dogorsy wrote: |
Yes, set %LF as the record terminator |
Hi Dogorsy,
I don't think this will work. The field concerned is a part of a bigger string. The input is something like :
<<Name 100 chars>><<OperatingCompany 10 characters>><<Description 200 characters>><<..........>>
Sorry, if I put my question badly and misguided you.
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Dec 31, 2013 6:38 am Post subject: Re: Ignoring line feed in fixed length message |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| somsayan wrote: |
| Sorry, if I put my question badly and misguided you. |
You didn't. He answered badly.
I'm otherwise confused by your answer to Kimbert.
Is the field fixed length? Or not? Either you need to get 200 characters, or you need to get all characters up to some delimiter.
If you have to get "all characters up to some delimiter, unless it's not the right delimiter", then you need to deal with quoting delimiters, and you're really talking about a CSV format. DFDL comes with a predefined message format for the general CSV pattern, and you can adjust your model for your specific delimiters. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Dec 31, 2013 9:01 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Your data format is rather...unusual. You should not be surprised to learn that DFDL has no built-in feature for dealing with this style of message. However, DFDL does have a very powerful general-purpose feature that should work...if you are prepared to put in some effort.
Set Length Kind to 'Data Pattern'
Set 'Data Pattern' to
| Code: |
| \n?([^\n]|[^\n]\n){200}\n |
You may need to adjust the data pattern.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| somsayan |
| Posted: Thu Jan 02, 2014 1:22 am Post subject: Re: Ignoring line feed in fixed length message |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| mqjeff wrote: |
| Is the field fixed length? Or not? Either you need to get 200 characters, or you need to get all characters up to some delimiter. |
Hi Mqjeff,
We are doing a migration project and unfortunately don't have the authority to change the source or destination systems behaviour.
| kimbert wrote: |
| Set Length Kind to 'Data Pattern' |
Hi Kimbert,
I'm getting the following error while doing this
| Code: |
| CTDV1150E : IBM DFDL implementation does not support 'lengthKind' value 'pattern'. |
Can you please advice the steps I missed ? |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jan 02, 2014 2:40 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I assume that you are using WMB v8. DFDL in WMB v8 does not support lengthKind='dataPattern'. Your data format really does need this feature. Would it be possible to upgrade to IIB v9?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| somsayan |
| Posted: Thu Jan 02, 2014 2:43 am Post subject: |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| kimbert wrote: |
| I assume that you are using WMB v8. DFDL in WMB v8 does not support lengthKind='dataPattern'. Your data format really does need this feature. Would it be possible to upgrade to IIB v9? |
Hi Kimbert,
Its really not an option currently to upgrade right now 
Is there some other workaround for this ? |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jan 02, 2014 2:57 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I cannot think of any other way to model this data format. It requires the power of a regular expression to parse it.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jan 02, 2014 4:21 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Actually, I can think of a workaround. I would not normally recommend using MRM when DFDL is available, but your requirements are very specific.
The MRM parser can use a data pattern to determine the length of a field or structure. On any complex type/group you can set 'Data Element Separation' to 'Use Data Pattern' and then you must supply a data pattern on each group member that matches the entire length of that group member.
Note that the MRM parser remains a fully-supported part of the product, so there's no risk involved here.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| somsayan |
| Posted: Thu Jan 02, 2014 5:00 am Post subject: |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| kimbert wrote: |
Actually, I can think of a workaround. I would not normally recommend using MRM when DFDL is available, but your requirements are very specific.
The MRM parser can use a data pattern to determine the length of a field or structure. On any complex type/group you can set 'Data Element Separation' to 'Use Data Pattern' and then you must supply a data pattern on each group member that matches the entire length of that group member.
Note that the MRM parser remains a fully-supported part of the product, so there's no risk involved here. |
Hi Kimbert,
Thanks for showing us light at the end of the tunnel
We'll try this and post the results. |
|
| Back to top |
|
|
| somsayan |
| Posted: Tue Jan 07, 2014 1:04 am Post subject: |
|
|
Apprentice
Joined: 06 Apr 2007
Posts: 49
|
| somsayan wrote: |
| We'll try this and post the results. |
Hi,
We are getting this file via MFT. We have asked the MFT team to run a script before putting the file to our queue. This script should strip the LF's from the data for that particular line.
Hopefully this will work |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jan 07, 2014 2:57 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| We have asked the MFT team to run a script before putting the file to our queue. This script should strip the LF's from the data for that particular line. |
That sounds like a very sensible solution. A data format should be simple to interpret. The format that MFT was sending was anything but simple!
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
|
|
