| Author |
Message
|
| mqjeff |
 Posted: Thu Jun 02, 2016 8:05 am Post subject: |
 |
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jun 02, 2016 8:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| http://www.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bc34044_.htm |
Yep, I know that. How would you leverage that to indicate that the event you've processed was triggered by the last insert of a row that's going to happen in the current business process?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jun 02, 2016 8:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The trigger that's written (by the user!) includes the last row to process...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 02, 2016 8:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| The trigger that's written (by the user!) includes the last row to process... |
So (pedantically and more to stop the OP heading off down a rabbit hole) there's nothing intrinsic to the DatabaseInput node that detects that it's finished processing any given business day's data.
It's possible to architect a solution where the user-written trigger populates the event table so that the DatabaseInput reacts to the "end of file" condition, presumably by the program populating the table adding something the trigger can react to.
I still think a different ETL tool would be easier.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jun 02, 2016 8:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| The trigger that's written (by the user!) includes the last row to process... |
So (pedantically and more to stop the OP heading off down a rabbit hole) there's nothing intrinsic to the DatabaseInput node that detects that it's finished processing any given business day's data. |
Pendantically, the DatabaseInput node stops processing records when the event table is empty (or all records are marked processed)
If that happens to coincide with the end of business day's data, all the better.
| Vitor wrote: |
| I still think a different ETL tool would be easier. |
Sure. Doesn't mean it will know when to stop processing records, either.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Thu Jun 02, 2016 8:40 am Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
| Vitor wrote: |
| mqjeff wrote: |
| Either when the last row is read, or when the business data indicates it's the end of file. |
The business data doesn't seem to indicate this. Lack of more data seems to be the "delimiter" |
The data will be in Database. So there is no "delimiter".
| mqjeff wrote: |
| "The last row is read" is a condition discoverable from how the DatabaseInput node works. |
Pseudo - code?[/quote]
I am not looking for Pseudo -code. Looking for the option to be set in Database Input Node so that IIB knows about "The last row is read" . |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 02, 2016 8:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| If that happens to coincide with the end of business day's data, all the better. |
Except the OP wants 1 file per day.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 02, 2016 8:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Partha.Baidya wrote: |
| Vitor wrote: |
| mqjeff wrote: |
| Either when the last row is read, or when the business data indicates it's the end of file. |
The business data doesn't seem to indicate this. Lack of more data seems to be the "delimiter" |
The data will be in Database. So there is no "delimiter". |
Exactly the point I was trying to make to my most worthy associate.
| Partha.Baidya wrote: |
| Vitor wrote: |
| Pseudo - code? |
I am not looking for Pseudo -code. |
I was asking my most worthy associate for some pseudo - code to support his assertion.
| Partha.Baidya wrote: |
| Looking for the option to be set in Database Input Node so that IIB knows about "The last row is read" . |
As I explained above, there isn't one. IIB can't know that the row just processed off the event table is the "last" one, be that the last one of the day or the last one of the current set of records being dripped into the database.
Again as explained above and espoused by my most worthy associate, you could develop something that fulfilled the function. But your current design is predicated on the erroneous belief that there's an "end of table" condition supported by database software in the same way an "end of file" condition is supported by an OS.
It isn't.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jun 02, 2016 8:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| If that happens to coincide with the end of business day's data, all the better. |
Except the OP wants 1 file per day. |
Sure. The event trigger can determine when the time has changed from "yesterday" to "today", and then write a record to the event table that says "read from here to here".
Assuming that records are being written to the database continually, without end.
If not, then there's various other ways the trigger can be run to determine if it should trigger the DatabaseInput node. Or a regular cron/at/etc. job can simply write a record to the event table.
The SELECT query used to read the data table does not have to make any use of the event table record, not really.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 02, 2016 8:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Assuming that records are being written to the database continually, without end. |
And the intention to do so is not foiled by network errors, application problems, etc.
| mqjeff wrote: |
| If not, then there's various other ways the trigger can be run to determine if it should trigger the DatabaseInput node. Or a regular cron/at/etc. job can simply write a record to the event table. |
Which brings you back to using a cron to start a flow and a Database node. Which brings us back to....
| mqjeff wrote: |
| The SELECT query used to read the data table does not have to make any use of the event table record, not really. |
....reading a large amount of data into memory and processing it in a slow, inefficient and impossible to scale process.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
