| Author |
Message
|
| smdavies99 |
 Posted: Tue Aug 25, 2015 7:42 am Post subject: Re: Logging XML instead of wireframe format |
 |
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| sleepyjamie wrote: |
We have IBM contractors who developed code for us, so we are just following their implementation. The trace logs are typically enabled when something goes wrong in production, and are only enabled for a short time period to capture data to analyze issues related 3rd party Web Services.
|
This is no substitute for a decent logging and auditing framework.
One that would
1) log all inputs and outputs + other items that make support a lot easier. e.g. SalesOrder
2) Be configurable so that when not needed the ouptuts in 1) above could be disabled on a flow by flow basis
3) Also log exceptions with enhanced information as well as the whole exceptionList
4) Log to a Database with proper indexes makes searching for that elusive sales order number much easier.
5) Be automatically purged to stop the tables from getting too big.
OR something else entirely. I know that other sites use different solutions.
The hours you might spend putting it all together soon pays off.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
 |
| sleepyjamie |
| Posted: Tue Aug 25, 2015 7:44 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| Vitor wrote: |
| mqjeff wrote: |
| It might not be terribly easy, but I'm reasonably sure the capability is there. |
I'm pleased to see your mastery of understatement remains undiminished 
But I agree - I'm sure you can write any code in any language to perform any task. Eventually.
I'll settle for being really pleased when the Trace node executes an ASBITSTREAM. Make my infrastructure a lot easier. |
Coming from Apache Camel world for 7 years, this has been baked-in since version 1.5 (and possibly earlier), and explicitly made available using their Simple Expression Language since version 2+.
Hopefully IIB has similar capability. I'll let you know in a few days. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Tue Aug 25, 2015 7:46 am Post subject: Re: Logging XML instead of wireframe format |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| smdavies99 wrote: |
| sleepyjamie wrote: |
We have IBM contractors who developed code for us, so we are just following their implementation. The trace logs are typically enabled when something goes wrong in production, and are only enabled for a short time period to capture data to analyze issues related 3rd party Web Services.
|
This is no substitute for a decent logging and auditing framework.
One that would
1) log all inputs and outputs + other items that make support a lot easier. e.g. SalesOrder
2) Be configurable so that when not needed the ouptuts in 1) above could be disabled on a flow by flow basis
3) Also log exceptions with enhanced information as well as the whole exceptionList
4) Log to a Database with proper indexes makes searching for that elusive sales order number much easier.
5) Be automatically purged to stop the tables from getting too big.
OR something else entirely. I know that other sites use different solutions.
The hours you might spend putting it all together soon pays off. |
Totally agree! I am pushing for this, however we have a big organization, with lots of people and departments, and lots of contractors (onsite and remote). Its not a WMB problem, its an organizational/political problem right now.
Also, i'm not the architect so I can only provide advice. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Tue Aug 25, 2015 7:50 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
We definitely do this. One side question. If we wanted to implement a trace/logging standard, does IIB have a way to import logging templates? Currently all trace nodes must be explicitly entered into the text field. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 25, 2015 7:57 am Post subject: Re: Logging XML instead of wireframe format |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
This is no substitute for a decent logging and auditing framework.
One that would
1) log all inputs and outputs + other items that make support a lot easier. e.g. SalesOrder
2) Be configurable so that when not needed the ouptuts in 1) above could be disabled on a flow by flow basis
3) Also log exceptions with enhanced information as well as the whole exceptionList
4) Log to a Database with proper indexes makes searching for that elusive sales order number much easier.
5) Be automatically purged to stop the tables from getting too big.
OR something else entirely. I know that other sites use different solutions.
The hours you might spend putting it all together soon pays off. |
I'm interested by the ability to run the very similar framework you describe (particularly the database for that elusive <insert data item> that absolutely was sent to the broker because our application never fails) along with a more lightweight solution for those of my customers that just want to enjoy a reassuring text file with lots of hard to search ids because it makes them feel good.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 25, 2015 8:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sleepyjamie wrote: |
| Coming from Apache Camel world for 7 years, this has been baked-in since version 1.5 (and possibly earlier), and explicitly made available using their Simple Expression Language since version 2+. |
IIB is not Apache. It is not now nor has it ever been from the Apache Java world; indeed it landed in the Java world from a different solar system 4 versions ago. Nothing about it structurally is the same except it can run Java code.
IIB is like Apache in the same way my dog is like a Bengal tiger; they both have 4 paws and a tail. Aside from that, rather different.
| sleepyjamie wrote: |
| Hopefully IIB has similar capability. I'll let you know in a few days |
The suspense is killing me. I'll understand if this causes you to wait even longer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Tue Aug 25, 2015 8:04 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| Vitor wrote: |
| sleepyjamie wrote: |
| Coming from Apache Camel world for 7 years, this has been baked-in since version 1.5 (and possibly earlier), and explicitly made available using their Simple Expression Language since version 2+. |
IIB is not Apache. It is not now nor has it ever been from the Apache Java world; indeed it landed in the Java world from a different solar system 4 versions ago. Nothing about it structurally is the same except it can run Java code.
IIB is like Apache in the same way my dog is like a Bengal tiger; they both have 4 paws and a tail. Aside from that, rather different.
| sleepyjamie wrote: |
| Hopefully IIB has similar capability. I'll let you know in a few days |
The suspense is killing me. I'll understand if this causes you to wait even longer. |
True, I'm not saying they are the same. However they are both integration products. I'm trying to find a solution in IIB for something simple that has existed in other integration products I've used. ie: output raw XML to a log file. To me this is something simple.
Haha ok, maybe I'll try it out tonight! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 25, 2015 8:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The basic message monitoring process with IIB is the Monitoring events.
These can be thrown into a record/reply setup and then viewed with a web UI.
Monitoring events can be enabled/disabled by admins without requiring code changes.
Again, my point about logstash was that you could write an input that would process and understand the wireformat data output by the Trace node's view of the logical message tree. Then convert that into XML before sending it to elasticsearch or whatever. This is probably difficult.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| timber |
| Posted: Tue Aug 25, 2015 10:26 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
What is this 'wireframe format' of which you speak? The term 'wire format' is sometimes used in reference to the on-the-wire format as opposed to the logical structure. But the Trace node output shows you the logical structure, which is the opposite of the 'wire format'.
I'm trying not to be pedantic, but I'm geniunely puzzled. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Tue Aug 25, 2015 10:29 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| timber wrote: |
What is this 'wireframe format' of which you speak? The term 'wire format' is sometimes used in reference to the on-the-wire format as opposed to the logical structure. But the Trace node output shows you the logical structure, which is the opposite of the 'wire format'.
I'm trying not to be pedantic, but I'm geniunely puzzled. |
Yeah my terminology might not be correct. This wireframe format is what an IBMer told me. The output from a trace node will show you what the format looks like.
${Root} |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 25, 2015 11:21 am Post subject: Re: Logging XML instead of wireframe format |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
I'm interested by the ability to run the very similar framework you describe (particularly the database for that elusive <insert data item> that absolutely was sent to the broker because our application never fails) along with a more lightweight solution for those of my customers that just want to enjoy a reassuring text file with lots of hard to search ids because it makes them feel good. |
If I was going to the MQTC this year I would gladly have sat down and discussed this with you. 
If you want to take this offline then I'll be happy to do it.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue Aug 25, 2015 1:52 pm Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1647
|
| Quote: |
| I was firmly of the opinion the Trace node did pattern matching & didn't support ESQL |
So the technical info here is that the Trace Node can run *any* ESQL expression inbetween the brackets as in: ${xxx}. Substitute xxx for any valid expression. It has had this ability since MQSI v2.0, but it's not been widely publicised! The ${ } part is only so each expression can be parsed out of any plain text surrounding it.
In fact when you look at the usual "patterns" such as ${Root} or {$LocalEnvironment} you will see it has access to the same Correlation Names as the Database node does rather than a Compute node (Root vs InputRoot, OutputRoot etc) apart from 'Database'
For example ${40 + 2} is a valid expression for a Trace node, although not a lot of use by itself 
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 26, 2015 4:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mgk wrote: |
| It has had this ability since MQSI v2.0, but it's not been widely publicised! |
Presumably the technote for this is locked in the bottom drawer of a filing cabinet in a disused basement lavatory under a sign saying "Beware of the Leopard"
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Aug 26, 2015 5:03 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
| mgk wrote: |
| It has had this ability since MQSI v2.0, but it's not been widely publicised! |
Presumably the technote for this is locked in the bottom drawer of a filing cabinet in a disused basement lavatory under a sign saying "Beware of the Leopard" |
I shudder as I recall the cases where a whole logging system was implemented in a TraceNode in V2.1 days. I even have the files somewhere on an old archive. almost as bad a using Log4J which is almost as bad as writitng to stderr & stdout from Java in production flows.
I'm glad the more enlightened of us do things a bit differently these days but back then there wasn't a lot of choice.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| maurito |
| Posted: Wed Aug 26, 2015 5:17 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| Vitor wrote: |
| mgk wrote: |
| It has had this ability since MQSI v2.0, but it's not been widely publicised! |
Presumably the technote for this is locked in the bottom drawer of a filing cabinet in a disused basement lavatory under a sign saying "Beware of the Leopard" |
This is actually documented in the trace node description of the Pattern:
"The data that is to be included in the trace record. Create an ESQL pattern to specify what information to write. If you write the trace record to the local error log, the pattern governs the information that is written in the text of the message number that is selected. If you use the default message catalog, and a number between 3051 and 3099, the pattern information is inserted as &1 in the message text.
You can write plain text, which is copied into the trace record exactly as you have entered it.
You can identify parts of the message to write to the trace record, specifying the full field identifiers enclosed between the characters ${ and }. To record the entire message, specify ${Root}. Other common patterns are ${LocalEnvironment} ${Environment} and ${ExceptionList}. For more information, see Viewing the logical message tree in trace output.
Use the ESQL functions to provide additional information; for example, use the ESQL function CURRENT_DATE to record the date, time, or both, at which the trace record is written.
The pattern shown here includes some of the options that are available. The pattern writes an initial line of text, records two elements of the current message, and adds a simple timestamp:
Message passed through with the following fields:
Store name is ${Body.storedetailselement.storename}
Total sales are ${Body.totalselement.totalsales}
Time is: ${EXTRACT(HOUR FROM CURRENT_TIMESTAMP)}
:${EXTRACT(MINUTE FROM CURRENT_TIMESTAMP)}The resulting trace record is:
Message passed through with the following fields:
Store name is 'SRUCorporation'
Total sales are '34.98'
Time is: 11:19A pattern that contains syntax errors does not prevent a message flow that contains the Trace node from deploying, but the node writes no trace records.
" |
|
| Back to top |
|
|
|
|
