| Author |
Message
|
| wmqstankela |
 Posted: Thu Nov 22, 2018 11:27 am Post subject: HTTP Reply node problem |
 |
|
Voyager
Joined: 29 Feb 2016
Posts: 94
|
Hi all,
I have rest web service that has timeout of 90s. This service invoke 3rd party service and responce back to client. Sometimes my service does not send response to client in that 90s and then send timeout message, but in log I saw just before http reply that it takes 30s total. It looks like sometimes http reply node can't send back response message to client.
Is someone has any idea how to solve this kind of problem?
Thanks in advance |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Nov 26, 2018 5:50 am Post subject: Re: HTTP Reply node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmqstankela wrote: |
| It looks like sometimes http reply node can't send back response message to client. |
So what's stopping it? Does the HTTP Reply node get an ack from the 3rd party service, an HTTP 200 or some other issue?
| wmqstankela wrote: |
| Is someone has any idea how to solve this kind of problem? |
Figure out what's stopping the reply.
You talk about "log" and if you don't mean "user trace" that's your next step. If the HTTP Reply node replies after 30 seconds, what's happening for the next 60?
I see 3 likely problems:
- your code is hosing up the session id and sometimes replies to the wrong client (especially if you have multiple instances and store session ids in any kind of shared memory)
- something like a firewall is terminating the connection, the reply is bouncing off that and the HTTP Reply node is not coded to respond to the HTTP failure code
- something like a load balancer is accepting the reply and then either eating it or replying to the wrong client.
This does not preclude your problem actually being a 4th, less likely scenario.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmqstankela |
| Posted: Tue Nov 27, 2018 6:03 am Post subject: |
|
|
Voyager
Joined: 29 Feb 2016
Posts: 94
|
Thanks Vitor for your answer!
HTTP Request get proper response with http 200 code. Log that I am able to see is from DB. I log message in queue and another application store message from queue to db. I log message few time during flow. Last log is just before http reply node, and in that log I see good response from 3rd party service.
I have haproxy load balancer in front of an IIB, and its timeout is bigger than timeout on http input node of my flow.
The whole situation is a little bit strange, because same request to IIB one time respond with http 504 timeout, and the next time respond with code 200. It looks like some network issue, maybe firewall like you said.
Where in message tree is stored session id? I thought that http reply node reads info, about where to send reply from LocalEnvironment.Destination.HTTP.RequestIdentifier.
I would be very thankful if you have another idea how to troubleshoot this problem. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 27, 2018 7:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmqstankela wrote: |
| The whole situation is a little bit strange, because same request to IIB one time respond with http 504 timeout, and the next time respond with code 200. It looks like some network issue, maybe firewall like you said. |
504 isn't a timeout, it's a flat out error - "Gateway error" according to Mr. Google. Ask your network people what's wrong.
| wmqstankela wrote: |
| Where in message tree is stored session id? I thought that http reply node reads info, about where to send reply from LocalEnvironment.Destination.HTTP.RequestIdentifier. |
It is.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmqstankela |
| Posted: Thu Nov 29, 2018 6:18 am Post subject: |
|
|
Voyager
Joined: 29 Feb 2016
Posts: 94
|
This is default IIB response when service does not respond in time
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>504 Gateway Timeout</title>
</head>
<body>
<h1>504 Gateway Timeout</h1>
<p>
Integration node TESTNODE did not provide a response within the specified time interval (5 seconds). The message reached the main flow, but timed out during processing.
</p>
<hr>
<i>
IBM Integration Bus v10.0.0.5
</i>
</body>
</html>
So 504 gateway timeout is generate on IIB |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 29, 2018 7:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmqstankela wrote: |
| This is default IIB response when service does not respond in time |
I'm confused. How does this tie up with:
| wmqstankela wrote: |
| because same request to IIB one time respond with http 504 timeout |
That's not a request to IIB, that's a request from IIB.
Likewise:
| wmqstankela wrote: |
| It looks like sometimes http reply node can't send back response message to client. |
You need to be very, very clear on exactly what your topology is here. Is it your rest service that's in TESTNODE?
| wmqstankela wrote: |
| So 504 gateway timeout is generate on IIB |
And that HTML is from an IIB log or the calling client? How have you determined that IIB is generating that message and not that it's propagating the 504 message the HTTPRequest node got when it called the 3rd party service?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmqstankela |
| Posted: Thu Nov 29, 2018 8:03 am Post subject: |
|
|
Voyager
Joined: 29 Feb 2016
Posts: 94
|
Yes, my service is on TESTNODE and this is response from my service to the client. This message was generated on IIB.
I simplified my scenario, just for troubleshooting. Whitout haproxy, just calling my service directly.
Everything is pointing on network issue. I need to consult with network department.
Thanks a lot Vitor for your answers! |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 29, 2018 8:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmqstankela wrote: |
| Yes, my service is on TESTNODE and this is response from my service to the client. This message was generated on IIB. |
No - passed back by IIB not generated by IIB.
| wmqstankela wrote: |
| Everything is pointing on network issue. I need to consult with network department. |
Like I said, 504 is a hard error from a network component.
| wmqstankela wrote: |
| Thanks a lot Vitor for your answers! |
Welcome I'm sure
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mgk |
| Posted: Fri Nov 30, 2018 1:26 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1639
|
| Quote: |
| No - passed back by IIB not generated by IIB. |
Actually I can confirm that this 504 message IS generated by IIB, in certain timeout cases, such as the main flow not sending a reply before the timeout period expires (which seems like this case) or the "timeout" terminal of the HTTPInput node being connected, but a reply not being sent by the flow during the timeout processing which is essentially "a timeout occurring during timeout processing". The message is a fallback to make sure that the client at least gets a reply in either case and is not left hanging around. I would start looking at any timeout processing your flow is doing... Also, is it possible your load balancer is getting confused in some way?
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| wmqstankela |
| Posted: Fri Nov 30, 2018 2:24 am Post subject: |
|
|
Voyager
Joined: 29 Feb 2016
Posts: 94
|
Thanks mgk. Yes, this message is generated when flow does not respond in time (maximum client wait time in HTTP Input properties).
If it is problem with load balancer then this 504 message also will not propagate to client. My opinion is that some network issue does not allow http reply node to send back message to load balancer and when message flow timeout is reached http input node send this 504 error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Nov 30, 2018 5:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mgk wrote: |
| Quote: |
| No - passed back by IIB not generated by IIB. |
Actually I can confirm that this 504 message IS generated by IIB, in certain timeout cases, such as the main flow not sending a reply before the timeout period expires (which seems like this case) or the "timeout" terminal of the HTTPInput node being connected, but a reply not being sent by the flow during the timeout processing which is essentially "a timeout occurring during timeout processing". The message is a fallback to make sure that the client at least gets a reply in either case and is not left hanging around. I would start looking at any timeout processing your flow is doing... Also, is it possible your load balancer is getting confused in some way? |
I'm enlightened 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
