| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| MQ vs HTTP or vice versa |
« View previous topic :: View next topic » |
| Author |
Message
|
| zpat |
 Posted: Sun Apr 12, 2015 12:13 pm Post subject: |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
You cannot include http in a coordinated transaction.
If you look at your design and ask yourself for every stage - what happens if the system crashes at this moment in time - you will inevitably come to the conclusion that data must be lost or duplicated.
For example - you send an update request via http. The update succeds at the other end, but the application there fails just before sending the response. What happens? Chances are you will have to repeat the request and risk duplicating data (processing a trade twice etc).
Detection of a duplicate can provide one way out of this conumdrum if your protocol is not transactionally safe. You can try to program your way out of this with sequence numbers and so on, when you have done this, you will have re-invented a wheel that IBM can do much better than you.
Of course there is http/r - for reliable http - but no-one uses it!
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
 |
| rekarm01 |
| Posted: Sun Apr 12, 2015 6:26 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
For a completely different opinion, see "Nobody Needs Reliable Messaging". Most of the same points it makes about WS-RM can also apply to MQ.
| zpat wrote: |
| If you do not use MQ as your transport - then you are happy to lose or duplicate data. That is the fact of the matter. |
Maybe not happy, but if the sending app can handle lost messages, and the receiving app can handle duplicate messages, then reliable messaging is not that essential.
| zpat wrote: |
| in fact non-persitent messages have barely measureable latency (over the underlying network latency). |
Except that non-persistent messages don't have the same "once-and-only-once" assured delivery. (Neither do MQ clients.) So the business layer would still have to handle the end-to-end reliability anyway.
| zpat wrote: |
| Simple - you don't use http clients. You don't use http servers. |
Not everyone has that luxury, particularly when dealing with legacy or third-party apps, or external business partners.
| zpat wrote: |
| Http is simply inferior is every way and there is no need to use it. |
HTTP has its place. It runs on platforms where MQ doesn't, it's non-proprietary, cheaper, simpler, and easier to use and administer. Applications that assume HTTP is more reliable than it is can run into problems, but that's true for MQ too.
| fjb_saper wrote: |
| Not quite. If you are dealing with a one way message, you still need to wait for your 204 status code... MQ put and forget... (+ for mq in my book). You will get a queue depth until App1 catches up... |
The server can send an HTTP 202 "Accepted" (or HTTP 503 "Service Unavailable") without waiting for App2 to process the message. If the MQ client calls MQPUT and it returns with a MQRC_CONNECTION_BROKEN, did the MQPUT succeed or fail? If the MQPUT call doesn't return at all, and App1 just blocks indefinitely, then what? Even if the MQPUT succeeds with MQRC_NONE, that just means the message was put on a queue; it doesn't mean that App2 will ever get it.
| zpat wrote: |
| You cannot include http in a coordinated transaction. |
You don't always need to. But when you do, there are standards for that, such as WS-Coordination or WS-Transaction.
| zpat wrote: |
| For example - you send an update request via http. The update succeds at the other end, but the application there fails just before sending the response. What happens? Chances are you will have to repeat the request and risk duplicating data (processing a trade twice etc). |
If you send an update request via MQ, you either have to MQPUT non-transactionally, or commit the transaction, before any application can get the update request and act on it. Any sort of response message would be part of a separate transaction. MQ cannot guarantee that the receiving app will get the request, or that the requesting app will get the response. What happens then?
| zpat wrote: |
| You can try to program your way out of this with sequence numbers and so on, when you have done this, you will have re-invented a wheel that IBM can do much better than you. |
Reliability at a lower layer cannot guarantee reliability at a higher layer. If end-to-end reliability is important to the business, then it has to be implemented at the business layer. Lower-layer reliability can be a bonus, but it's not enough, by itself. |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Apr 12, 2015 9:50 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
MQ just works, it also doesn't need load-balancing hardware as multiple listeners can read the same queue. Pull type load balancing is far superior to push type load-balancing. The queue will buffer any peak loads which otherwise can overwhelm http based applications.
I've been working in IT for a long time, before the dawn of MQ. Application to application communications used to be complex (APPC etc), then it became simple with MQ, and now it's become complex and unreliable with http.
Yes, you can design the app around it, but why should you? Also you are assuming application developers understand all this - my experience is that they understand very little these days and perform almost no "what if" type testing (e.g. of crash scenarios).
Combining the limitations of http with the new breed of inexperienced developers has lead to unreliable integration, I see it every day and spend a lot of my time dealing with the consequences that could have simply been avoided by using MQ.
The same applies to using file transfer over the crappy RFE protocols. If you don't understand why IBM developed MQ and why so many large companies use it for good reasons - then I can't help you. The IT world is entering a new dark age, when people don't understand ACID transactions.
I just hope that it isn't my money that gets lost (duplication not so bad  )
One example of where synchronous protocols are a disaster- take a broker message flow that calls a service. In the naive http world - it makes a web service request. Now it has to sit and wait. The flow is blocked (although using memory). The flow has to decide when to time out etc and how and when to retry and all the other messy logic (rarely tested properly).
Let's assume the backend is down. All the flows can then get blocked (no matter how many instances) and no other workload can flow - even ones that don't need that particular backend because all the flows can be waiting for the broken service). Http requests into the broker now start to timeout as well!
The sensible approach is to use MQ to send the request asynchronously, then TERMINATE the flow. It does not have to wait, it does not block anything. The response message (when it comes) is processed by another flow. no memory is wasted. This design is massively scaleable and response times can be just as fast as http, plus they are transactionally recovered for message flow backout (and against system failure if also persistent).
Such a elegant stateless design is familar to "old hands" using CICS transactions. It only requires a few flow instances to have massive capacity. Memory requirements are low, and it used to be that if did it any other way you would get fired!
Contrast to the naive http approach when developers ask to have thousands of flow instances (over multiple EGs) in a desparate attempt to make this memory resident, blocking design cope with large peak workloads. Requestors to this http service will time out when everything is blocked and they have to implement their own retry logic. It's hard to monitor or manage as http is invisible (no queue depth to monitor).
When the excessive memory use causes the EG to abend, then everything is lost of course and then they blame message broker for being rubbish! Whereas the apps that took my MQ advice just work every day without issues.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Sun Apr 12, 2015 10:09 pm; edited 3 times in total |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Apr 12, 2015 10:06 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
MQ just works, it also doesn't need load-balancing hardware as multiple listeners can read the same queue. Pull type load balancing is far superior to push type load-balancing. The queue will buffer any peak loads which otherwise can overwhelm http based applications.
I've been working in IT for a long time, before the dawn of MQ. Application to application communications used to be complex (APPC etc), then it became simple with MQ, and now it's become complex and unreliable with http.
Yes, you can design the app around it, but why should you? Also you are assuming application developers understand all this - my experience is that they understand very little these days and perform almost no "what if" type testing (e.g. of crash scenarios).
Combining the limitations of http with the new breed of inexperienced developers has lead to unreliable integration, I see it every day and spend a lot of my time dealing with the consequences that could have simply been avoided by using MQ.
The same applies to using file transfer over the crappy RFE protocols. If you don't understand why IBM developed MQ and why so many large companies use it for good reasons - then I can't help you. The IT world is entering a new dark age, when people don't understand ACID transactions.
I just hope that it isn't my money that gets lost (duplication not so bad ) |
Like Zpat, I've been in IT a long time.
I wrote a paper on the use of Queues in IT as an undergrad in 1974.
I've been involved with OS development where you don't get very far without things like command queues.
I even wrote my own distributed Queue Manager in the 1980's.
It just makes sense to use queues and let the underlying Queue Subsystem manage getting the data to the endpoint.
Biased towards MQ? Yep I sure am. It just works!
but what do I know eh? I'm just an old  and probably should be put out to grass. 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Apr 13, 2015 5:06 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
Can a Moderator (that knows what they are doing with splitting threads) split this thread where it forks from the original subject to this valuable discussion of MQ versus HTTP?
Maybe give it a new title (MQ vs HTTP) and move it to the General Forum.
zpat, I appreciate your in depth responses. Sometimes I get the feeling everyone's looking at me and thinking "There's the MQ guy think MQ is the answer to everything, of course he says use MQ".
I wish there was an official DeveloperWorks article or session at MQ Tech Conference 2015 that laid out all the pros/cons of MQ versus HTTP. Because no matter how much good info there is in this thread so far and hopefully yet to come, in the end its just a bunch of internet starngers spouting off. I respect the opinions of a "zpat" or a "rekarm01" or a "smdavies99", but anyone I refer to this thread doesn't "know" you guys like I do. An official write up would be useful.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Apr 13, 2015 5:29 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
So split.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| akil |
| Posted: Mon Apr 13, 2015 10:38 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
I don't mean to tick anyone off, but to avoid the 'stuck' problem, one could use HTTPAsyncRequest instead of HTTPRequest, that's provided by the broker.
_________________
Regards |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
