ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Performance tuning for Network Latency

Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next
 Performance tuning for Network Latency « View previous topic :: View next topic » 
Author Message
fjb_saper
PostPosted: Thu Jul 04, 2019 10:58 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Bandwidth utilization has very little to do with MQ. A lot more is dictated by TCP/IP. Especially when long distances, net lag of 140ms or thereabouts and some packet loss are involved.

At this point TCP/IP bandwidth utilization becomes quite poor and you'd be well advised to look for alternative solutions.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Thu Jul 04, 2019 11:35 pm    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

saurabh25281 wrote:
...We could have packed a lot more of messages on our network, and improved our performance, if such feature was available with MQ.

TBH, this is a very basic feature that one expect from a mature product like MQ.

That's like expecting your postman to 'compress' more letters into his sack so more can be delivered in one round - but guess what? The more letters, the heavier the sack, and the heavier the sack, the slower the postman!

It bears repeating again, no amount of tuning of MQ will mitigate poor underlying transport issues.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Jul 05, 2019 12:23 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

saurabh25281 wrote:
This is evidence that our network bandwidth is not utilized fully. We could have packed a lot more of messages on our network, and improved our performance, if such feature was available with MQ.

TBH, this is a very basic feature that one expect from a mature product like MQ.

Can you just confirm whether you are talking about client channels or sender/receiver channels here?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Fri Jul 05, 2019 4:41 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

saurabh25281 wrote:
The DCs in question are located across continents (4600 miles apart) and the ping represents realistic response times, looking at sources in the public domain I would assume. Correct me If I am wrong on this.


That's a lot further apart than most of us are used to. DR DCs are more commonly are as close as possible given the disasters they're protecting against (for example, 300 miles away the other side of a mountain range protects from flooding; repeat for all other disasters in your list after "flooding".

But yes, that's a good response for 4,600 miles. Explain to your business users that if you go from using the local DC to the DC 4,600 miles away because there's been a disaster, SLA will be impacted. 4,600 miles is a long way for the electricity to run.

saurabh25281 wrote:
I should be able to choose the buffer size as per my application needs. We can do that for TCP buffer for Sender/Receiver channels.


But the MCA determines how much buffer to use. In this instance, your application would need that code.

saurabh25281 wrote:
We are the application owner and know the average message size and hence can approximate the buffer size we want.


No you're not. The point of decoupled messaging is that 2 separate applications are taking to each other so you're not the owner of the sending application and have no control over what they're doing. If you do own both applications, remove MQ from the picture and use a replicated database!

saurabh25281 wrote:
We can decide an acceptable number of messages which does not overrun the RAM by any measure. And the messages in RAM would still be there in Disk until there is a commit request from the client application, so there is no risk of loosing messages.


You're again assuming message size, and if you pull an acceptable number of messages (like the 2,000 in my example) then my comments about the 2001st message apply.

saurabh25281 wrote:
It's our responsibility on how we handle failure. Let the application decide for themselves.


MQ controls this with MQGET. Message control (especially for persistent messages) is a feature of MQ. Are you saying IBM can abdicate this responsibility for this new verb?

saurabh25281 wrote:
Like I mentioned earlier, the application can decide how much buffer it chooses and what is the optimum no. of messages that generates a good enough throughput and acceptable response time. Probable batch size can be 50 for e.g. for my application needs, same as Sender channel batch size.


So you want read ahead with this, so the application can be processing one filled buffer while MQ is filling the next one? Otherwise you're still waiting for the buffer to fill with all that latency.

saurabh25281 wrote:
Doesn't the batch feature apply to Sender/Receiver channels? Do we provide a lower QoS for persistent messages in the case of Sender/Receiver channels?


Not a valid comparison. The MCAs on a sender/receiver channel are in an asynchronous partnership controlled by handshakes. The application client connection is synchronous.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Jul 05, 2019 4:42 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

hughson wrote:
Can you just confirm whether you are talking about client channels or sender/receiver channels here?


The OP is talking about it taking longer to do an MQGET from his DR queue manager than the local one. I'm thinking client.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
saurabh25281
PostPosted: Fri Jul 05, 2019 2:18 pm    Post subject: Reply with quote

Centurion

Joined: 05 Nov 2006
Posts: 107
Location: Bangalore

Small correction, I never mentioned that the DCs are used for DR.

Quote:

vitor wrote:
No you're not. The point of decoupled messaging is that 2 separate applications are taking to each other so you're not the owner of the sending application and have no control over what they're doing. If you do own both applications, remove MQ from the picture and use a replicated database!

The sending application is MQ Queue Manager and it publishes Configuration events onto it local Queue, which is read by a Java based tool, that is seperated by such distance. We dont own Qmgr app, but we know the approx size of Configuration event messages.

Quote:

vitor wrote:
MQ controls this with MQGET. Message control (especially for persistent messages) is a feature of MQ. Are you saying IBM can abdicate this responsibility for this new verb?

No I am not suggesting abdication of responsibility. But there can be many solution for e.g. responding back with a rollback with the msgID that caused the problem, or, there can be GMO that send such error prone messages to designated backout queues or DLQs. I would assume rollback not to take much time for its just a question of clearing the buffer or not.

Quote:

vitor wrote:
So you want read ahead with this, so the application can be processing one filled buffer while MQ is filling the next one? Otherwise you're still waiting for the buffer to fill with all that latency.

I did not think of it as a client side buffer, but server side only, that will be cleared once there is a commit message from the client.

Quote:

exerc wrote:
That's like expecting your postman to 'compress' more letters into his sack so more can be delivered in one round - but guess what? The more letters, the heavier the sack, and the heavier the sack, the slower the postman!

Didn't the Ping demonstrate, that the heavier sack did not slow the postman.

Quote:

hughson wrote:
Can you just confirm whether you are talking about client channels or sender/receiver channels here?

SVRCONNs to be precise.

Quote:

fjb_saper wrote:
Bandwidth utilization has very little to do with MQ. A lot more is dictated by TCP/IP. Especially when long distances, net lag of 140ms or thereabouts and some packet loss are involved.

At this point TCP/IP bandwidth utilization becomes quite poor and you'd be well advised to look for alternative solutions.

My messages are 250 bytes approx.
Ping response shows 655,00 bytes takes 128 ms.
For 200 such messages, size would have been 500,00 bytes.

What you are saying that with even such feature, delivering 200 such msgs will take around 30-40 secs or more although ping responds within 128 ms? I mention 30-40 secs for comparing it with fetching each message individually.

Regards
Saurabh
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
hughson
PostPosted: Fri Jul 05, 2019 4:48 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

saurabh25281 wrote:
hughson wrote:
Can you just confirm whether you are talking about client channels or sender/receiver channels here?

SVRCONNs to be precise.

Indeed - so I would reiterate my earlier comment - have you looked into Read Ahead? Or are you only using persistent messages in which case it doesn't help?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
saurabh25281
PostPosted: Sat Jul 06, 2019 10:51 pm    Post subject: Reply with quote

Centurion

Joined: 05 Nov 2006
Posts: 107
Location: Bangalore

hughson wrote:
saurabh25281 wrote:
hughson wrote:
Can you just confirm whether you are talking about client channels or sender/receiver channels here?

SVRCONNs to be precise.

Indeed - so I would reiterate my earlier comment - have you looked into Read Ahead? Or are you only using persistent messages in which case it doesn't help?

We are using persistent messages.
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
bruce2359
PostPosted: Sun Jul 07, 2019 1:52 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

A quick reminder of how client-binding differ from server-bindings apps.

For server-bindings apps, MQI calls are shipped cross-memory (within the same o/s instance). Memory speeds are much, much faster than network speeds. You can research google for your RAM type vs. clock-speed vs memory bandwidth.

For client-bindings apps, each MQI call (MQCONN, MQOPEN, MQPUT, ...) is shipped cross-network to the server, which executes the call, then returns ReasonCode/CompletionCode to the client. For MQPUTs and MQGETs, the call also ships your message content. In short, two network flows per MQI call.

If your client-binding app runs very close to the server, response time improves.
Move the server 5,000 miles away, response time degrades.

If the app you describe is the one and only instance of the app, I'd strongly recommend that you install MQ, and create a qmgr, on the client platform, and let MQ qmgr-to-qmgr channels transmit your messages.

You might want to do this as a proof-of-concept.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
saurabh25281
PostPosted: Sun Jul 07, 2019 10:09 pm    Post subject: Reply with quote

Centurion

Joined: 05 Nov 2006
Posts: 107
Location: Bangalore

bruce2359 wrote:

If the app you describe is the one and only instance of the app, I'd strongly recommend that you install MQ, and create a qmgr, on the client platform, and let MQ qmgr-to-qmgr channels transmit your messages.

It would be too expensive to have a MQ server just for transmitting configuration events for monitoring and auditing purpose.

We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help?

Regards
Saurabh
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
exerk
PostPosted: Sun Jul 07, 2019 11:01 pm    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

saurabh25281 wrote:
...We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help?

More threads waiting on the same slow network? What do you think?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
saurabh25281
PostPosted: Mon Jul 08, 2019 12:19 am    Post subject: Reply with quote

Centurion

Joined: 05 Nov 2006
Posts: 107
Location: Bangalore

exerk wrote:
saurabh25281 wrote:
...We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help?

More threads waiting on the same slow network? What do you think?

We cannot beat the latency, but can improve the throughput by parallelism. Does every thread performing a MQGET, wait for the previous thread to complete?
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
exerk
PostPosted: Mon Jul 08, 2019 12:57 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

saurabh25281 wrote:
...We cannot beat the latency, but can improve the throughput by parallelism...

A good idea...

saurabh25281 wrote:
...Does every thread performing a MQGET, wait for the previous thread to complete?

Depends on how you code it...

Think of your messages and threads as the bulbs of an hour-glass, and the network as the thin neck connecting those bulbs - even if you increase the size of the bulbs, if the neck remains the same size the rate of flow through the neck will not increase. I'm sure parallelism will bring some benefit, but not necessarily the amount you seek.

Have you considered having a 'local' application read off the messages to file, and transfer the file?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Jul 08, 2019 4:22 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

exerk wrote:
...Have you considered having a 'local' application read off the messages to file, and transfer the file?

Or have a local app that combines all the data into into one MQ message, and the MQ client app does one MQGET across the n/w. This will avoid the two-way TCP handshake for each message, and avoid sending the MQMD and MQI control structures for each message (a few hundred bytes).
_________________
Glenn
Back to top
View user's profile Send private message
saurabh25281
PostPosted: Wed Jul 10, 2019 8:19 am    Post subject: Reply with quote

Centurion

Joined: 05 Nov 2006
Posts: 107
Location: Bangalore

gbaddeley wrote:
exerk wrote:
...Have you considered having a 'local' application read off the messages to file, and transfer the file?

Or have a local app that combines all the data into into one MQ message, and the MQ client app does one MQGET across the n/w. This will avoid the two-way TCP handshake for each message, and avoid sending the MQMD and MQI control structures for each message (a few hundred bytes).

We are initially trying to multi-thread our application to read multiple messages concurrently and see how much improvement it brings.
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Performance tuning for Network Latency
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.