ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Performance Monitoring » How can I check if read-ahead is actually enabled?

Post new topic  Reply to topic Goto page 1, 2, 3  Next
 How can I check if read-ahead is actually enabled? « View previous topic :: View next topic » 
Author Message
chmd
PostPosted: Fri Dec 11, 2015 8:36 am    Post subject: How can I check if read-ahead is actually enabled? Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

Hi,

I wrote a client application that pulls messages from a queue and discards them. I am trying to maximize the throughput (I will do something with the messages later). This client code works, but it is slow (typically 8-10 transactions per second or 160 KiB/s). I suspect that the MQOO_READ_AHEAD option is ignored. How can I check that?

If you think that MQOO_READ_AHEAD is already enabled, how can I increase the throughput? We mesured the performance of the link between the client machine and the queue manager with iperf, and found out that the link is capable of 500MiB/s, which is much more that 160kiB/s, so the network is clearly not the issue. I think that the problem is either in the MQ configuration or in my code. And since the same code, running locally on the queue manager, easily reaches 1500 transactions per second (200 times faster), I am more inclined to believe that MQ is misconfigured for what I am attempting to do.

Here is the channel I pull from:
Code:

     1 : display channel(NOT.REAL.CHANNEL.NAME)
AMQ8414: Display Channel details.
   CHANNEL(NOT.REAL.CHANNEL.NAME)             CHLTYPE(SVRCONN)
   ALTDATE(2015-12-04)                     ALTTIME(13.51.29)
   COMPHDR(NONE)                           COMPMSG(NONE)
   DESCR( )                                DISCINT(0)
   HBINT(300)                              KAINT(AUTO)
   MAXINST(999999999)                      MAXINSTC(999999999)
   MAXMSGL(4194304)                        MCAUSER( )
   MONCHL(QMGR)                            RCVDATA( )
   RCVEXIT( )                              SCYDATA( )
   SCYEXIT( )                              SENDDATA( )
   SENDEXIT( )                             SHARECNV(10)
   SSLCAUTH(REQUIRED)                   
   SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA)
   SSLPEER( )                              TRPTYPE(TCP)


The queue itself has been altered to have DEFREADA to YES.
Code:

     1 : display qlocal(NOT.REAL.QUEUE.NAME)
AMQ8409: Display Queue details.
   QUEUE(NOT.REAL.QUEUE.NAME)                TYPE(QLOCAL)
   ACCTQ(QMGR)                             ALTDATE(2015-12-09)
   ALTTIME(16.16.01)                       BOQNAME( )
   BOTHRESH(0)                             CLUSNL( )
   CLUSTER( )                              CLCHNAME( )
   CLWLPRTY(0)                             CLWLRANK(0)
   CLWLUSEQ(QMGR)                          CRDATE(2015-06-22)
   CRTIME(14.11.03)                        CURDEPTH(0)
   CUSTOM( )                               DEFBIND(OPEN)
   DEFPRTY(0)                              DEFPSIST(NO)
   DEFPRESP(SYNC)                          DEFREADA(YES)
   DEFSOPT(SHARED)                         DEFTYPE(PREDEFINED)
   DESCR( )                                DISTL(NO)
   GET(ENABLED)                            HARDENBO
   INITQ( )                                IPPROCS(1)
   MAXDEPTH(50000)                         MAXMSGL(4194304)
   MONQ(QMGR)                              MSGDLVSQ(PRIORITY)
   NOTRIGGER                               NPMCLASS(NORMAL)
   OPPROCS(8)                              PROCESS( )
   PUT(ENABLED)                            PROPCTL(COMPAT)
   QDEPTHHI(80)                            QDEPTHLO(20)
   QDPHIEV(DISABLED)                       QDPLOEV(DISABLED)
   QDPMAXEV(ENABLED)                       QSVCIEV(NONE)
   QSVCINT(999999999)                      RETINTVL(999999999)
   SCOPE(QMGR)                             SHARE
   STATQ(QMGR)                             TRIGDATA( )
   TRIGDPTH(1)                             TRIGMPRI(0)
   TRIGTYPE(FIRST)                         USAGE(NORMAL)


We made sure that this queue only gets non persistent messages. I had doubts so recently I checked by myself and only found non persistent messages:
Code:

>/opt/mqm/samp/bin/amqsbcg NOT.REAL.QUEUE.NAME NOT.REAL.QMGR.NAME  | grep Persistence
  Priority : 0  Persistence : 0
  Priority : 0  Persistence : 0
  Priority : 0  Persistence : 0
   ...


The queue manager is running MQ 7.5 and runs on a Red Hat server.

The client code runs on another machine (VM on debian). I am using pymqi, compiled against latest version of mqclient (8.0.0.4-WS-MQC-Redist-LinuxX64.tar.gz). It connects over SSL (certificates have been configured and work just fine).

In the client code, the queue is open with the following options:
Code:

                    CMQC.MQOO_INPUT_AS_Q_DEF |
                    CMQC.MQOO_READ_AHEAD


If you need more information, please let me know!

Thank you,
Christophe-Marie
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Dec 11, 2015 2:11 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Christophe Marie, mon ami, tu devrais aussi utiliser dans tes OpenOptions
MQOO et MQGMO la petite perle, oh combien importante: FAIL_IF_QUIESCE...
Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Fri Dec 11, 2015 4:02 pm    Post subject: Re: How can I check if read-ahead is actually enabled? Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

chmd wrote:

Here is the channel I pull from:

Quote:
1 : display channel(NOT.REAL.CHANNEL.NAME)
AMQ8414: Display Channel details.
CHANNEL(NOT.REAL.CHANNEL.NAME) CHLTYPE(SVRCONN)
ALTDATE(2015-12-04) ALTTIME(13.51.29)
COMPHDR(NONE) COMPMSG(NONE)
DESCR( ) DISCINT(0)
HBINT(300) KAINT(AUTO)
MAXINST(999999999) MAXINSTC(999999999)
MAXMSGL(4194304) MCAUSER( )
MONCHL(QMGR) RCVDATA( )
RCVEXIT( ) SCYDATA( )
SCYEXIT( ) SENDDATA( )
SENDEXIT( ) SHARECNV(10)
SSLCAUTH(REQUIRED)
SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA)
SSLPEER( ) TRPTYPE(TCP)




What is the SHARECNV value for the CLNTCONN?

I ask because:
Quote:
High SHARECNV limits have the advantage of reducing queue manager thread usage. However, if many conversations sharing a socket are all busy, there is a possibility of delays as the conversations contend with one another to use the receiving thread. In this situation, a lower SHARECNV value is better.


If the app is faster when run locally (still client-bindings) to a nearby qmgr, and slower when the qmgr is miles or thousands of miles away, then network contention is still a possibility.

Désolé , mon français est nul.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Dec 11, 2015 8:57 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Note that the OP only showed the MQOO options. The READ_AHEAD might also be needed in the MQGMO options for it to happen.

And as advised please change the sharecnv to 1 (best practice on MQ8).
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
tczielke
PostPosted: Sat Dec 12, 2015 7:37 pm    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

When you OPEN the queue with MQOO_READ_AHEAD, there are some conditions that need to be met for it to be enabled -> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q027330_.htm?lang=en

Quote:

•Both the client and remote queue manager must be at WebSphere® MQ Version 7 or later.
•The client application must be compiled and linked against the threaded IBM MQ MQI client libraries.
•The client channel must be using TCP/IP protocol
•The channel must have a non-zero SharingConversations (SHARECNV) setting in both the client and server channel definitions.


When you compiled your client code on Linux, did you link with mqic or mqic_r? Note the requirement that you should be linking against mqic_r.

One simple test you could do is remove the MQOO_READ_AHEAD option and see if your performance changes for the worse. If the performance does not really change, you probably are not using read ahead. I did some tests, and using read ahead made some very significant and noticeable performance improvements (the overall elapsed time went from around 72 seconds to 10 seconds with read ahead enabled).
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
chmd
PostPosted: Mon Dec 14, 2015 4:52 am    Post subject: Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

Thank you for all the answers!
Quote:

Christophe Marie, mon ami, tu devrais aussi utiliser dans tes OpenOptions
MQOO et MQGMO la petite perle, oh combien importante: FAIL_IF_QUIESCE...


My open options are:
Code:

CMQC.MQGMO_NO_WAIT | CMQC.MQGMO_FAIL_IF_QUIESCING


Quote:

The READ_AHEAD might also be needed in the MQGMO options for it to happen.


I just tried, but it does not seem to be a valid get message option (as in: the code crashes).

Quote:

If the app is faster when run locally (still client-bindings) to a nearby qmgr, and slower when the qmgr is miles or thousands of miles away, then network contention is still a possibility.


I agree. The machines are physically very close to each other, so at first I did not consider the possibility. Then when running the code on the queue manager vs on this machine we had this problem, that was my first clue. So to be sure, we actually measured: we installed iperf on the queue manager and confirmed that we were geting 500MiB/s (!) to this very client machine.

Quote:

And as advised please change the sharecnv to 1 (best practice on MQ8).


I did as you asked. No noticeable change. Note that when running on the queue manager, I am using a different channel with no ssl encryption. The value of SHARECNV is 10 on that one. I also considered that the slowness could be blamed on the encryption, but the messages coming to the queue are coming through a different SSL encrypted channel, which has no performance problem whatsoever. It could be the cipher, though (I will check).

Quote:

When you compiled your client code on Linux, did you link with mqic or mqic_r? Note the requirement that you should be linking against mqic_r.


It is definitely linked to mqic_r. When I alter LD_LIBRARY_PATH and remove /opt/mqm/lib64 from there, the program crashes with
Code:

ImportError: libmqic_r.so: cannot open shared object file: No such file or directory


As for the other criterias: the qmgr version is 7.5, the client lib is 8.0.0.4-WS-MQC-Redist-LinuxX64; The connection is TCP; SHARECNV is non-zero on the queue manager.

Quote:

What is the SHARECNV value for the CLNTCONN?


Good question! I have no idea how to check that. I am using pymqi and I found no information in the source code of the lib telling me if there was a default setting in the lib. Do you know how does one specify that in a C/C++ client? The python binding is no more than a fine wrapper around the C API, so I should be able to derive from a sample.[/list]
Back to top
View user's profile Send private message
tczielke
PostPosted: Mon Dec 14, 2015 6:14 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

I am not sure if you have the MQ Client installed on your client server. It sounds like you might not since you mention the redistributable client. I think would need the MQ Client software installed to run a client trace (i.e. strmqtrc). If you do have the MQ Client, it looks like you can validate if read ahead is enabled or not by doing the following tracing on the MQ Client side:

#turn tracing on for the client application myprog
strmqtrc -t all -p myprog

Run your program. Your traces be default on Linux will go to /var/mqm/trace. You need to format your traces with dspmqtrc. In the formatted trace for your client program, search for the text "ahead".

This is what I saw in the trace for a queue I was opening when MQOO_READ_AHEAD was set:

Code:

 07:29:04.101929     3101.1      RSESS:000001      Requested read-ahead=0x00100000
 07:29:04.101934     3101.1      RSESS:000001      Requested Maximum Q size=100KB


This is what I saw in the trace for a queue I was opening when MQOO_READ_AHEAD was NOT set and the queue had DEFREADA(NO):

Code:

 07:36:39.597749    14101.1      RSESS:000001      Read-ahead taken from object def [DefReadAhead=0x00000000]
 07:36:39.597755    14101.1      RSESS:000001      Requested read-ahead=0x00080000
 07:36:39.597760    14101.1      RSESS:000001      Requested Maximum Q size=100KB


Here are the hex values for MQOO_ options of read ahead:
0x00080000 MQOO_NO_READ_AHEAD
0x00100000 MQOO_READ_AHEAD

You can see in the above examples that read ahead resolved to 0x00100000 MQOO_READ_AHEAD when read ahead was set in the open options and it resolved to 0x00080000 MQOO_NO_READ_AHEAD when read ahead was not set in the open options and the queue had DEFREADA(NO).
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
tczielke
PostPosted: Mon Dec 14, 2015 6:40 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

You can also use the Application Activity Trace to find the negotiated SharingConversations for your channel instance. You need to have TraceLevel set to high for the activity trace. After you get your activity trace (you could use amqsact), if you look for the MQCD Structure for your CONNX of your channel and look near the bottom, you will see something like the following:

Code:

  00000750:  0000 0032 0000 0011 FFFF FFFD 0000 000A  '...2............' <- SharingConversations starts at offset 75C for 4 bytes and is 10 = 0x0A
  00000760:  0000 0000 3B9A C9FF 3B9A C9FF 0000 0000  '....;...;.......'
  00000770:  0000 0001 0000 1388 0000 0002 0000 0000  '................'
  00000780:  2020 2020 2020 2020 2020 2020 2020 2020  '                ' <- CertificateLabel starts at offset 780 (NOTE: edited and was initially incorrect at offset 770)
  00000790:  2020 2020 2020 2020 2020 2020 2020 2020  '                '
  000007A0:  2020 2020 2020 2020 2020 2020 2020 2020  '                '
  000007B0:  2020 2020 2020 2020 2020 2020 2020 2020  '                '


CertificateLabel should be at the bottom of the MQCD structure. SharingConversations should be 3 lines above the CertificateLabel and the last 4 byte on that third line.
_________________
Working with MQ since 2010.


Last edited by tczielke on Mon Dec 14, 2015 8:31 am; edited 1 time in total
Back to top
View user's profile Send private message
chmd
PostPosted: Mon Dec 14, 2015 7:22 am    Post subject: Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

I do have the strmqtrc program on my client machine.

I tried to run
Code:
strmqtrc -t all -p mq2kafka #does not do anything with kafka yet


However, /var/mqm/trace was not populated after running my program.Since this is a python program calling the lib, I also tried
Code:
strmqtrc -t all -p python

with no luck.

However, we did confirm that read_ahead *can* work with this installation, by running "q" on the client machine. We created a test queue TEST.Q with all default options, and a test channel with all default options too.

Synchronous put:
Code:

time ./q -oTEST.Q -lmqic -M'#!1000/-5000'
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May  2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.

real    0m21.905s
user    0m0.035s
sys     0m0.033s


Synchronous get:
Code:

 time ./q -ITEST.Q -lmqic -q
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May  2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
No more messages.

real    0m31.980s
user    0m0.038s
sys     0m0.036s


Asynchronous put:
Code:

time ./q -af -oTEST.Q -lmqic -M'#!1000/-5000'
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May  2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.

real    0m0.586s
user    0m0.003s
sys     0m0.005s


Asynchronous get:
Code:

 time ./q -aa -ITEST.Q -lmqic -q
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May  2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
No more messages.

real    0m1.985s
user    0m0.014s
sys     0m0.010s


Now that we have established that the client installation and the server installation can work in read_ahead mode, I am rewriting my python code to do exactly the same as q, on this very test queue, with this very channel. I will try to narrow it down to very simple code that I can hopefully paste here.
Back to top
View user's profile Send private message
tczielke
PostPosted: Mon Dec 14, 2015 7:25 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

If you do the following, it will trace any client program:

Code:

strmqtrc -t all

_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
chmd
PostPosted: Mon Dec 14, 2015 8:27 am    Post subject: Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

All right!

So it turns out that the traces do not go to /var/mqm. In my case, they go to a directory IBM/MQ/data/trace that is created in the current directory.

I cleaned it and rerun the program, then formatted all the trace files. Only one of them showed infos:
Code:

 16:09:32.440594    20302.1      RSESS:000001      Read-ahead taken from object def [DefReadAhead=0x00000000]
 16:09:32.440597    20302.1      RSESS:000001      Requested read-ahead=0x00080000
 16:09:32.440599    20302.1      RSESS:000001      Requested Maximum Q size=100KB


Then all the subsequent matches are
Code:

 16:09:32.851279    20302.1      RSESS:000001      Requested read-ahead=0x00100000
 16:09:32.851282    20302.1      RSESS:000001      Requested Maximum Q size=100KB


Something weird is going on. I will make sure I only pull from one queue, because right now my program is a bit more complicated than necessary.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Dec 14, 2015 9:17 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

This channel isn't being used by other apps, is it? I'd recommend using test channels, as well as test queues, while you do your debugging.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
chmd
PostPosted: Mon Dec 14, 2015 1:14 pm    Post subject: Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

No, it is a brand new channel specially created for that purpose.
Back to top
View user's profile Send private message
tczielke
PostPosted: Mon Dec 14, 2015 1:28 pm    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

This manual document also lists the MQGMO options that are not allowed for read ahead -> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q026320_.htm?lang=en

One that caught my eye was this one:

MQGMO_SYNCPOINT

"If these options are specified on the first MQGET call then read ahead is disabled. If these options are specified on a subsequent MQGET call a reason code MQRC_OPTIONS_ERROR is returned."

You had mentioned earlier about transactions. Are you using MQGMO_SYNCPOINT for your GETs?
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
chmd
PostPosted: Tue Dec 15, 2015 2:21 am    Post subject: Reply with quote

Novice

Joined: 11 Dec 2015
Posts: 18

No, my get options are
Code:

    gmo.Options = CMQC.MQGMO_NO_WAIT | \
                  CMQC.MQGMO_FAIL_IF_QUIESCING
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2, 3  Next Page 1 of 3

MQSeries.net Forum Index » IBM MQ Performance Monitoring » How can I check if read-ahead is actually enabled?
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.