MQSeries.net :: View topic - AMQ6037: WebSphere MQ was unable to obtain enough storage

Jhap · Posted: Wed Mar 17, 2010 3:28 am Post subject:

Hello !

We are facing with following problem. When application which puts and gets messages from queue runs with 1-3 instances everything is OK, but when instances increase above 3 we get error messages like below.

This is error information from application side:

RC2009(2009) Explanation:Connection to the queue manager has been lost. This can occur because the queue manager has ended. If the call is an MQGET call with the GMWT option, the wait has been canceled. All connection and object handles are now invalid.For MQ client applications, it is possible that the call did complete successfully, even though this reason code is returned with a CMPCOD of CCFAIL. Completion Code: CCFAIL

Programmer Response: Applications can attempt to reconnect to the queue manager by issuing the MQCONN or MQCONNX call. It may be necessary to poll until a successful response is received.

Any uncommitted changes in a unit of work should be backed out. A unit of work that is coordinated by the queue manager is backed out automatically.

This is from qmgrs error log:

03/16/10 12:39:08 AMQ6037: WebSphere MQ was unable to obtain enough storage.

EXPLANATION:
The product is unable to obtain enough storage. The product's error recording routine may have been called.

ACTION:
Stop the product and restart it. If this does not resolve the problem see if a problem has been recorded. If a problem has been recorded, use the standard facilities supplied with your system to record the problem identifier, and to save the generated output files. Contact your IBM support center. Do not discard these files until the problem has been resolved.amqrsrva.c : 506

From IBM site:
2071 AMQ6037 WebSphere MQ was unable to obtain enough storage Technote (troubleshooting) Problem(Abstract)

You are running WebSphereÂ® MQ V5.3 on AIXÂ® V5.1 and you

intermittently receive reason code 2071 ( 0x00000817, MQRC

STORAGE NOT AVAILABLE ) when getting messages from a local queue.

Symptom 2071 0x00000817 MQRC STORAGE NOT AVAILABLE

Cause

Applications requesting more buffers than maximum available heap size. Resolving the problem Tune the queue manager to use fewer threads per agent, so that each agent will stay within its limits.

You can limit the number of threads in each agent process to 4 by adding the following parameters to the /var/mqm/qmgrs/<qmgrname>/qm.ini file. The applications should then be able to specify 40 MB buffers in MQGET calls without negative effects. Be sure to stop the queue manager before adding these details, and restart it afterwards.

TuningParameters:

AgentClassMap=(1,1,1)

AgentClassLimit1=(4,4,4)

If larger buffers are necessary, AgentClassLimit1=(2,2,2) should work for 100 MB messages, but it is difficult to be 100% sure. If in doubt, go to (1,1,1).

Note: These agent tuning parameters were initially introduced in MQSeriesÂ® V5.0 to allow for multi-threaded agents (amqzlaa0). SupportPacâ„¢ MP02 was used to describe these parameters and their use. Multi-threaded agents were incorporated into the default configuration beginning with MQSeries V5.1, so these tuning parameters were no longer needed for that purpose and SupportPac MP02 was withdrawn.

The tuning parameters are used here to alter the default configuration to use a smaller number of threads per agent process to allow for larger messages. Although SupportPac MP02 was withdrawn from support the parameters are still recognized by the queue manager. Problem Details If WebSphere MQ requests memory and the request is refused, then WebSphere MQ is in an irretrievable situation. The

system is not giving WebSphere MQ the memory needed, which generally means a limit has been reached. The trace shows a xcsGetMem for 4MB was issued, but it received rc=20006037. The return code 0x20006037 = xecS_E_NONE = AMQ6037 "MQSeries was unable to obtain enough storage." Several applications issue MQGET requests in the trace with 20 MB and 40 MB buffers. In response to this, the agent threads allocate similar sized buffers. This pattern of activity can not be sustained, since on AIX the maximum heap available in each WebSphere MQ process is 256 MB.

We have set to qmgr ini.file:

TuningParameters:
AgentClassMap=(1,1,1)
AgentClassLimit1=(2,2,2)

$ more qm.ini
#*******************************************************************#
#* Module Name: qm.ini *#
#* Type : WebSphere MQ queue manager configuration file *#
# Function : Define the configuration of a single queue manager *#
#* *#
#*******************************************************************#
#* Notes : *#
#* 1) This file defines the configuration of the queue manager *#
#* *#
#*******************************************************************#
ExitPath:
ExitsDefaultPath=/var/mqm/exits/
#* *#
#* *#
Log:
LogPrimaryFiles=3
LogSecondaryFiles=7
LogFilePages=12800
LogType=CIRCULAR
LogBufferPages=0
LogPath=/var/mqm/log/XXXX/
LogWriteIntegrity=TripleWrite
Service:
Name=AuthorizationService
EntryPoints=10
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/usr/mqm/lib/amqzfu
ComponentDataSize=0
TuningParameters:
AgentClassMap=(1,1,1)
AgentClassLimit1=(2,2,2)

Now all limits of mqm-users, soft and hard, are removed and there is lots of free memory. Physical memory is 8 GB
mqm-user ulimit is set to unlimit

$ uname -a
AIX xxxxxx 3 5 00C4AFF44C00

$ dspmqver
Name: WebSphere MQ
Version: 530.14 CSD14
CMVC level: p530-14-L071214
BuildType: IKAP - (Production)
$

After those tunings we get same AMQ6037 error message. After set mqm user ulimit to unlimit it helped when there was 1-3 instances but if there is more than 3 instances AMQ6037 appears again.

Any idea ?