| Author |
Message
|
| subhro |
 Posted: Thu May 02, 2013 12:44 pm Post subject: AMQ6119: shmall filling up.. Any help appreciated. |
 |
|
Newbie
Joined: 02 May 2013
Posts: 5
|
The shmall is filling up very quickly.
[/url] |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu May 02, 2013 12:53 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
When? On Thursdays, because it's never quite got the hang of Thursdays? When the queue manager is under load, under the weather, under the gun, understandably upset?
When did this start happening? If this started happening when you changed something (not necessarily anything MQ related!) have you tried changing it back?
Have you reviewed AMQ6119 in the product documentation? Have you tried the advice you found there? Have you tried researching AMQ6119 in a new tool they've developed called "Google"?
Have you in fact done anything apart from post an almost meaningless sentence here and expect us to infer your entire situation? If so you expect too much of me for sure and probably most of us. I for one am psychotic not psychic which is an important distinction, especially when I'm surrounded by lovely, shiny, sharp objects.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| subhro |
| Posted: Thu May 02, 2013 1:08 pm Post subject: |
|
|
Newbie
Joined: 02 May 2013
Posts: 5
|
Vitor, Thanks for ur compliments..
yes google and other dcumentation were all completed.. PMR has been opened with IBM nothing has happened so far...
This is is new QMGR and put in to production last sunday..
Previously the shmall was below as recommned by IBM,
2097152 pages
But IBM recommned to increase to double , still it fills up within an hr |
|
| Back to top |
|
|
| subhro |
| Posted: Thu May 02, 2013 1:11 pm Post subject: |
|
|
Newbie
Joined: 02 May 2013
Posts: 5
|
Vitor,
Actually now i see my error messages which is posted doidnot get attached.. i am going to attach it now..
Last edited by subhro on Thu May 02, 2013 1:21 pm; edited 2 times in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 02, 2013 1:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| subhro wrote: |
PMR has been opened with IBM nothing has happened so far...
This is is new QMGR and put in to production last sunday.. |
These 2 statements seem at variance. If this is a production queue manager it sounds rather like the priority of your PMR is too low.
| subhro wrote: |
Previously the shmall was below as recommned by IBM,
2097152 pages
But IBM recommned to increase to double , still it fills up within an hr |
OK, if you're not following IBM recommendations then you're in a trap of your own making. Also that number isn't the IBM recommended value, it's the IBM recommended minimum value. Depending on what you're doing to or with this queue manager you could need a value in excess of double that value.
And you've still not said anything about the circumstances of this error apart from it takes about a hour of queue manager run time to manifest. What's the queue manager doing in that hour? Processing messages from online transactions at 800 TPS? Servicing connections from 4000 simultaneously active clients? Knitting?
Better information, better advice. As a minimum tell us which version of WMQ & which OS you're using.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| subhro |
| Posted: Thu May 02, 2013 1:21 pm Post subject: |
|
|
Newbie
Joined: 02 May 2013
Posts: 5
|
My server has 64GB of memeory and 50 GB free. Its running MQ 7.1.02 on RHEL 6.2
Here is the mqconfig output
[mqm@dc1nix2p51 errors]$ /tmp/mqconfig -v 7.1
mqconfig: V3.6 analyzing Red Hat Enterprise Linux Server release 6.2
(Santiago) settings for WebSphere MQ V7.1
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=500 PASS
semmns (sem:2) 62 of 256000 semaphores (0%) IBM>=256000 PASS
semopm (sem:3) 250 operations IBM>=250 PASS
semmni (sem:4) 18 of 1024 sets (1%) IBM>=1024 PASS
System V Shared Memory
shmmax 1073741824 bytes IBM>=268435456 PASS
shmmni 167 of 4096 sets (4%) IBM>=4096 PASS
shmall 6285286 of 6291456 pages (99%) IBM>=2097152 FAIL
System Settings
file-max 8384 of 524288 files (1%) IBM>=524288 PASS
tcp_keepalive_time 300 seconds IBM<=300 PASS
Current User Limits (mqm)
nofile (-Hn) 10240 files IBM>=10240 PASS
nofile (-Sn) 10240 files IBM>=10240 PASS
nproc (-Hu) 44 of 4096 processes (1%) IBM>=4096 PASS
nproc (-Su) 44 of 4096 processes (1%) IBM>=4096 PASS
snippet form the last FDC file
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Thu May 02 2013 10:01:06 EDT |
| UTC Time :- 1367503266.387382 |
| UTC Time Offset :- -240 (EST) |
| Host Name :- dc1nix2p51 |
| Operating System :- Linux 2.6.32-220.4.1.el6.x86_64 |
| PIDS :- 5724H7230 |
| LVLS :- 7.1.0.2 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (1) |
| Probe Id :- ZC046010 |
| Application Name :- MQM |
| Component :- zcpError |
| SCCS Info :- lib/zc/amqzcpb0.c, 1.113.1.1 |
| Line Number :- 3178 |
| Build Date :- Oct 30 2012 |
| CMVC level :- p710-002-121029 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 50083 (mqm) |
| Real UserID :- 50083 (mqm) |
| Program Name :- runmqsc |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 16280 |
| Process(Thread) :- 16280 |
| Thread(n) :- 1 |
| QueueManager :- DSSPRD |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 11610 |
| ConnId(3) QM-P :- 10663 |
| Last HQC :- 1.0.0-1241592 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- krcE_UNEXPECTED_ERROR |
| Minor Errorcode :- OK |
| Probe Type :- INCORROUT |
| Probe Severity :- 2 |
| Probe Description :- AMQ6125: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
mqscMAIN
uscRunScript
uscRunCommand
"AMQ16280.0.FDC" 1151L, 56746C
Snippet from AMQERR01.LOG
AMQ6119: An internal WebSphere MQ error has occurred (Failed to get memory
segment: shmget(0x00000000, 268435456) [rc=-1 errno=28] No space left on
hm
evice)
EXPLANATION:
MQ detected an unexpected error when calling the operating system. The MQ error
recording routine has been called.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier and to save any generated output files. Use either the MQ Support
site: http://www.ibm.com/software/integration/wmq/support/, or IBM Support
Assistant (ISA): http://www.ibm.com/software/support/isa/, to see whether a
solution is already available. If you are unable to find a match, contact your
IBM support center. Do not discard these files until the problem has been
Here is the qm.ini
#*******************************************************************#
#* Module Name: qm.ini *#
#* Type : WebSphere MQ queue manager configuration file *#
# Function : Define the configuration of a single queue manager *#
#* *#
#*******************************************************************#
#* Notes : *#
#* 1) This file defines the configuration of the queue manager *#
#* *#
#*******************************************************************#
ExitPath:
ExitsDefaultPath=/var/mqm/exits
ExitsDefaultPath64=/var/mqm/exits64
#* *#
#* *#
Log:
LogPrimaryFiles=10
LogSecondaryFiles=5
LogFilePages=10240
LogType=CIRCULAR
LogBufferPages=4096
LogPath=/MQHA/qm5/log/DSSPRD/
LogWriteIntegrity=TripleWrite
Service:
Name=AuthorizationService
EntryPoints=14
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=amqzfu
ComponentDataSize=0
TCP:
KeepAlive=yes
PORT=1480
Broker:
SyncPointIfPersistent=yes
Channels:
MaxChannels=3000
MQIBindType=FASTPATH |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 02, 2013 1:33 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| subhro wrote: |
| My server has 64GB of memeory and 50 GB free. Its running MQ 7.1.02 on RHEL 6.2 |
Thank you.
| subhro wrote: |
Here is the mqconfig output
[mqm@dc1nix2p51 errors]$ /tmp/mqconfig -v 7.1
mqconfig: V3.6 analyzing Red Hat Enterprise Linux Server release 6.2
(Santiago) settings for WebSphere MQ V7.1
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=500 PASS
semmns (sem:2) 62 of 256000 semaphores (0%) IBM>=256000 PASS
semopm (sem:3) 250 operations IBM>=250 PASS
semmni (sem:4) 18 of 1024 sets (1%) IBM>=1024 PASS
System V Shared Memory
shmmax 1073741824 bytes IBM>=268435456 PASS
shmmni 167 of 4096 sets (4%) IBM>=4096 PASS
shmall 6285286 of 6291456 pages (99%) IBM>=2097152 FAIL
System Settings
file-max 8384 of 524288 files (1%) IBM>=524288 PASS
tcp_keepalive_time 300 seconds IBM<=300 PASS
Current User Limits (mqm)
nofile (-Hn) 10240 files IBM>=10240 PASS
nofile (-Sn) 10240 files IBM>=10240 PASS
nproc (-Hu) 44 of 4096 processes (1%) IBM>=4096 PASS
nproc (-Su) 44 of 4096 processes (1%) IBM>=4096 PASS |
Isn't it interesting that the shmall entry has FAIL next to it, while the rest have PASS?
Presumably I should assume the queue manager is knitting for the hour before it crashes, as you still refuse to offer any details on what's actually happening but just seem determined I should review your configuration as you (presumably) have already done rather than attempt to help you?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| subhro |
| Posted: Thu May 02, 2013 1:36 pm Post subject: |
|
|
Newbie
Joined: 02 May 2013
Posts: 5
|
No user calls, users not having any dificiluty in pub/sub.
I need to know if i increase the shmall, it would fill up again or not? and need to know why its filling up so quickly.
Please let me know if you need any more configuration files to be looked in |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 02, 2013 2:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| subhro wrote: |
| No user calls, users not having any dificiluty in pub/sub. |
Gosh, the information just floods out of you now! So you have user applications doing publish/subscribe; thank you! How many application, how many connections, what's their reconnect strategy? Do you say "users not having any difficulty" because there are "no user calls" in the sense that no-one's picking up the phone and shouting at you? Because one very possible theory is that you have a large number of connected clients with no real idea how to connect to a queue manager efficiently, when the queue manager tries to handle the problem by throwing them out they automatically reconnect filling the queue manager's memory to the point where it fills up and you see this problem. But they don't because they're sitting on this spinning pool of connections and nothing's wrong in their world until the queue manager collapses under them.
| subhro wrote: |
| I need to know if i increase the shmall, it would fill up again or not? |
Well probably. If something's causing the resource to be exhausted due to poor management (as I outline above) then it will fill any amount of resource allocated. If it's simply that the queue manager is insufficiently sized for the normal running load then no it won't. How am I supposed to answer that question of yours with the information you've given?
| subhro wrote: |
| and need to know why its filling up so quickly. |
Again, how can I tell? You won't even tell me how you use the queue manager.
| subhro wrote: |
| Please let me know if you need any more configuration files to be looked in |
No. I don't see any more configuration as being relevant to the problem.
What's needed is information, the sort of information you get from investigating a problem rather than blindly parroting the configuration as you seem determined to do.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
