| Author |
Message
|
| jefflowrey |
 Posted: Mon Jan 28, 2008 2:47 pm Post subject: WAS, Bindings connections, and EXTSHM |
 |
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I have a WAS v5.1.1.13 (I know...) that started failing to establish bindings connections (for MDB Listener Ports) at server startup after an upgrade to MQ v6.0.2.1.
MQ is throwing FDCs regarding AMQ6207: Failed to attach shared memory segment as Segment table is Full.
This is on AIX v5.3 - we've applied IY99425 (http://www-1.ibm.com/support/docview.wss?uid=swg1IY99425) and the AIX IY mentioned in there. We set EXTSHM=ON in both the mqm profile and the wasadm profile, and have confirmed that this is in effect.
Based on some other sources, I ran 'svmon -P' on the WAS server PID to produce a listing of the ext shm segments in use. This showed that the WAS server really is using up all of the segments - and that there really are none available.
However, there are dev and QA environments (this is a Perf environment) that are running the exact same levels of software and functioning just fine.
So this seems to be a WAS instance configuration problem - where the Perf systems have been tuned to use a different amount of memory or etc.
Any thoughts on what might be different, that could be tuned to make this work?
I will be checking JVM minHeap/maxHeap as well as some of the more obvious choices as soon as reasonably possible.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Jan 30, 2008 12:19 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Just me and the crickets?
Nobody has any thoughts on this? Nobody has seen this with any WAS environment on AIX with MQ bindings connections?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| ramires |
| Posted: Wed Jan 30, 2008 4:05 pm Post subject: |
|
|
Knight
Joined: 24 Jun 2001
Posts: 523
Location: Portugal - Lisboa
|
Hi Jeff,
some years ago I had a similar problem, it was with MQ 5.2 and WAS 3.5. The bypass was to use a qm.ini stanza IPCCBaseAddress like:
Sample qm.ini:
....
QueueManager:
Name=FMCQM
Prefix=/var/mqm
Directory=FMCQM
IPCCBaseAddress=12
...
Not sure if MQ v6 uses this IPCCBaseAdress..
You can also see:
http://www.ibm.com/developerworks/aix/library/au-segcollide.html
Hope it helps!
Joao |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 30, 2008 4:14 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Hey, thanks! 
Unfortunately, not only does MQ v6 not support IPCCBaseAddress... but it doesn't have the firm requirement on segment 8 being availble either. There is an IPCCPrimSegSize or some such, we've tried adjusting that too with no luck.
I guess I'll keep working the PMR.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jan 31, 2008 2:04 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| jefflowry wrote: |
| I have a WAS v5.1.1.13 (I know...) that started failing to establish bindings connections (for MDB Listener Ports) at server startup after an upgrade to MQ v6.0.2.1 |
Jeff have you tried upgrading to MQ v6.0.2.2 ?
I remember seeing something about MDBs and an APAR that would be part of the V6.0.2.2 fixpack...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jan 31, 2008 2:37 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I'll look for the APAR.
Going to 6.0.2.2 would require going back to square one, and applying again on (working) Dev and (working) QA, just to get a (hopefully working) Perf, leading to a (hopefully working) Prod.
It's an option, of course. But given that this isn't really an MDB problem, but really a memory management problem... The MDB listeners for client connections works just fine.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| seanb |
| Posted: Wed Feb 06, 2008 12:25 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2003
Posts: 39
|
Jeff,
Have you progressed any further on this problem?
We are having similar problems since moving to MQ 6 on a WAS 5.1.1.8 (yes, even older than your version) application. We migrated two servers. The one with JVM minHeap of 512MB is ok but the one with JVM minHeap of 1GB gets this error.
After applying 6.0.2.3 and following the instructions contained in IY99425 the AMQ6207 error went away. Perhaps simply applying IY99425 is not enough?
I am still stuck with the AMQ6119: An internal WebSphere MQ error has occurred ('24 - The process file
table is full.' from shmat.) errors though. I have checked the ulimit (segment/data) settings for both mqm and WAS and they are both unlimited.
Do you see a similar error accompanying your AMQ6207 error?
I am not sure what else to check for but the AMQ6119 error does feel related to the AMQ6207 problem, especially seeing the error doesn't occur with MQ 5.3.
Before I raise a PMR I want to be sure the second error I am getting is not due to a silly mistake on my behalf.
Regards,
Sean. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Feb 06, 2008 2:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
We did finally get a resolution, through a PMR that bounced through several hands.
We had to set the variable LDR_CNTRL to "MAXDATA=0X60000000@DSA" (without the quotes), in the server Environment Variables in the process definition. We'd tried setting it in setupCmdLine.sh, but had an old value being set in the process definition that was taking precedence.
Things you should validate before opening your PMR - 1) that you also applied the AIX iFix, 2) that you enabled EXTSHM in *both* MQ and WAS runtime environments, 3) that you aren't setting LDR_CNTRL anywhere, 4) that setting it to this "magic number" doesn't fix the problem, 5) that svmon -P on the JVM PID shows that all 16 segments are being used.
Especially if the last is *not* true, then you have a different problem.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| seanb |
| Posted: Wed Feb 06, 2008 5:24 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2003
Posts: 39
|
No luck.
1/2/4...done (re. 3, set to 0X60000000@DSA although from the previous FDC I see the value was 0XA0000000@DSA); 3...didn't see it being set anywhere but will double check; the FDC file suggests it has been set correctly; 5...if I read the output correctly I only see 7 segments being used.
I'll investigate further and then raise a PMR. Thanks for your suggestions. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Feb 06, 2008 6:31 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Apparently, it's not "MAXDATA=0X60000000@DSA", it's "MAXDATA=0XB0000000@DSA"
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| seanb |
| Posted: Sat Feb 09, 2008 12:03 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2003
Posts: 39
|
| The new "magic number" works a charm. Setting to "MAXDATA=0XB0000000@DSA" solved our problems. Thanks. |
|
| Back to top |
|
|
| manojsu |
| Posted: Mon Feb 18, 2008 4:03 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
I am havin the same prob, and one of the replies by Jeff says that we need to set the variable.
"We had to set the variable LDR_CNTRL to "MAXDATA=0X60000000@DSA" (without the quotes), in the server Environment Variables in the process definition."
Does the above statement mean that i need to create and additional WebSphere Varibale in WAS set it to LDR_CNTRL=MAXDATA=0XB0000000@DSA,
or
Jus execute export LDR_CNTRL=MAXDATA=0XB0000000@DSA and an restart the Whole of my MQ, MB and WAS. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Feb 18, 2008 4:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
This all only applies to AIX.
If you aren't using AIX, then it's entirely different.
Likely, you need to actually remove any LDR_CNTRL variables you have set, both in your environment and in the WAS variables.
but first you should make sure of the other things mentioned in this thread - the AIX iFix, EXTSHM, etc.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| manojsu |
| Posted: Mon Feb 18, 2008 4:52 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
Many Thanks for the quick reply,
I am definetly on an AIX machine,
I have set the EXTSHM,
Would be helpful if you help me on the AIX iFix and also you need to actually remove any LDR_CNTRL variables you have set, both in your environment and in the WAS variables.
Apologies if me asking a little too much |
|
| Back to top |
|
|
| seanb |
| Posted: Mon Feb 18, 2008 9:50 am Post subject: |
|
|
Apprentice
Joined: 02 Aug 2003
Posts: 39
|
The AIX iFix refers to fix IY66127 on AIX 5.2 or fix IY66147 on AIX 5.3. Ask your AIX admin to check whether they're installed.
If you follow the URL in the original posting it will contain this information and other important information.
If you log in as the user id WAS runs under (root or other) and type "export" check whether LDR_CNTRL is set. If it is, make the necessary changes (.profile, startup scripts, etc, wherever it is set) to remove it.
Also check the Process Definition (Servers/Application Servers/<your-server-name>/Additional Properties/Process Definition/Environment Entries) in the WAS admin console to ensure it is not set. You'll need to create the variable mentioned with Name of LDR_CNTRL and Value of MAXDATA=0XB0000000@DSA. |
|
| Back to top |
|
|
|
|
