| Author |
Message
|
| smeunier |
 Posted: Fri Aug 21, 2015 7:55 am Post subject: Looping FDC generation on QMGR |
 |
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
This problem started about a week back with no know system changes the proceeded it.
The basics:
Hardware:
$ lsconf
System Model: IBM,7778-23X
Machine Serial Number: 10D288A
Processor Type: PowerPC_POWER6
Processor Implementation Mode: POWER 6
Processor Version: PV_6
Number Of Processors: 4
Software:
Name: WebSphere MQ
Version: 7.5.0.4
Level: p750-004-140807
BuildType: IKAP - (Production)
Platform: WebSphere MQ for AIX
Mode: 64-bit
O/S: AIX 6.1
InstName: Installation1
InstDesc:
Primary: No
InstPath: /usr/mqm
DataPath: /var/mqm
MaxCmdLevel: 750
The Problem:
reoccurring FDC files being generated at the rate of about 6/minute with SIGSEGV errors. Most times, the QMGR will not even start. However, once in a while it will, where it just constantly generates FDC files.
What we have done thus far is to:
- delete qmgr, recreate. This failed
- delete qmgr, delete /var/mqm file system, recreate file system, re-install product, create qmgr. This failed. It would not start, but hours later, it did?!
- stop all other installed application process, WebSphere App Server, DB2, etc. Qmgr didn't start, but found couple semaphores that should not have been there, and it started, but started generating FDC files
Have opened IBM PMR (no response yet), and searched the globe for similar problems, but nothing hits the mark.
Looking for ANY Suggestions on how to resolve.
Much appreciated. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Aug 21, 2015 7:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is the ProbeId in all of the FDCs the same?
Or different?
Do the FDCs come in a fixed series - i.e. a repeating set of records?
Is it a set of FDCs in the /var/mq/errors, or in the /var/qmgr/<qm name>/errors ?
Which program is throwing the FDCs?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Aug 21, 2015 8:35 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
Back to basics...
Have all the corresponding kernel parameters been set high enough?
Have the limits been set high enough for the mqm user?
Worth checking 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smeunier |
| Posted: Fri Aug 21, 2015 8:44 am Post subject: |
|
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
| Quote: |
Is the ProbeId in all of the FDCs the same?
Or different? |
It starts out with two unique:
Probe Id :- XC130003
Application Name :- MQM
Component :- xehExceptionHandler
Probe Type :- HALT6109
Probe Severity :- 1
Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred.
FDCSequenceNumber :- 0
Arith1 :- 11 (0xb)
Comment1 :- SIGSEGV: invalid address permissions(110170000)
Probe Id :- MQ000040
Application Name :- MQM
Component :- Unknown
Probe Type :- MSGAMQ6004
Probe Severity :- 1
Probe Description :- AMQ6004: An error occurred during WebSphere MQ
initialization or ending.
FDCSequenceNumber :- 0
then loops with this(6/minute)
Probe Id :- ZC002050
Application Name :- MQM
Component :- zcpAttachPipe
Probe Type :- INCORROUT
Probe Severity :- 4
Probe Description :- AMQ6125: An internal WebSphere MQ error has occurred.
FDCSequenceNumber :- 0
| Quote: |
| Is it a set of FDCs in the /var/mq/errors, or in the /var/qmgr/<qm name>/errors ? |
FDC files are in the /var/mqm/errors directory |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 21, 2015 8:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
/var/mqm/errors means they are not specific to a single queue manager.
Which strongly suggests that F.J. is right about kernel/file limits.
Did any maintenance of the OS occur recently?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smeunier |
| Posted: Fri Aug 21, 2015 8:56 am Post subject: |
|
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
| Quote: |
Back to basics...
Have all the corresponding kernel parameters been set high enough?
Have the limits been set high enough for the mqm user? |
mqconfig: Analyzing AIX 6.1 TL9 (powerpc) settings for WebSphere MQ V7.5
System V Semaphores
semmni 14 sets [Auto Tuned] PASS
semmns 279 semaphores [Auto Tuned] PASS
System V Shared Memory
shmmni 10 sets [Auto Tuned] PASS
System Settings
maxuproc 9 of 4096 processes (0%) IBM>=1024 PASS
Current User Limits (mqm)
nofiles (-Hn) unlimited files IBM>=10240 PASS
nofiles (-Sn) 10240 files IBM>=10240 PASS
data (-Sd) unlimited kbytes IBM>=unlimited PASS
stack (-Ss) unlimited kbytes IBM>=unlimited PASS
Shell Default Options (mqm)
ksh bgnice:off IBM:off PASS |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 21, 2015 10:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| smeunier |
| Posted: Fri Aug 21, 2015 10:44 am Post subject: |
|
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
| Quote: |
| Did any maintenance of the OS occur recently? |
Finally got a reply from admins on this.
AIX TL7 to TL9
But, as pointed out, this was done on the other machines in this group servers as well. They are all running fine, except this one. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 22, 2015 9:03 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20768
Location: LI,NY
|
Well depending on the number of qmgrs and connections you use on this box you may need to boost those kernel settings. The V8 knowledge center has a list of parameters with the mention of whether or not you should increase them depending on system load.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 24, 2015 4:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I know you did do the mqconfig check - but all that does, afaik, is show you whether the settings are as large or larger than the minimum requirements.
As FJ says, these may need to be larger based on the amount of traffic or number of queue managers.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smeunier |
| Posted: Mon Aug 24, 2015 4:55 am Post subject: |
|
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
| Quote: |
| Well depending on the number of qmgrs and connections you use |
There is a single QMGR and no connections at this time as I do not start with a listener and also try to start with no services (strmqm -ns) any resource needs should be minimum at the point of start-up. There woulkd only be two message channel trying to start. All WebSphere MQ apps have been disabled and all cron MQ apps removed. All other products (DB2 and WAS) have been shut down trying to determine issue.
Monitoring of system resources by AIX admins show no elevated usage/spikes from before issue to current problem state. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 24, 2015 5:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I hope you have opened a PMR before now.
The simple facts that you don't have a queue manager running and the FDCs are appearing at the MQ level rather than at individual queue manager level suggest that the issue is with the MQ install, rather than with the OS configuration.
It's not unreasonable to simply uninstall the FP, restart the MQ services and verify that you don't get the FDCs again. Then reinstall the FP and see if they appear.
If they do, look for iFixes for your platform. And open a PMR if you haven't.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smeunier |
| Posted: Mon Aug 24, 2015 5:16 am Post subject: |
|
|
Partisan
Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont
|
PMR has been opened, awaiting contact on that.
We were running on V7.5.0.3, which is where most of our QMGRS are at, and upgraded this one to FP4, after experiencing these issues.
Hopefully I'm contacted soon on the PMR. Will post findings here on any resolution. |
|
| Back to top |
|
|
|
|
