| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| '22 - Invalid argument' from semop |
« View previous topic :: View next topic » |
| Author |
Message
|
| liamf |
 Posted: Wed Aug 13, 2025 10:44 pm Post subject: '22 - Invalid argument' from semop |
 |
|
Novice
Joined: 22 Apr 2021
Posts: 15
|
Versions:
IIB: v10.0.0.0
MQ: v8.0.0.0
Description:
1.We found that on every 6 days, IIB will lost listening on port 4414, after a deep check in FDC file, we found it's a Semaphore issue - AMQ6119: An internal WebSphere MQ error has occurred |
| ('22 - Invalid argument' from semop.) and the corresponding QM status became NOT AVALIABLE.
2.We temporally fix it with a system reboot, but after 6 days it happens again.
3.We have a cron job that listening to a specific queue, when this queue has message in it, cron job will start a messageflow and vice versa, i'm not sure if it is the reason that cause the problem because this action was ongoing all along not just before this issue happens.
4.From another server which has the same issue, we found this from FMT file:
---------} xcsGetMemFn rc=OK FunctionTime=6
03:30:36.572084 3454.398 : ---------{ xstGetSubpoolsLock
03:30:36.572087 3454.398 : Subpool = 4
03:30:36.572089 3454.398 : ----------{ xstAccessSubpoolsLock
03:30:36.572091 3454.398 : -----------{ xcsQueryValue
03:30:36.572096 3454.398 : 0x0000: 2f766172 2f6d716d 2f736f63 6b657473 |/var/mqm/sockets|
03:30:36.572098 3454.398 : -----------} xcsQueryValue rc=OK FunctionTime=7
03:30:36.572112 3454.398 : About to get semaphore lock /var/mqm/sockets/QMESB/qmgrlocl/ESB-IIB-03/subpool.lck.
03:30:36.572115 3454.398 : -----------{ xcsCreateFile
03:30:36.572122 3454.398 : File: '/var/mqm/sockets/QMESB/qmgrlocl/ESB-IIB-03/subpool.lck', openopts: 0, flags: 0, permissions: 436, ccsid: 0, ffstOnError: TRUE
03:30:36.572141 3454.398 : Data: 0x00000000 0x000001b4 0x00000000 0x00000189
03:30:36.572149 3454.398 : Changing permissions failed with errno 1
03:30:36.572154 3454.398 : fd: 393, error: 1
03:30:36.572156 3454.398 : -----------} xcsCreateFile rc=OK FunctionTime=41
03:30:36.572170 3454.398 : Failed to inquire on semaphore set(163842) errno(22)
03:30:36.572173 3454.398 : Data: 0x00000189
03:30:36.572179 3454.398 : ----------}! xstAccessSubpoolsLock rc=xecI_E_NO_SUCH_SUBPOOL FunctionTime=90
03:30:36.572181 3454.398 : ---------}! xstGetSubpoolsLock rc=xecI_E_NO_SUCH_SUBPOOL FunctionTime=97
03:30:36.572183 3454.398 : ---------{ xcsFreeMemFn
Any idea? much thanks!
FDC:
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Thu July 24 2025 03:30:50 CST |
| UTC Time :- 1753299050.173335 |
| UTC Time Offset :- 480 (CST) |
| Host Name :- ESB-IIB-03 |
| Operating System :- Linux 3.10.0-327.el7.x86_64 |
| PIDS :- 5724H7251 |
| LVLS :- 8.0.0.0 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| O/S Registered :- 1 |
| Data Path :- /var/mqm |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (1) |
| License Type :- Production |
| Probe Id :- XY324103 |
| Application Name :- MQM |
| Component :- xstGetSubpoolsLock |
| SCCS Info :- |
| /build/slot1/p000_P/src/lib/cs/unix/generic/amqxiinx.c, |
| Line Number :- 2172 |
| Build Date :- Apr 29 2014 |
| Build Level :- p000-L140429.1 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 1003 (aviation) |
| Real UserID :- 1003 (aviation) |
| Program Name :- DataFlowEngine |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 3457 |
| Process(Thread) :- 4229 |
| Thread :- 44 |
| QueueManager :- QMESB |
| UserApp :- TRUE |
| Last HQC :- 1.0.0-144000 |
| Last HSHMEMB :- 1.2.2-37663424 |
| Last ObjectName :- |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| ('22 - Invalid argument' from semop.) |
| FDCSequenceNumber :- 0 |
| Arith1 :- 22 (0x16) |
| Comment1 :- '22 - Invalid argument' from semop. |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
MQDISC
zstMQDISC
ziiClearUpAgent
xcsTerminate
xcsDisconnectSharedSubpool
xcsDetachSharedSubpool
xstGetSubpoolsLock
xcsFFST
MQM Trace History
---} zutLookupLock rc=OK
---{ zutLookupUnlock
---} zutLookupUnlock rc=OK
--} zutLookupRelease rc=OK
-} zstReleaseSharedPCD rc=OK
} MQCB rc=OK
{ MQBACK
-{ zstMQBACK
--{ zstAcquireSharedPCD
---{ zutLookupGet
Data: 0x0145e8a9
----{ zutLookupLock
----} zutLookupLock rc=OK
----{ zutLookupUnlock
----} zutLookupUnlock rc=OK
---} zutLookupGet rc=OK
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsUseSharedCtlOnThisThread
---} xcsUseSharedCtlOnThisThread rc=OK
---{ ziiMovePCDToThread
---} ziiMovePCDToThread rc=OK
--} zstAcquireSharedPCD rc=OK
--{ zstGetPCD
---{ zstVerifyPCD
---} zstVerifyPCD rc=OK
--} zstGetPCD rc=OK
--{ ziiMQBACK
---{ ziiCreateIPCCMessage
----{ zcpCreateMessage
----} zcpCreateMessage rc=OK
---} ziiCreateIPCCMessage rc=OK
---{ ziiSendReceiveAgent
----{ zcpSendOnPipe
-----{ xlsResetEvent
-----} xlsResetEvent rc=OK
-----{ xlsPostEvent
Data: 0x00018a20 0x00006ca1
-----} xlsPostEvent rc=OK
----} zcpSendOnPipe rc=OK
----{ zcpReceiveOnPipe
-----{ xlsWaitEvent
Data: 0x00000000
Data: 0x00000000
Data: 0x00000001
Data: 0x00000000
Data: 0x80000000 0x00000001
-----} xlsWaitEvent rc=OK
----} zcpReceiveOnPipe rc=OK
---} ziiSendReceiveAgent rc=OK
---{ zcpDeleteMessage
---} zcpDeleteMessage rc=OK
--} ziiMQBACK rc=OK
-} zstMQBACK rc=OK
-{ zstReleaseSharedPCD
--{ xcsReleaseThread
--} xcsReleaseThread rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
--{ zutLookupRelease
---{ zutLookupLock
---} zutLookupLock rc=OK
---{ zutLookupUnlock
---} zutLookupUnlock rc=OK
--} zutLookupRelease rc=OK
-} zstReleaseSharedPCD rc=OK
} MQBACK rc=OK
{ MQCLOSE
-{ zstMQCLOSE
--{ zstAcquireSharedPCD
---{ zutLookupGet
Data: 0x0145e8a9
----{ zutLookupLock
----} zutLookupLock rc=OK
----{ zutLookupUnlock
----} zutLookupUnlock rc=OK
---} zutLookupGet rc=OK
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsUseSharedCtlOnThisThread
---} xcsUseSharedCtlOnThisThread rc=OK
---{ ziiMovePCDToThread
---} ziiMovePCDToThread rc=OK
--} zstAcquireSharedPCD rc=OK
--{ zstGetPCD
---{ zstVerifyPCD
---} zstVerifyPCD rc=OK
--} zstGetPCD rc=OK
--{ xcsCheckPointer
--} xcsCheckPointer rc=OK
--{ ziiMQCLOSE
---{ ziiCreateIPCCMessage
----{ zcpCreateMessage
----} zcpCreateMessage rc=OK
---} ziiCreateIPCCMessage rc=OK
---{ ziiSendReceiveAgent
----{ zcpSendOnPipe
-----{ xlsResetEvent
-----} xlsResetEvent rc=OK
-----{ xlsPostEvent
Data: 0x00018a20 0x00006ca1
-----} xlsPostEvent rc=OK
----} zcpSendOnPipe rc=OK
----{ zcpReceiveOnPipe
-----{ xlsWaitEvent
Data: 0x00000000
Data: 0x00000000
Data: 0x00000001
Data: 0x00000000
Data: 0x80000000 0x00000001
-----} xlsWaitEvent rc=OK
----} zcpReceiveOnPipe rc=OK
---} ziiSendReceiveAgent rc=OK
---{ zcpDeleteMessage
---} zcpDeleteMessage rc=OK
--} ziiMQCLOSE rc=OK
--{ zstAsyncMarkHobjClosed
--} zstAsyncMarkHobjClosed rc=OK
--{ xcsFreeMemFn
--} xcsFreeMemFn rc=OK
-} zstMQCLOSE rc=OK
-{ zstReleaseSharedPCD
--{ xcsReleaseThread
--} xcsReleaseThread rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
--{ zutLookupRelease
---{ zutLookupLock
---} zutLookupLock rc=OK
---{ zutLookupUnlock
---} zutLookupUnlock rc=OK
--} zutLookupRelease rc=OK
-} zstReleaseSharedPCD rc=OK
} MQCLOSE rc=OK
{ MQDISC
-{ zstMQDISC
--{ xcsCheckPointer
--} xcsCheckPointer rc=OK
--{ zstAcquireSharedPCD
---{ zutLookupGet
Data: 0x0145e8a9
----{ zutLookupLock
----} zutLookupLock rc=OK
----{ zutLookupUnlock
----} zutLookupUnlock rc=OK
---} zutLookupGet rc=OK
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsUseSharedCtlOnThisThread
---} xcsUseSharedCtlOnThisThread rc=OK
---{ ziiMovePCDToThread
---} ziiMovePCDToThread rc=OK
--} zstAcquireSharedPCD rc=OK
--{ zstGetPCD
---{ zstVerifyPCD
---} zstVerifyPCD rc=OK
--} zstGetPCD rc=OK
--{ zstAsyncMarkHobjClosed
--} zstAsyncMarkHobjClosed rc=OK
--{ ziiMQDISC
---{ ziiCreateIPCCMessage
----{ zcpCreateMessage
----} zcpCreateMessage rc=OK
---} ziiCreateIPCCMessage rc=OK
---{ ziiSendReceiveAgent
----{ zcpSendOnPipe
-----{ xlsResetEvent
-----} xlsResetEvent rc=OK
-----{ xlsPostEvent
Data: 0x00018a20 0x00006ca1
-----} xlsPostEvent rc=OK
----} zcpSendOnPipe rc=OK
----{ zcpReceiveOnPipe
-----{ xlsWaitEvent
Data: 0x00000000
Data: 0x00000000
Data: 0x00000001
Data: 0x00000000
Data: 0x80000000 0x00000001
-----} xlsWaitEvent rc=OK
----} zcpReceiveOnPipe rc=OK
---} ziiSendReceiveAgent rc=OK
---{ zcpDeleteMessage
---} zcpDeleteMessage rc=OK
--} ziiMQDISC rc=OK
--{ zutLockLocalThreads
--} zutLockLocalThreads rc=OK
--{ ziiClearUpAgent
---{ zcpDetachPipe
----{ xcsUnregisterDestructor
-----{ xcsFreeMemFn
-----} xcsFreeMemFn rc=OK
----} xcsUnregisterDestructor rc=OK
---} zcpDetachPipe rc=OK
---{ ziiFinishedWithAgent
---} ziiFinishedWithAgent rc=OK
---{ xcsUnregisterDestructor
----{ xcsFreeMemFn
----} xcsFreeMemFn rc=OK
---} xcsUnregisterDestructor rc=OK
---{ xcsTerminate
----{ xcsRequestThreadMutexSem
----} xcsRequestThreadMutexSem rc=OK
----{ xihRemoveQueueManager
----} xihRemoveQueueManager rc=Unknown(1)
----{ xcsDisconnectSharedSubpool
-----{ xlsThreadTermination
------{ xcsFreeQuickCell
------} xcsFreeQuickCell rc=OK
-----} xlsThreadTermination rc=OK
-----{ xcsDetachSharedSubpool
------{ xstGetSubpoolsLock
-------{ xstAccessSubpoolsLock
--------{ xcsQueryValue
--------} xcsQueryValue rc=OK
-------} xstAccessSubpoolsLock rc=OK
-------{ xcsBuildDumpPtr
--------{ xcsGetMemFn
--------} xcsGetMemFn rc=OK
-------} xcsBuildDumpPtr rc=OK
-------{ xcsStrerror
-------} xcsStrerror rc=OK
-------{ xcsFFST
SemID passed to semop
0x7f579c21e3f0 00000000 ....
System Log:
Jul 24 03:28:10 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'TB_YB_ERROR_LOG' of type '.APPZIP'.
Jul 24 03:28:13 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2271I: Deployed resource 'TB_YB_ERROR_LOG' (uuid='22172dc6-e69b-48a0-beb2-76c6aa8ff40b',type='.APPZIP') successfully stopped.
Jul 24 03:28:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:28:15 ESB-IIB-03 systemd: Started Session c18331 of user aviation.
Jul 24 03:28:15 ESB-IIB-03 systemd: Starting Session c18331 of user aviation.
Jul 24 03:28:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:28:16 ESB-IIB-03 systemd: Started Session c18332 of user aviation.
Jul 24 03:28:16 ESB-IIB-03 systemd: Starting Session c18332 of user aviation.
Jul 24 03:29:04 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'TB_YB_ERROR_LOG' of type '.APPZIP'.
Jul 24 03:29:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:29:15 ESB-IIB-03 systemd: Started Session c18333 of user aviation.
Jul 24 03:29:15 ESB-IIB-03 systemd: Starting Session c18333 of user aviation.
Jul 24 03:29:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:29:16 ESB-IIB-03 systemd: Started Session c18334 of user aviation.
Jul 24 03:29:16 ESB-IIB-03 systemd: Starting Session c18334 of user aviation.
Jul 24 03:30:01 ESB-IIB-03 systemd: Started Session 5388 of user root.
Jul 24 03:30:01 ESB-IIB-03 systemd: Starting Session 5388 of user root.
Jul 24 03:30:01 ESB-IIB-03 systemd: Started Session 5389 of user root.
Jul 24 03:30:01 ESB-IIB-03 systemd: Starting Session 5389 of user root.
Jul 24 03:30:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:30:15 ESB-IIB-03 systemd: Started Session c18335 of user aviation.
Jul 24 03:30:15 ESB-IIB-03 systemd: Starting Session c18335 of user aviation.
Jul 24 03:30:15 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'TB_YB_ERROR_LOG' of type '.APPZIP'.
Jul 24 03:30:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:30:16 ESB-IIB-03 systemd: Started Session c18336 of user aviation.
Jul 24 03:30:16 ESB-IIB-03 systemd: Starting Session c18336 of user aviation.
Jul 24 03:30:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:30:16 ESB-IIB-03 systemd: Started Session c18337 of user aviation.
Jul 24 03:30:16 ESB-IIB-03 systemd: Starting Session c18337 of user aviation.
Jul 24 03:30:18 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2271I: Deployed resource 'TB_YB_ERROR_LOG' (uuid='22172dc6-e69b-48a0-beb2-76c6aa8ff40b',type='.APPZIP') successfully stopped.
Jul 24 03:30:35 ESB-IIB-03 systemd: Removed slice user-1002.slice.
Jul 24 03:30:35 ESB-IIB-03 systemd: Stopping user-1002.slice.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 1/1) BIP2152I: Configuration message received from integration node.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 1/1) BIP2153I: About to 'Stop' an integration server.
Jul 24 03:30:35 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2152I: Configuration message received from integration node.
Jul 24 03:30:35 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/1) BIP2153I: About to 'Stop' an integration server.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/BKESB/Status/ExecutionGroup/EG2' using the 'MQ' transport.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'QMESB': MQCC=2; MQRC=2059.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG2' using the 'MQTT' transport.
Jul 24 03:30:35 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3725] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11884_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG2' using server connection 'tcp://localhost:11884', Client id 'BKESB.3455', Security identity username ''. The exception is 'Client is not connected'. The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:102)]'.
Jul 24 03:30:35 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG1' using the 'MQTT' transport.
Jul 24 03:30:35 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3726] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11884_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG1' using server connection 'tcp://localhost:11884', Client id 'BKESB.3457', Security identity username ''. The exception is 'Client is not connected'. The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:102)]'.
Jul 24 03:30:35 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 1/1) BIP2152I: Configuration message received from integration node.
Jul 24 03:30:35 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 1/1) BIP2153I: About to 'Stop' an integration server.
Jul 24 03:30:36 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/BKESB/Status/ExecutionGroup/EG' using the 'MQ' transport.
Jul 24 03:30:36 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'QMESB': MQCC=2; MQRC=2059.
Jul 24 03:30:36 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG' using the 'MQTT' transport.
Jul 24 03:30:36 ESB-IIB-03 IIB[3454]: IBM Integration Bus v10000 (BKESB.EG) [Thread 3727] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11884_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/BKESB/Status/ExecutionGroup/EG' using server connection 'tcp://localhost:11884', Client id 'BKESB.3454', Security identity username ''. The exception is 'Client is not connected'. The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:102)]'.
Jul 24 03:30:37 ESB-IIB-03 IIB[3023]: IBM Integration Bus v10000 (BKESB) [Thread 3023] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11.
Jul 24 03:30:37 ESB-IIB-03 IIB[3023]: IBM Integration Bus v10000 (BKESB) [Thread 3023] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10000; process ID 91196.
Jul 24 03:30:38 ESB-IIB-03 IIB[91196]: IBM Integration Bus v10000 (BKESB) [Thread 91196] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is inactive.
Jul 24 03:30:38 ESB-IIB-03 IIB[91196]: IBM Integration Bus v10000 (BKESB) [Thread 91196] (Msg 1/1) BIP8236I: A standby instance of integration node BKESB has been started against the standby queue manager QMESB. The active integration node instance and active queue manager instance are running elsewhere.
Jul 24 03:30:42 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3455.0.FDC
Jul 24 03:30:50 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3457.0.FDC
Jul 24 03:30:50 ESB-IIB-03 IIB[3457]: IBM Integration Bus v10000 (BKESB.EG1) [Thread 3457] (Msg 1/1) BIP2204I: Integration server using process '3457' thread '3457' stopped.
Jul 24 03:30:56 ESB-IIB-03 IIB[3455]: IBM Integration Bus v10000 (BKESB.EG2) [Thread 3455] (Msg 1/1) BIP2204I: Integration server using process '3455' thread '3455' stopped.
Jul 24 03:31:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:31:15 ESB-IIB-03 systemd: Started Session c18338 of user aviation.
Jul 24 03:31:15 ESB-IIB-03 systemd: Starting Session c18338 of user aviation.
Jul 24 03:31:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:31:16 ESB-IIB-03 systemd: Started Session c18339 of user aviation.
Jul 24 03:31:16 ESB-IIB-03 systemd: Starting Session c18339 of user aviation.
Jul 24 03:31:17 ESB-IIB-03 abrt-hook-ccpp: /var/spool/abrt is 15400493422 bytes (more than 1279MiB), deleting 'ccpp-2025-07-23-08:42:53-91142'
Jul 24 03:31:17 ESB-IIB-03 abrt-server: Executable '/esb/EAsmbl_image/iib-10.0.0.0/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jul 24 03:31:17 ESB-IIB-03 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2025-07-24-03:30:54-3454' exited with 1
Jul 24 03:31:17 ESB-IIB-03 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2025-07-24-03:30:54-3454'
Jul 24 03:31:30 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ100896.0.FDC
Jul 24 03:31:30 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ100896.0.FDC
Jul 24 03:31:34 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ4249.0.FDC
Jul 24 03:31:34 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ4249.0.FDC
Jul 24 03:32:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:32:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:32:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:32:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:32:01 ESB-IIB-03 systemd: Started Session 5390 of user root.
Jul 24 03:32:01 ESB-IIB-03 systemd: Starting Session 5390 of user root.
Jul 24 03:32:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:32:15 ESB-IIB-03 systemd: Started Session c18340 of user aviation.
Jul 24 03:32:15 ESB-IIB-03 systemd: Starting Session c18340 of user aviation.
Jul 24 03:32:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:32:16 ESB-IIB-03 systemd: Started Session c18341 of user aviation.
Jul 24 03:32:16 ESB-IIB-03 systemd: Starting Session c18341 of user aviation.
Jul 24 03:33:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:33:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:33:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:33:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:33:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:33:15 ESB-IIB-03 systemd: Started Session c18342 of user aviation.
Jul 24 03:33:15 ESB-IIB-03 systemd: Starting Session c18342 of user aviation.
Jul 24 03:33:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:33:16 ESB-IIB-03 systemd: Started Session c18343 of user aviation.
Jul 24 03:33:16 ESB-IIB-03 systemd: Starting Session c18343 of user aviation.
Jul 24 03:33:47 ESB-IIB-03 sshd[91890]: Accepted password for root from 10.128.2.61 port 39578 ssh2
Jul 24 03:33:47 ESB-IIB-03 dbus[1137]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Jul 24 03:33:47 ESB-IIB-03 dbus-daemon: dbus[1137]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Jul 24 03:33:47 ESB-IIB-03 dbus[1137]: [system] Successfully activated service 'org.freedesktop.problems'
Jul 24 03:33:47 ESB-IIB-03 dbus-daemon: dbus[1137]: [system] Successfully activated service 'org.freedesktop.problems'
Jul 24 03:34:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:34:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:34:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:34:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:34:01 ESB-IIB-03 systemd: Started Session 5391 of user root.
Jul 24 03:34:01 ESB-IIB-03 systemd: Starting Session 5391 of user root.
Jul 24 03:34:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:34:15 ESB-IIB-03 systemd: Started Session c18344 of user aviation.
Jul 24 03:34:15 ESB-IIB-03 systemd: Starting Session c18344 of user aviation.
Jul 24 03:34:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:34:16 ESB-IIB-03 systemd: Started Session c18345 of user aviation.
Jul 24 03:34:16 ESB-IIB-03 systemd: Starting Session c18345 of user aviation.
Jul 24 03:34:28 ESB-IIB-03 su: (to aviation) root on pts/1
Jul 24 03:34:28 ESB-IIB-03 systemd-logind: New session c18346 of user aviation.
Jul 24 03:34:28 ESB-IIB-03 systemd: Started Session c18346 of user aviation.
Jul 24 03:34:28 ESB-IIB-03 systemd: Starting Session c18346 of user aviation.
Jul 24 03:35:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:35:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:35:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:35:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:35:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:35:15 ESB-IIB-03 systemd: Started Session c18347 of user aviation.
Jul 24 03:35:15 ESB-IIB-03 systemd: Starting Session c18347 of user aviation.
Jul 24 03:35:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:35:16 ESB-IIB-03 systemd: Started Session c18348 of user aviation.
Jul 24 03:35:16 ESB-IIB-03 systemd: Starting Session c18348 of user aviation.
Jul 24 03:36:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:36:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:36:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:36:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:36:01 ESB-IIB-03 systemd: Started Session 5392 of user root.
Jul 24 03:36:01 ESB-IIB-03 systemd: Starting Session 5392 of user root.
Jul 24 03:36:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:36:15 ESB-IIB-03 systemd: Started Session c18349 of user aviation.
Jul 24 03:36:15 ESB-IIB-03 systemd: Starting Session c18349 of user aviation.
Jul 24 03:36:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:36:16 ESB-IIB-03 systemd: Started Session c18350 of user aviation.
Jul 24 03:36:16 ESB-IIB-03 systemd: Starting Session c18350 of user aviation.
Jul 24 03:37:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:37:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:37:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:37:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:37:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:37:15 ESB-IIB-03 systemd: Started Session c18351 of user aviation.
Jul 24 03:37:15 ESB-IIB-03 systemd: Starting Session c18351 of user aviation.
Jul 24 03:37:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:37:16 ESB-IIB-03 systemd: Started Session c18352 of user aviation.
Jul 24 03:37:16 ESB-IIB-03 systemd: Starting Session c18352 of user aviation.
Jul 24 03:38:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:38:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:38:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:38:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:38:01 ESB-IIB-03 systemd: Started Session 5393 of user root.
Jul 24 03:38:01 ESB-IIB-03 systemd: Starting Session 5393 of user root.
Jul 24 03:38:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:38:15 ESB-IIB-03 systemd: Started Session c18353 of user aviation.
Jul 24 03:38:15 ESB-IIB-03 systemd: Starting Session c18353 of user aviation.
Jul 24 03:38:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:38:16 ESB-IIB-03 systemd: Started Session c18354 of user aviation.
Jul 24 03:38:16 ESB-IIB-03 systemd: Starting Session c18354 of user aviation.
Jul 24 03:39:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:39:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:39:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:39:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:39:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:39:15 ESB-IIB-03 systemd: Started Session c18355 of user aviation.
Jul 24 03:39:15 ESB-IIB-03 systemd: Starting Session c18355 of user aviation.
Jul 24 03:39:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:39:16 ESB-IIB-03 systemd: Started Session c18356 of user aviation.
Jul 24 03:39:16 ESB-IIB-03 systemd: Starting Session c18356 of user aviation.
Jul 24 03:40:01 ESB-IIB-03 systemd: Started Session 5394 of user root.
Jul 24 03:40:01 ESB-IIB-03 systemd: Starting Session 5394 of user root.
Jul 24 03:40:01 ESB-IIB-03 systemd: Started Session 5395 of user root.
Jul 24 03:40:01 ESB-IIB-03 systemd: Starting Session 5395 of user root.
Jul 24 03:40:01 ESB-IIB-03 systemd: Removed slice user-0.slice.
Jul 24 03:40:01 ESB-IIB-03 systemd: Stopping user-0.slice.
Jul 24 03:40:02 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:40:02 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:40:02 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:40:02 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:40:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:40:15 ESB-IIB-03 systemd: Started Session c18357 of user aviation.
Jul 24 03:40:15 ESB-IIB-03 systemd: Starting Session c18357 of user aviation.
Jul 24 03:40:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:40:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:40:16 ESB-IIB-03 systemd: Started Session c18358 of user aviation.
Jul 24 03:40:16 ESB-IIB-03 systemd: Starting Session c18358 of user aviation.
Jul 24 03:40:16 ESB-IIB-03 systemd: Started Session c18359 of user aviation.
Jul 24 03:40:16 ESB-IIB-03 systemd: Starting Session c18359 of user aviation.
Jul 24 03:41:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:41:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:41:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:41:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:41:15 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:41:15 ESB-IIB-03 systemd: Started Session c18360 of user aviation.
Jul 24 03:41:15 ESB-IIB-03 systemd: Starting Session c18360 of user aviation.
Jul 24 03:41:16 ESB-IIB-03 su: (to aviation) root on none
Jul 24 03:41:16 ESB-IIB-03 systemd: Started Session c18361 of user aviation.
Jul 24 03:41:16 ESB-IIB-03 systemd: Starting Session c18361 of user aviation.
Jul 24 03:42:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:42:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:42:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC
Jul 24 03:42:00 ESB-IIB-03 MQSeries: FFST record created in /var/mqm/errors/AMQ3415.0.FDC |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Aug 14, 2025 6:55 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9479
Location: US: west coast, almost. Otherwise, enroute.
|
Other than posting here, what research have you done? What were the results? Did you open a PMR with IBM?
Did you research Probe Id :- XY324103
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| liamf |
| Posted: Thu Aug 14, 2025 8:17 am Post subject: |
|
|
Novice
Joined: 22 Apr 2021
Posts: 15
|
| bruce2359 wrote: |
| Other than posting here, what research have you done? What were the results? |
Hi, bruce. I found a known issue DT430583：
https://www.ibm.com/mysupport/s/defect/aCIKe000000XuJ5/dt430583?language=en_US
but it's reported in V7, i'm not sure if it's still exist in V8, but it can help me grab some clue.
| Quote: |
| Failed to inquire on semaphore set(163842) errno(22) |
From the FMT file i found this "ghost" semaphore was in set 163842, then i search the entire FMT file and found all xecI_E_NO_SUCH_SUBPOOL exceptions were reported with this same set id, so my guess is(correct me if wrong):
1.This set 163842 was left over because we, like what DT430583 said, end the queue manager badly (a system reboot).
2.Just because we restart that specific messageflow quiet frequently and this restart leads to a subpool re-allocation, then somehow this subpool was allocated to set 163842 and cause this issue.
But this still can not tell why it happens every 6 days, but to find a deeper reason would cost to much for now, so i think to proceed a fully restart would be better:
1.using endmqm to end queue manager properly.
2.reboot system.
2.using user mqm under mqm group instead of user avation under mqm group to run startup script.
and let's check if this issue still exist.
Please, let me know if it's the best practice.
To be frankly, i know it's a bad behavior to force reboot system, but we've done this many times before and never seen this issue appears.
Thanks. |
|
| Back to top |
|
|
| markt |
| Posted: Thu Aug 14, 2025 10:46 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 512
|
| So you've got MQ V8 at its original level from 2014 with no fixes applied ... Would be hard to get support to look at that. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 14, 2025 6:03 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9479
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
