| Author |
Message
|
| yanaK |
 Posted: Thu May 28, 2020 3:22 pm Post subject: xcsCreateThread error |
 |
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
I am getting a `Resource temporarily unavailable` error from an IBM MQ queue manager. I tried changing my kernel parameters and I still see its there. Here is the top of my FDC:
| Date/Time :- Mon May 25 2020 23:58:03 PDT |
| UTC Time :- 1590389941.820409 |
| UTC Time Offset :- -420 (PST) |
| Host Name :- x0321 |
| Operating System :- Linux 3.10.0-1127.el7.x86_64 |
| PIDS :- 5724H7230 |
| LVLS :- 7.1.0.7 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (1) |
| Probe Id :- XC035040 |
| Application Name :- MQM |
| Component :- xcsCreateThread |
| SCCS Info :- lib/cs/unix/linux_2/amqxprmx.c, 1.176.1.1 |
| Line Number :- 1974 |
| Build Date :- Nov 4 2015 |
| CMVC level :- p710-007-151104 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 3732 (mqm) |
| Real UserID :- 3732 (mqm) |
| Program Name :- runmqlsr |
| Arguments :- -r -m MQPSEPX -t TCP -p 1417 |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 68029 |
| Process(Thread) :- 68029 |
| Thread :- 1 |
/xcsCreateThread
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| ('11 - Resource temporarily unavailable' from pthread_create.) |
| FDCSequenceNumber :- 6 |
| Arith1 :- 11 (0xb) |
| Comment1 :- '11 - Resource temporarily unavailable' from |
| pthread_create. |
|
I am on RHEL 7 and my kernel params are:
fs.file-max = 583180
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
kernel.sem = 500 256000 250 1024
net.ipv4.tcp_keepalive_time = 300
kernel.threads-max = 95346
This is intermittent and whenever this happens I cannot runmqsc (I get
AMQ9508: Program cannot connect to the queue manager.)
I am stuck on this - any help is appreciated.
Thanks |
|
| Back to top |
|
 |
| PaulClarke |
| Posted: Thu May 28, 2020 4:02 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
|
| Back to top |
|
|
| yanaK |
| Posted: Thu May 28, 2020 4:25 pm Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
Yes I changed the params based on that article (and the existing queue managers we've) - if you see all my kernel params are above those numbers (although it will depend on clients and OS I think).
I still see the issue intermittently - so need to know what's causing it.
What do you mean by how many clients? This queue manager is a part of a cluster (of 5 queue managers) and the cluster has over 10k clients |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu May 28, 2020 4:53 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
Are you seeing the same error on all 5 qmgrs in the cluster?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| yanaK |
| Posted: Thu May 28, 2020 5:04 pm Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
| No - this is a new one we added (replacing an old one) |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu May 28, 2020 5:19 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
So, am I right in thinking that you are trying to run a lot of clients in to your queue manager ? Are you running these channels in the listener or are you using channel pool processes ? I would have expected new threads be continually created in the agent process or in the channel pool process AMQRMPPA. Why is it that RUNMQLSR is complaining about creating a thread ? I would recommend that you run with MQ with channel pool processes active.
Can you give us a feel for how many threads and processes you have running at the time of the failure.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| yanaK |
| Posted: Thu May 28, 2020 5:28 pm Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
So, am I right in thinking that you are trying to run a lot of clients in to your queue manager ?
Theoretically yes (but I'd expect it to get distributed over all the q mgrs)
Are you running these channels in the listener or are you using channel pool processes ?
How do I know?
I would have expected new threads be continually created in the agent process or in the channel pool process AMQRMPPA. Why is it that RUNMQLSR is complaining about creating a thread ? I would recommend that you run with MQ with channel pool processes active.
Can you tell how?
Can you give us a feel for how many threads and processes you have running at the time of the failure.
What command do I use to get that value?
Also when I run this I get 0:
| Code: |
$ echo "dis conn(*) all" | runmqsc MQPOSEPD|grep EXTCONN|wc -l
|
Can you suggest what other info I can generate to help troubleshoot?
Thanks to all for the help. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu May 28, 2020 5:37 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
I believe the commands to display thread information along with processes varies depending on the version of Linux you are using. I seem to remember that originally Linux used to show threads by default in the ps -ef command which confused people because they thought they had 10 listeners running etc. But I think you can still ask for threads using -L or -T or similar. You can also use something like top to display thread information.
Essentially what I am interested in is whether the bulk of your channels are running in RUNMQLSR or do you have lots of AMQRMPPA processes running ?
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| yanaK |
| Posted: Thu May 28, 2020 6:21 pm Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
| Code: |
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
75049 mqm 20 0 172556 2492 1624 R 0.7 0.0 0:00.79 top
15501 mqm 20 0 1342584 40068 35588 S 0.3 0.7 0:46.05 amqzlaa0
100253 mqm 20 0 1303784 57096 31024 S 0.3 1.0 2:26.63 amqrmppa
107590 mqm 20 0 1342432 33640 29592 S 0.3 0.6 0:23.62 amqzlaa0
113602 asxk 20 0 186908 3812 1272 S 0.3 0.1 0:05.40 sshd
1 root 20 0 202360 6832 3928 S 0.0 0.1 81:10.58 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:16.17 kthreadd
4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:54.72 ksoftirqd/0
7 root rt 0 0 0 0 S 0.0 0.0 0:34.72 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 18:13.82 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:05.98 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:07.43 watchdog/1 |
I see this when I run top |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu May 28, 2020 8:28 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
If you changed you kernel parms but the queue manager was left running you may have to do the following:
- logon to the box (sudo to mqm)
- stop the queue manager (closes the old shell)
- start the queue manager from the new shell (with the new kernel parms)
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yanaK |
| Posted: Thu May 28, 2020 10:54 pm Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
|
| Back to top |
|
|
| tczielke |
| Posted: Fri May 29, 2020 3:28 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
What is your nproc setting for the mqm id? You can find that with "ulimit -u" when run under the mqm id.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 29, 2020 4:26 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
It will load the new values into the current shell. Is that the same shell the queue manager was started in?
Beware also of other running queue managers. They might suddenly starve for resources because you started a queue manager in the new shell not submitted to the restrictions of the ones running in the old shells.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yanaK |
| Posted: Fri May 29, 2020 8:20 am Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
[i]ulimit -u gives 47673
[/i]
There is only one queue manager in that box.
It will load the new values into the current shell. Is that the same shell the queue manager was started in?
No I started the queue manager few weeks back
Last edited by yanaK on Fri May 29, 2020 9:47 am; edited 1 time in total |
|
| Back to top |
|
|
| yanaK |
| Posted: Fri May 29, 2020 9:18 am Post subject: |
|
|
Acolyte
Joined: 28 May 2020
Posts: 69
|
Also I see in FDC:
xcsCreateThread rc=xecP_E_PROC_LIMIT
Does that correspond to nproc? Is there a way to find out how much processes were running at the time of the failures? |
|
| Back to top |
|
|
|
|
