Author |
Message
|
flsb |
Posted: Mon Oct 14, 2019 5:43 pm Post subject: 1300 character Message not transmitted |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
Hi everyone,
My issue is similar to the thread "Messages stalled on XMITQ" but it is slightly different..
i usually can google and found the solution to my MQ problems but this issue really had me running in circles..
No FDC file created.. running MQ 9.1.0.3 in Linux RHEL7
Error in AMQERR log
Quote: |
AMQ9259E: Connection timed out from host 'xxx'.
EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 60 seconds call timed out. Check to see why data was not
received in the expected time. Correct the problem. Reconnect the channel, or
wait for a retrying channel to reconnect itself.
AMQ9999E: Channel 'xxx.CHANNEL' to host 'xxx' ended
abnormally.
EXPLANATION:
The channel program running under process ID 26742 for channel
'xxx.CHANNEL' ended abnormally. The host name is 'xxx'; in
some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. |
Other channels to the same location is running OK, and only this channel is with messages stuck in XMITQ.
It looks like the message is not able to fully transmit to the destination.
Unable to find amqsbcg in /opt/mqm/bin
We split the 1.3k characters message to smaller chunks and is able to send through but with altering the channel batchsz to 1.
Another weird thing is we currently migrated to MQ v9.1 from MQv5.3 (AIX).
While it works perfectly in MQv5.3 AIX previously.
WE have few parties with different channels and destination, but out of 10-20 parties, only 2 parties are having this issue..
I searched high and low and it all asked to contact network side.. but the channel is able to ping, it just data stuck in XMITQ. Network has found no issue at their end.
Would appreciate anybody that has experienced this before.. |
|
Back to top |
|
|
bruce2359 |
Posted: Mon Oct 14, 2019 6:24 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
Did you use the exact same v5.3 channel definitions in the 9.1 qmgrs?
Post the sender side channel definition here. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
flsb |
Posted: Mon Oct 14, 2019 6:47 pm Post subject: |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
we create using the same definition..
DEFINE CHANNEL(xxx.CHANNEL) CHLTYPE(SVR) TRPTYPE(TCP) XMITQ(xxx.TRAN.QUEUE) DISCINT(0) CONNAME('xxx')
Quote: |
CHANNEL(xxx.CHANNEL) CHLTYPE(SVR)
ALTDATE(2019-10-12) ALTTIME(09.18.17)
BATCHHB(0) BATCHINT(0)
BATCHLIM(5000) BATCHSZ(50)
CERTLABL( ) COMPHDR(NONE)
COMPMSG(NONE) CONNAME(xxx)
CONVERT(NO) DESCR( )
DISCINT(0) HBINT(300)
KAINT(AUTO) LOCLADDR( )
LONGRTY(999999999) LONGTMR(1200)
MAXMSGL(4194304) MCANAME( )
MCATYPE(PROCESS) MCAUSER( )
MODENAME( ) MONCHL(QMGR)
MSGDATA( ) MSGEXIT( )
NPMSPEED(FAST) PASSWORD( )
PROPCTL(COMPAT) RCVDATA( )
RCVEXIT( ) RESETSEQ(NO)
SCYDATA( ) SCYEXIT( )
SENDDATA( ) SENDEXIT( )
SEQWRAP(999999999) SHORTRTY(10)
SHORTTMR(60) SSLCAUTH(REQUIRED)
SSLCIPH( ) SSLPEER( )
STATCHL(QMGR) TPNAME( )
TRPTYPE(TCP) USEDLQ(YES)
USERID( ) XMITQ(xxx.TRAN.QUEUE)
|
|
|
Back to top |
|
|
exerk |
Posted: Mon Oct 14, 2019 11:21 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006 Posts: 6339
|
1300 characters, but what is the actual message size? Do you know what batch size is set at the receiving end? And do you really want your channels to run continuously once started, i.e. with no DISCINT? _________________ It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
Back to top |
|
|
flsb |
Posted: Tue Oct 15, 2019 12:53 am Post subject: |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
it is just plain text..
total 1300 characters...
default MAXMSGL is 4Mb... i don't think the message can exceed 4Mb even with headers and encryption etc..
we have tried increasing to 10mb on Qmgr, channel and queue level.. and still fail to send..
The batchsz for both channel is set to 50... we tried to set as 1 but still failed to send.. |
|
Back to top |
|
|
bruce2359 |
Posted: Tue Oct 15, 2019 4:15 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
flsb wrote: |
it is just plain text..
total 1300 characters...
default MAXMSGL is 4Mb... i don't think the message can exceed 4Mb even with headers and encryption etc..
we have tried increasing to 10mb on Qmgr, channel and queue level.. and still fail to send..
The batchsz for both channel is set to 50... we tried to set as 1 but still failed to send.. |
How/where do you encrypt the message? In an MQ exit?
Please use one of the supplied utilities to display the message in the XMITQ. Post the results here.
The supplied utilities are found MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin (32-bit versions) MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin64 (64-bit versions) _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
hughson |
Posted: Tue Oct 15, 2019 1:45 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013 Posts: 1948 Location: Bay of Plenty, New Zealand
|
The error you supply from the AMQERR01.LOG is from the SVR side of the channel I assume? Please can you also supply the error in the AMQERR01.LOG from the other end of the channel (and also tell us whether it is a RCVR or a RQSTR).
Thanks,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
|
gbaddeley |
Posted: Tue Oct 15, 2019 3:28 pm Post subject: Re: 1300 character Message not transmitted |
|
|
Jedi Knight
Joined: 25 Mar 2003 Posts: 2527 Location: Melbourne, Australia
|
flsb wrote: |
Hi everyone,
AMQ9259E: Connection timed out from host 'xxx'.
EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 60 seconds call timed out. Check to see why data was not
received in the expected time. Correct the problem. Reconnect the channel, or
wait for a retrying channel to reconnect itself.
AMQ9999E: Channel 'xxx.CHANNEL' to host 'xxx' ended
abnormally.
... WE have few parties with different channels and destination, but out of 10-20 parties, only 2 parties are having this issue....
.... I searched high and low and it all asked to contact network side.. but the channel is able to ping, it just data stuck in XMITQ. Network has found no issue at their end.... |
select() timeout usually means that the network connection was lost at the TCP socket level. Nothing you can do in MQ will fix this. It is a network issue. Request your network team to run a packet trace between the two IPs. Check that firewalls or switches are not dropping the connection. _________________ Glenn |
|
Back to top |
|
|
flsb |
Posted: Tue Oct 15, 2019 5:53 pm Post subject: |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
Quote: |
How/where do you encrypt the message? In an MQ exit?
The supplied utilities are found MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin (32-bit versions) MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin64 (64-bit versions)) |
is this windows? Im running in RHEL7
Installation path is /opt/mqm/bin and i am unable to locate this amqsbcg
Can i copy another amqsbcg from MQ7.5 into this server?
I encrypt the data on application level before putting the message into qremote |
|
Back to top |
|
|
flsb |
Posted: Tue Oct 15, 2019 5:55 pm Post subject: |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
hughson wrote: |
The error you supply from the AMQERR01.LOG is from the SVR side of the channel I assume? Please can you also supply the error in the AMQERR01.LOG from the other end of the channel (and also tell us whether it is a RCVR or a RQSTR).
Thanks,
Morag |
AMQERR01 at receiver's end
AMQ9788: Slow DNS lookup for address 'xxx'.
EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.
If the address was an IP address then the slow operation was a reverse DNS
lookup. Some DNS configurations are not capable of reverse DNS lookups and some
IP addresses have no valid reverse DNS entries. If the problem persists,
consider disabling reverse DNS lookups until the issue with the DNS can be
resolved.
----- amqcrhna.c : 785 --------------------------------------------------------
10/9/2019 10:34:37 - Process(6524.13237038) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(KULDCPWFLS01) Installation(FLS01)
VRMF(8.0.0.4) QMgr(queue.manager.aia)
AMQ9259: Connection timed out from host 'xxx'.
EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 360 seconds call timed out. Check to see why data was
not received in the expected time. Correct the problem. Reconnect the channel,
or wait for a retrying channel to reconnect itself.
----- amqccita.c : 4452 -------------------------------------------------------
10/9/2019 10:34:37 - Process(6524.13237038) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(KULDCPWFLS01) Installation(FLS01)
VRMF(8.0.0.4) QMgr(queue.manager.aia)
AMQ9999: Channel 'xxx.CHANNEL' to host 'xxx' ended abnormally.
EXPLANATION:
The channel program running under process ID 6524(1912) for channel
'xxx.CHANNEL' ended abnormally. The host name is 'xxx'; in
some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide. |
|
Back to top |
|
|
flsb |
Posted: Tue Oct 15, 2019 5:57 pm Post subject: Re: 1300 character Message not transmitted |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
gbaddeley wrote: |
select() timeout usually means that the network connection was lost at the TCP socket level. Nothing you can do in MQ will fix this. It is a network issue. Request your network team to run a packet trace between the two IPs. Check that firewalls or switches are not dropping the connection. |
Receiver's side has run WIreshark and noted packet is receiving..
Firewall at my side is showing packet is sending..
it just seems like the message is not committing at receiver's end completely.. |
|
Back to top |
|
|
fjb_saper |
Posted: Tue Oct 15, 2019 8:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003 Posts: 20729 Location: LI,NY
|
Here is your problem
Quote: |
EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.
If the address was an IP address then the slow operation was a reverse DNS lookup. Some DNS configurations are not capable of reverse DNS lookups and some IP addresses have no valid reverse DNS entries.
If the problem persists, consider disabling reverse DNS lookups until the issue with the DNS can be resolved. |
Have the receiving side disable reverse DNS lookup, and also disable it on your side.
Enjoy _________________ MQ & Broker admin |
|
Back to top |
|
|
flsb |
Posted: Wed Oct 16, 2019 12:15 am Post subject: |
|
|
Apprentice
Joined: 01 Apr 2010 Posts: 43
|
fjb_saper wrote: |
Here is your problem
Quote: |
EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.
If the address was an IP address then the slow operation was a reverse DNS lookup. Some DNS configurations are not capable of reverse DNS lookups and some IP addresses have no valid reverse DNS entries.
If the problem persists, consider disabling reverse DNS lookups until the issue with the DNS can be resolved. |
Have the receiving side disable reverse DNS lookup, and also disable it on your side.
Enjoy |
i will try to get them to disable reverse DNS lookup..
ALTER QMGR REVDNS(DISABLED)
but the problem only happen on one of the channel where large 1.3k dat is sent..
it is ok after we split the message into smaller chunks and batchsz(1) |
|
Back to top |
|
|
HubertKleinmanns |
Posted: Wed Oct 16, 2019 12:56 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004 Posts: 732 Location: Germany
|
flsb wrote: |
i will try to get them to disable reverse DNS lookup..
ALTER QMGR REVDNS(DISABLED)
but the problem only happen on one of the channel where large 1.3k dat is sent..
it is ok after we split the message into smaller chunks and batchsz(1) |
Hmm, 1.3 k is not really large. I worked with MQ Systems, which transferred messages of several 100 MB (with messsage segmentation ) within one batch without problems . Even the MQ header has about 500 b.
When you split the message:
- Which size did these spitted messages have?
- Did it work when they where sent within one batch (e. g. 130 b and BATCHSZ 100)?
- Could you sent the bulk of splitted messages again and deliver the output of the following command before and after the transfer:
Code: |
dis chs(<your Sender channel>) all |
Some further questions:
- Does 'xxx' stand for an IP address or a host name?
- What's about the log size at your parties side? The content of "qm.ini" of your parties would be helpful.
- What's about the load on the parties side?
In Addition: I don't think, that DNS use used on a runnung channel. I assume, the DNS resolution only occurs when the channel starts. _________________ Regards
Hubert |
|
Back to top |
|
|
hughson |
Posted: Wed Oct 16, 2019 1:08 am Post subject: |
|
|
Padawan
Joined: 09 May 2013 Posts: 1948 Location: Bay of Plenty, New Zealand
|
I concur that the DNS problem is a red herring - only happens at channel start, not part of message delivery.
The fact that both ends of the channel are timing out (as per the error messages you have provided), this suggests that some part of the networking stack is not happy with your socket. I would be inclined to begin with the firewall as that would be a very common cause of such behaviour.
Perhaps there is something in your message it doesn't like the look of.
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
|
|