MQSeries.net :: View topic - Recurring QM Outage that Gens Probid = XC130003 and ZX005022

belchman · Posted: Tue Apr 11, 2006 6:15 am Post subject:

Sorry, I am not getting email notify's of your posts. Haven't been here in a few days.

Let me see if I can respond to all of this in an orderly fashion.

mvic asks "What do your network admins say about this?",

NW guys say it is an app issue. They do not see NW probs. We have had sniffer on line between both nodes.

fjb_saper asks "Could it be that you run out of bandwidth at those times ?"
Sorry jefflowrey

, for misinformation. My "constantly" usage is grossly over-exaggerated. It does appear that the log spamming correlates to a batch application that fires off a bunch of accumulated datagrams. I can do some research to ID volumes.

fjb_saper asks "How about faulty switch... ?"

I have to assume that NW sniffer would have caught that.

Reminder:

The node spamming the logs is the least of my worries. The real problem is that we frequently experience and outage on the QM that last 5 - 15 secs. All client conns over SVRCONNS are booted and forced to reconnect.

This spamming node is a suspect simply because it is spamming the logs. It could be a false lead. But IMHO, if it is spamming it is causing problems. My "possibly hairbrained" theory

, is that this node is acting up and MQ is trying to be patient with it. MQ maintains its patience until it blows its stack and boots everyone out of the playground (closes all conns) so it can meditate and heal itself.

Here is another piece of info that could support my theory. The memory usage of the amqpcsea constantly increases over time until our Sunday bounce of the MQ host. The curve for memory utilization is a straight line with a 30 - 40 degree slope. Today at 7:40 hrs, mem usage was 46M. At 8:40, it was 46.8M. So we have a .8M/hr increase constantly all week until bounce.

If this batch process is putting to clustered queues, could this be causing what looks like a memory leak in amqpcsea? And could amqpcsea situation be causing client disconnect issue?

Once again! I appreciate all of your input into this discussion. I welcome any other "hairbrained theories" about root cause.