| Author |
Message
|
| smdavies99 |
 Posted: Thu Feb 16, 2012 5:56 am Post subject: Clustered Broker Startup problem |
 |
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I'm seeing a strange (to me at least) problem with Broker 7.0.0.3 on a Wondows Server 2008 R2 Cluster.
I've configured WMQ into the cluster and the QMGR fails over correctly.
Now, I've created a broker on one node and following the instructions in the
infoecentre, I've failed the WMQ service over to the other node and then added the broker to
the second node in the cluster.
When I come to start the broker I get the following event logged.
| Code: |
( TESTBKR ) Unsuccessful WebSphere Message Broker service initialization.
The WebSphere Message Broker service could not be registered.
Check that the machine has adequate memory and that the userid and password
used to start the broker are correct and that the
userid is a member of the mqbrkrs group.
|
As far as I can see it, the broker is setup correctly.
Here is the script I used to create the broker and after removing the REM, I used it again for adding the broker to the second node.
| Code: |
@echo off
set acnt=SMD\domwbi7user
set pwd=QAZwsxedc2012
set bkr=TESTBKR
set qmgr=TESTQMGR
set dest=s:\broker\workspace
echo mqsilist
mqsilist
rem mqsicreatebroker -i %acnt% -a %pwd% -q %qmgr% -e %dest% %bkr%
rem mqsiaddbrokerinstance %bkr% -i %acnt% -a %pwd% -e %dest%
echo mqsilist
mqsilist
|
MYBKR is a LOCAL broker to the 2nd node.
TESTBKR is the broker to failover.
| Code: |
C:\temp>mqsireportbroker MYBKR
BIP8927I: Broker Name 'MYBKR'
Install path = 'C:\IBM\MQSI\7.0'
Work path = 'C:\ProgramData\Application Data\IBM\MQSI'
Broker UUID = ''
Process id = '3572'
Queue Manager = 'MYQMGR'
User lil path = ''
User exit path = ''
Active user exits = ''
LDAP principal = ''
LDAP credentials = ''
ICU converter path = ''
Trusted (fastpath) Queue Manager application = 'false'
Configuration change timeout = '300' seconds
Internal configuration timeout = '60' seconds
Statistics major interval = '60' minutes
Operation mode = 'enterprise'
Fixpack capability level = '' (effective level '7.0.0.1')
Broker registry format = 'v7.0'
Administration security = 'inactive'
Multi-instance Broker = 'false'
Shared Work Path = 'none'
Start as WebSphere MQ Service = 'undefined'
HTTP listener port = '7080'
BIP8071I: Successful command completion.
C:\temp>mqsireportbroker TESTBKR
BIP8927I: Broker Name 'TESTBKR'
Install path = 'C:\IBM\MQSI\7.0'
Work path = 'C:\ProgramData\Application Data\IBM\MQSI'
Broker UUID = ''
Process id = '0'
Queue Manager = 'TESTQMGR'
User lil path = ''
User exit path = ''
Active user exits = ''
LDAP principal = ''
LDAP credentials = ''
ICU converter path = ''
Trusted (fastpath) Queue Manager application = 'false'
Configuration change timeout = '300' seconds
Internal configuration timeout = '60' seconds
Statistics major interval = '60' minutes
Operation mode = 'enterprise'
Fixpack capability level = '' (effective level '7.0.0.1')
Broker registry format = 'v7.0'
Administration security = 'inactive'
Multi-instance Broker = 'true'
Shared Work Path = 's:\broker\workspace\mqsi'
Start as WebSphere MQ Service = 'undefined'
HTTP listener port = '7080'
BIP8071I: Successful command completion.
C:\temp>mqsiservice MYBKR
BIPmsgs en_US
Console OEM CP=437, ICU CCSID=5348
Default codepage=ibm-5348_P100-1997, in ascii=ibm-5348_P100-1997
JAVA console codepage name=cp437
Install Path = C:\IBM\MQSI\7.0
Shared Work Path = NOT_HA_ENABLED_BROKER
Local Work Path = C:\ProgramData\Application Data\IBM\MQSI
process id = 3572
broker db userId = SMD\domwbi7user
broker db password =
queue manager = MYQMGR
pubsub migration = no
fastpath Queue Manager = no
configuration timeout = 300
configuration delay timeout = 60
statistics major interval = 60
ComponentType = Broker
Fixpack capability level = (effective level 7.0.0.1)
BIP8071I: Successful command completion.
c:\temp>mqsiservice TESTBKR
BIPmsgs en_US
Console OEM CP=437, ICU CCSID=5348
Default codepage=ibm-5348_P100-1997, in ascii=ibm-5348_P100-1997
JAVA console codepage name=cp437
Install Path = C:\IBM\MQSI\7.0
Shared Work Path = s:\broker\workspace\mqsi
Local Work Path = C:\ProgramData\Application Data\IBM\MQSI
process id = 0
broker db userId = SMD\domwbi7user
broker db password =
queue manager = TESTQMGR
pubsub migration = no
fastpath Queue Manager = no
configuration timeout = 300
configuration delay timeout = 60
statistics major interval = 60
ComponentType = Broker
Fixpack capability level = (effective level 7.0.0.1)
BIP8071I: Successful command completion.
|
I tried to take a service trace for the startup of the two brokers. This failed as the Clusered broker never created the .servicve.trace files.
| Code: |
C:\temp>mqsichangetrace TESTBKR -t -b -l none
BIP8071I: Successful command completion.
C:\temp>mqsireadlog TESTBKR -t -b agent -f -o service_trace.xml
Failed to open trace file
C:\ProgramData\Application Data\IBM\MQSI\common\log\TESTBKR.agent.trace.bin.0
with error The system cannot find the file specified.
BIP8041E: Unable to open the selected file
'C:\ProgramData\Application Data\IBM\MQSI\common\log\TESTBKR.agent.trace.bin.*'.
A product component cannot open the specified file for read or write, for example a log file.
|
I'm doing this in a VM as preparation for doing it on a ustomer site in the near future.
At the moment I'm stumped.
A local broker runs ok on the same machine where the clustered broker does not.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Feb 16, 2012 6:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
Notice the workpath for your failover broker? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 16, 2012 6:50 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Do you mean "S:\broker...."?
That's on the shared drive. The same one that holds the queue manager data.
Should the Local Work path be there as well?
The instructions in the InfoCentre only seems to indicate that the Shared Work Path should be on the drive that fails over. In this case, that is Drive "S:" , Drive "Q:" is the quorum.
this is the page I've been referencing
http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/topic/com.ibm.etools.mft.doc/be13800_.htm
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 16, 2012 7:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
I would have expected that:
| Quote: |
| Work path = 'C:\ProgramData\Application Data\IBM\MQSI' |
to be on the shared drive for an HA broker...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 16, 2012 7:13 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
This seems to indicate otherwise
| Quote: |
When a broker has been enabled for multi-instance mode using the –e flag, the broker workPath is divided between data that is specific to this broker instance, and that which is shared between this broker and any of its instances created using the mqsiaddbrokerinstance command. Data specific to the multi-instance enabled broker is stored in the workPath directory on the local server, whereas the shared data is held in a directory on network storage at the location specified using the –e flag.
|
This is from the InfoCentre page on mqsicreatebroker (for Windows)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 16, 2012 7:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
Sorry I missed the part where it said shared workpath and local workpath...
Did you run the mqsiaddnrokerinstance on both instances?
It looks like only one instance reports the shared path...?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 16, 2012 7:39 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
All the output in the post is from the second system The broker runs perfectly well on the first broker (where it was created)
According to the InfoCentre page on mqsiaddbrokerinstance
| Quote: |
Use the mqsiaddbrokerinstance command to add a broker instance to any additional server on which you require multi-instance support. You must first create a multi-instance enabled broker on one server by using the mqsicreatebroker command.
|
So the pattern is
1) Create the broker with 'mqsicreatebroker'
2) fail over the WMQ service to the second system
3) run mqsiaddbrokerinstance.
As that point, you should be able to start the broker on the second node in the cluster.
OR
maybe not.
I want a setup that works like Broker 6 on Server 2003. You fail over the WMQ/MB service from one node to another. As I see it, this is not how a multinstance broker works.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 16, 2012 7:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
And the first (original) broker was definately created to be multi instance (-e flag)?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 16, 2012 7:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You need to create the first broker as a multi-instance aware broker, as mentioned.
You then need to ensure that you have mqsiaddbrokerinstance, as indicated.
If you want to perform 'failover', like a traditional HA broker, you should review the doc on doing traditional HA config with Broker v7. (note, this still ends up using MI technology and configuration steps...)
If you want to use MI, then you have the broker actually started on both machines at all times, and leave it for the queue manager and the broker to determine which machine is the active instance. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 16, 2012 8:33 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
If you want to perform 'failover', like a traditional HA broker, you should review the doc on doing traditional HA config with Broker v7. (note, this still ends up using MI technology and configuration steps...)
|
Which doc do you mean? Is it the IC91 Doc?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 16, 2012 9:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| smdavies99 wrote: |
| Which doc do you mean? Is it the IC91 Doc? |
No
IC91 is obsolete. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 16, 2012 9:47 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I was actually using that bit of the docs in the first place.
http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/topic/com.ibm.etools.mft.doc/be13800_.htm
This points to the Windows bit.
It seems that the steps in 5. have not worked properly.
I'll go back and revisit (Again)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 17, 2012 4:55 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Well, I went back and deleted everything and did the following
1 failed over the WMQ service to Node 1
2 Created a Broker on Node 1 using the script I included at the outset.
3 manually started the broker - It runs! I can see all the processes owned by the domain user created for the broker
4 shut the broker down
5 created a generic service, select the broker from the list - This all went well.
Now I run into problems.
I can't seem to place a dependency on the WMQ service.
In the Broker Service properties-Dependencies tab I can't find a way to make this service dependent on the other WMQ Service. I'm obviously doing something wrong.
I started the Broker service and it brought up the Broker fine on Node 1.
So I took the BrokerService is offline on Node 1.
I failed the WMQ service over to node 2. it came up as expected.
I then failed the Broker service over to Node 2. It tried to start but failed.
The only things in the event log are
1) A message saying that the component verification for the broker is starting
This is followed less than a second later by
2) A message saying that the component verification for the broker has failed.
Nothing else. Nada, zip.
I feel like taking a  to the whole box at the moment.
The only saving grace is that I'm not trying to do this in front of the customer.
Any ideas about where to go next are most welcome.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 17, 2012 5:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| Any ideas about where to go next are most welcome. |
Any OS other than Windoze? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 17, 2012 6:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
| smdavies99 wrote: |
| Any ideas about where to go next are most welcome. |
Any OS other than Windoze? |
I'd really like to use Linux. I have this fully documented for a DRBD cluster and broker. I can configure it all in an hour easy.
Sadly, the customer has bought into the Windows myth.
I'm sure there is a simple explanation to my problem. The trick is finding it.
Server 2003 is much nicer than 2008. I shudder to think how much Microsoft are going to mess it all up with Windows 8 Server Edition.
When you create a directory but you do don't have the rights to read or write to the directory you just created, I have to think that something is very wrong with their security model.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
