ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 07 / 2009

This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page 1, 2  Next
 Challenge Question - 07 / 2009 « View previous topic :: View next topic » 
Author Message
Challenger
PostPosted: Thu Jul 02, 2009 12:05 am    Post subject: Challenge Question - 07 / 2009 Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Hello;

After a couple months of absence, I am back with a challenge for you:

You have an MQ cluster present in 2 locations. A & B
You are going to physically move machines currently in location A to location C
You do not want any interruption of service.
Some functionality is only provided by machines in A/C
Your full repositories are in A

Precautions you have taken prior to the move:
- Create a 3rd FR in location B
- Move a PR from A to C ahead of time (no service interruption)

Challenge: How would you look at shutdown and restore operations for the planned move to minimize problems?

good luck !
Back to top
View user's profile Send private message
Challenger
PostPosted: Fri Jul 10, 2009 10:20 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

OK Everybody. This is not a trick question nor is it an exam question... (If it is I am truly flattered as I never took the exam and have absolutely no idea to its content).

This comes from a real life event that I hoped to be quite smooth but presented some problems....

What I would like for you all is to write up, how you would do it so that I can get some information/confirmation as to what I did wrong, what I might have missed and just plain ideas on how to do it differently...

Thanks in advance for your participation.
Back to top
View user's profile Send private message
Michael Dag
PostPosted: Sat Jul 11, 2009 4:29 am    Post subject: Reply with quote

Jedi Knight

Joined: 13 Jun 2002
Posts: 2602
Location: The Netherlands (Amsterdam)

you physically moved machines, did IP addresses change too?
please describe the problems you encountered during your 'move'.
_________________
Michael



MQSystems Facebook page
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Challenger
PostPosted: Sat Jul 11, 2009 3:05 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Michael Dag wrote:
you physically moved machines, did IP addresses change too?
please describe the problems you encountered during your 'move'.

Due to the fact that we are not changing IPs and the the cluster spans both sides (A & C) there is some natting involved. Of course we use DNS name resolution and help it along with entries in the /etc/hosts file where needed (lone box predeployed to C).

Some of the problems we encountered were that once we suspended (force) the boxes in A some routing problems popped up to the extend that we had to force shutdown of some channels on some of the remaining qmgrs (i.e. from B to C) for the destination resolution to work as expected in the cluster (and not with 1 in 3 or 2 in 3)

After the move and restore of all members to the cluster everything was fine again...
Back to top
View user's profile Send private message
exerk
PostPosted: Sun Jul 12, 2009 2:44 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

With 20/20 hindsight (always the clearest vision) I'd not just have suspended the queue managers from the cluster, but removed them, then put them back in post-move. Of course, that wouldn't have worked if the were all being moved in one go, unless it was a reasonable number of boxes, and you could stand 50% being off-line.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
Challenger
PostPosted: Sun Jul 12, 2009 3:36 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Hindsight seems to always be 20/20...

What I am expecting from you guys is a step by step guide of how you would have gone about quiescing the 2 boxes in A and moving them to C.
(there are 2 qmgrs per box).

As stated this is to give me ideas and find out what you would have done differently... So far noone has dared venture there... so no brownie points...
Back to top
View user's profile Send private message
exerk
PostPosted: Sun Jul 12, 2009 4:58 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

1. Demote both the FR's in A.
2. On one of the boxes above, drop the queue managers from the cluster.
3. Move the box into C.
4. Bring up the queue managers on the box just moved, promote the one that was originally a PR back to PR status, linking to the 3rd FR previously created, and put both servers back into the cluster.
5. On each remaining server, in A, drop the queue managers from the cluster, move the server, bring up the queue managers, and put them back in the cluster.

All the above done in the correctly documented fashion of course, although I would slightly deviate by not deleting the channels when they had quiesced/stopped. I always expect problems with moves, especially when network people tell me there'll be no issue, and as my experience with channel issues, just altering CONNAME's has produced the sort of issues you encountered, where clusters are concerned, I'd rather take the hit on time and effort that may not have been needed, rather than go through the pain when issues arise.

That's about as step by step as I would make it.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.


Last edited by exerk on Wed Jul 29, 2009 4:10 am; edited 1 time in total
Back to top
View user's profile Send private message
Challenger
PostPosted: Wed Jul 15, 2009 5:46 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Can't believe that this is so far the only answer.
Nobody to propose different/additional steps or a different order?

Come on, share your past experiences in physically moving part of an MQ cluster...
Back to top
View user's profile Send private message
AkankshA
PostPosted: Tue Jul 28, 2009 10:05 pm    Post subject: Reply with quote

Grand Master

Joined: 12 Jan 2006
Posts: 1494
Location: Singapore

Hi,
My attempt

Currently the cluster is at location A & B and as specified each location has 2 boxes and each box has 2qmgrs.

So current cluster has 2*2+2*2 = 8 qmgrs
Both the FRs reside on A and i assume they reside on diff boxes else the cluster would have gone for a toss in case of a single machine crash...

Also its told that IP address would remain same and DNS lookup is used for too.

As per current set up A has 2 PR 2 FR and B has 4 PR

Steps I would followCreate a FR in B

    Move a PR from A to C

  1. First demote one FRs in A to PR status

  2. Ensure that the qmgrs at B, who were connected to this qmgr are now connected to another FR and channels are up and running.

  3. Suspend this qmgr from cluster so that no new requests come along.

  4. Once the qdepth of all application q is 0, stop/delete the cluster channels and get the qmgr out of cluster

  5. Create the qmgr on C box as FR

  6. Connect it to the FR in B, create channels etc

  7. Ensure that the same is functioning and catering to application requests.

  8. Ensure the qmgrs on B are able to connect to cluster and their auto defined channels are in running status indeed.

    Now, one box at A is moved successfully

  9. Lets repeat these steps for the other FR in A too.

    Now location A is left with only 1 PR and C has already got 2FR+1PR.

    Cross check that the cluster is working fine, specially the qmgrs on B and their auto defined channels.

  10. Now demote the FR in B to PR

  11. Move PR in A to C

_________________
Cheers
Back to top
View user's profile Send private message Visit poster's website
Challenger
PostPosted: Wed Jul 29, 2009 3:57 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Thanks Akanksha.

The premise is a wee tiny bit different.
You start with 10 qmgrs originally (5 boxes).
you have moved one box in advance to C.

So you are left with 2 boxes to move at the same time. (They go on a truck with a lot of other non MQ boxes), and they have FR's on it.
Previous to this last step of the move you have 3 FRs in the cluster: 2 in A and 1 in B.

You are not going to create any additional qmgr... or if you would like to do so for the move, you better come up with an explanation as to why...

Yes your cluster can shift around FR's and PR's demote FRs and promote PRs.

So I'd have to say nearly there but not quite.
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Jul 29, 2009 4:10 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

Challenger wrote:
...So you are left with 2 boxes to move at the same time...


A new bit of information methinks...however, it doesn't affect my earlier post, merely slightly adjusts it:

Quote:
1. Demote both the FR's in A.
2. Drop all 'A' queue managers from the cluster.
3. Move the boxes into C.
4. Bring up the queue managers on the boxes just moved, promote one that was originally a PR back to PR status, linking to the 3rd FR previously created, and put all queue managers back into the cluster, redefining CLUSSDR's on the queue managers which were tied to the original FR, i.e. the one that was demoted, and is now remaining as a PR.

_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jul 29, 2009 4:51 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

1) move all cluster queues hosted in A to qmgrs in B&C
2) reconnect all applications talking to A to qmgrs in B or C
3) remove all qmgrs in A from the cluster entirely
4) move the boxes.
5) readd all qmgrs that were in A back to the cluster
6) migrate cluster queues and app connections back to qmgrs that used to be in A.

There's no way to "avoid an interruption in service" for applications that are connected to qmgrs in A. They have to disconnect and reconnect at least once, and twice if you need to move their load back to the moved machines. (if you don't, why did you move the machines in the first place?)
Back to top
View user's profile Send private message
Challenger
PostPosted: Wed Jul 29, 2009 7:00 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

mqjeff wrote:
1) move all cluster queues hosted in A to qmgrs in B&C
2) reconnect all applications talking to A to qmgrs in B or C
3) remove all qmgrs in A from the cluster entirely
4) move the boxes.
5) readd all qmgrs that were in A back to the cluster
6) migrate cluster queues and app connections back to qmgrs that used to be in A.

There's no way to "avoid an interruption in service" for applications that are connected to qmgrs in A. They have to disconnect and reconnect at least once, and twice if you need to move their load back to the moved machines. (if you don't, why did you move the machines in the first place?)

1), 2) and 6) are taken care of by the early move of the box to C.
The box in C caries all the apps and queues in A.
So we are left with 3, 4 and 5.
What I was looking for was an easy way not to have to remove the qmgr from the cluster entirely as it would just be considered down in maintenance and have a minor change to the cluster receiver channel...once in C.

Maybe that should just not be done so...
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Jul 29, 2009 7:34 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

Challenger wrote:
...and have a minor change to the cluster receiver channel...once in C...


Experience has taught me that 'minor' changes to CLUSRCVR's tends to generate major headaches...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
AkankshA
PostPosted: Wed Jul 29, 2009 9:10 pm    Post subject: Reply with quote

Grand Master

Joined: 12 Jan 2006
Posts: 1494
Location: Singapore

I still would go by my earlier post, works with this new piece of info too :
 Moved PR box and added up again in cluster
 Demote the FRs in A to PR
 Suspend them from cluster, bring them down and move the boxes
 Once boxes reached C, bring up box
 Add the qmgrs in cluster as FR
 And demote the FR of B to PR status


In a case wherein, you want to avoid huge time increase in maintenance documents (which we all strive for), the probable steps would be(not a self tested method though)

 Make an FR in B
 Suspend A’s qmgr from cluster
 Bring A’s box down afterwards
 Once movement completes, start the qmgrs
 Resume qmgrs back in cluster
 Make B’s FR back to PR status
_________________
Cheers
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 07 / 2009
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.