| Author |
Message
|
| deepak_paul |
 Posted: Mon Sep 05, 2011 10:25 pm Post subject: Message Broker Build using Ant/Hudson/SubVersion |
 |
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
All,
We plan to implement the WMB build process completely automated with using Ant and Hudson. So far we have defined the Build process as mentioned below, [this Build process is triggered whenever developer checks in a project code in SubVersion]
1. Get the latest project code from SubVersion.
2. Create BAR file using MQSI command mqsicreatebar. [executed as an Ant task on Hudson Server]
3. Apply env specifics using MQSI command mqsiapplyoverride.[executed as an Ant task on Hudson Server]
4. Ftp the generated bar file to WMB server.
5. Do a local deploy to WMB server using MQSI command mqsideploy
6. Send email notification to Build Admin.
A few questions:
1. For those workbench commands(mqsicreatebar and mqsiapplybaroverride), I strongly believe that we need to install Broker Toolkit on Hudson server in order to create BAR files from the code which Hudson gets from Subversion. As Broker toolkit comes with no cost, we don’t mind installing it on Hudson server but before that, just wondering to see if there is a way we can still create BAR files without toolkit installation?
2. Just wanted to know if there are any disadvantages in using mqsideploy for remote broker deploys. (As I go through older posts, I see that this approach is not recommended or at least not a best practice)
3. Is there way we can configure Hudson to (s)ftp the generated BAR files to the remote WMB server and call a shell script in the remote server that executes mqsideploy.
Thanks in advance.
_________________
Regards
Paul |
|
| Back to top |
|
 |
| WGerstma |
| Posted: Tue Sep 06, 2011 1:43 am Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 55
|
Thanks for going to the older posts. You are correct in most aspects. To your questions:
1) mqsicreatebar needs a toolkit installation to run Eclipse in headless mode.
2) mqsideploy for remote is problematic in some aspects. We use cygwin and sftp to transfer the BAR files to the broker server. There a cron is checking the transfer folder for new files and deploys them
3) We have a cron; but it should also be possible to call a script- |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Sep 06, 2011 4:29 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Second WGerstma's opinion: use mqsideploy locally not remotely. Hudson has an agent that can run mqsidelpoy on the local machine.
Also know that this has been done many times before the way you are approaching it. It is possible and proven.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Tue Sep 06, 2011 9:58 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
Thanks for your input, WGerstma.
| lancelotlinc wrote: |
| ... Hudson has an agent that can run mqsidelpoy on the local machine... |
Thanks lancelotlinc for your inputs. I am exploring all the options in Hudson. You meant slave agents? ...Can you please educate me on how we can create agents in Hudson that can run mqsideploy on the WMB server. Indeed, i am looking for some configuration in Hudson that enables the deployment of the BAR file in remote WMB server.
Any external link is also appreciated.
_________________
Regards
Paul |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Tue Sep 06, 2011 6:40 pm Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
One more question with regard to Change Management:
When we get a change request and modify the common components(subflows)(which are referenced by other Message flow projects) in the workspace, Is there a way we can make sure that all referencing message flow projects also get updated and create respective BAR files?
In other words, when i update my common subflows, i need the following steps to be done as part of Change Management in the Build Process.
1. Get the list of all Message flow/set projects that have reference to the changed common component(an instance - subflow). I believe that .project file in each broker project(message flow/set) has reference project info in it.
| Code: |
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>Test</name>
<comment></comment>
<projects>
<project>MFP_ESB_COMMON_COMPONENTS</project>
<project>MSP_ESB_COMMON_ExceptionMsgDef</project>
<project>SMC_ErrorHandling_SFP</project>
</projects>
<buildSpec>
<buildCommand>
<name>com.ibm.etools.mft.esql.lang.esqllangbuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>com.ibm.etools.mft.mapping.builder.mappingbuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>com.ibm.etools.mft.flow.msgflowxsltbuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>com.ibm.etools.mft.flow.msgflowbuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>com.ibm.etools.mft.bar.ext.barbuilder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>com.ibm.etools.mft.flow.messageflownature</nature>
<nature>com.ibm.etools.mft.bar.ext.barnature</nature>
</natures>
</projectDescription> |
2. Build all referencing message flow/set projects and create BAR files automatically.
We can implement these steps using batch or shell script depending on the environment but i am trying to find out if there is any option in Hudson that can trigger the respective builds where there is a change in the references.
Thanks in advance.
_________________
Regards
Paul |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Sep 07, 2011 4:33 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| deepak_paul wrote: |
Thanks for your input, WGerstma.
| lancelotlinc wrote: |
| ... Hudson has an agent that can run mqsidelpoy on the local machine... |
Thanks lancelotlinc for your inputs. I am exploring all the options in Hudson. You meant slave agents? ...Can you please educate me on how we can create agents in Hudson that can run mqsideploy on the WMB server. Indeed, i am looking for some configuration in Hudson that enables the deployment of the BAR file in remote WMB server.
Any external link is also appreciated. |
Yes, http://wiki.hudson-ci.org/display/HUDSON/Distributed+builds#Distributedbuilds-Example%3AConfigurationonUnix
| deepak_paul wrote: |
When we get a change request and modify the common components(subflows)(which are referenced by other Message flow projects) in the workspace, Is there a way we can make sure that all referencing message flow projects also get updated and create respective BAR files?
|
You should be starting with a blank directory structure every time. No existing source code should be used. The first step in your automation is to populate your blank directory structure with the latest source code files. There are a number of reasons to do it this way, one of them being that you are sure to get the latest files.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 07, 2011 6:07 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
You should be starting with a blank directory structure every time. No existing source code should be used. The first step in your automation is to populate your blank directory structure with the latest source code files. There are a number of reasons to do it this way, one of them being that you are sure to get the latest files. |
Can I re-phrase that?
You should be starting with a blank directory structure every time. No existing source code should be used. The first step in your automation is to populate your blank directory structure with the source code files. There are a number of reasons to do it this way, one of them being that you are sure to get the desired version of the files you have extracted from source control
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Sep 07, 2011 6:16 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Yes, much better Davies. You could choose to get source code files by tag, which is not necessarily the latest version of the files, but the desired version. Hopefully, the Ant script would have the ability to parameterize the selection criteria.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Wed Sep 07, 2011 6:46 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
Thanks lancelotlinc for the link.
And i believe i had a little different question which regard to change management. (maybe i didn't phrase it well. )
Question: When we get a change request on the common component say Subflows and we implement the change and check in back SCM, say Subversion, How we can configure Hudson in order to
1. Get the latest common components which are checked in after the change implementation
2. Get the list of all the main projects(message flow/set) affected by this change in common components. [The projects have references to common components]
3. Pull out the latest version of its referencing projects(message flow/set projects)
4. Build and create BAR files for these referencing projects.
Is there a way we can automate this process in Hudson?
In short, when i change my subflow and check in back to Subversion, How can i automate the process of building BAR file for all referencing projects
in Hudson?
Does Hudson support change management for such these scenarios?
_________________
Regards
Paul |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Sep 07, 2011 6:50 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Your questions are not broker questions, they are Configuration Management questions. The answers should come from your Build Engineer. What does she say?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Thu Sep 08, 2011 6:11 pm Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
| lancelotlinc wrote: |
| Your questions are not broker questions, they are Configuration Management questions. The answers should come from your Build Engineer. What does she say? |
I agree. But i would like to know how we are doing in other enterprises for especially Message broker Projects? Do we have same set of steps for automation or Is it completely by manual?
_________________
Regards
Paul |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 09, 2011 4:33 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| deepak_paul wrote: |
| lancelotlinc wrote: |
| Your questions are not broker questions, they are Configuration Management questions. The answers should come from your Build Engineer. What does she say? |
I agree. But i would like to know how we are doing in other enterprises for especially Message broker Projects? Do we have same set of steps for automation or Is it completely by manual? |
Our builds are fully automated. In the last seven years, I have worked at two major household brand insurance companies, two pharmaceutical companies, a major household bank, and a major credit card network. In all cases, build automation is a priority because after the development environment, no human hands affect the configuration of QA env, Perf Test env, or Production. The only way changes to those environments happen is through the Configuration Management system (BuildForge/ClearCase/ClearQuest in our current case).
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mattfarney |
| Posted: Fri Sep 09, 2011 8:52 am Post subject: Re: Message Broker Build using Ant/Hudson/SubVersion |
|
|
Disciple
Joined: 17 Jan 2006
Posts: 167
Location: Ohio
|
| deepak_paul wrote: |
| 2. Just wanted to know if there are any disadvantages in using mqsideploy for remote broker deploys. (As I go through older posts, I see that this approach is not recommended or at least not a best practice) |
Is this still a problem under 7.X? Most of the references I could find were of ConfigMgr deploys. What sort of problems are we talking about?
-mf |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 09, 2011 9:28 am Post subject: Re: Message Broker Build using Ant/Hudson/SubVersion |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mattfarney wrote: |
| deepak_paul wrote: |
| 2. Just wanted to know if there are any disadvantages in using mqsideploy for remote broker deploys. (As I go through older posts, I see that this approach is not recommended or at least not a best practice) |
Is this still a problem under 7.X? Most of the references I could find were of ConfigMgr deploys. What sort of problems are we talking about?
-mf |
Remote deployment is not recommended. The most reliable way is to move your deployable files to the local platform and deploy them from the local file system. Since many automated build packages include other artifacts besides BAR files (like SSL certificates, jars and properties files), the automated build process would need to move those resources to the local file system anyway so a remote mqsideploy-ment is not advantageous and fraught with opportunities to derail the automation.
OTOH - local deployments can be more reliably executed. For example, part of the automation would ensure that particular settings were correct, as in, are the SSL certificates in the truststore, is the EG set to the right port, etc. None of these things can be done remotely with mqsi commands and make better sense to perform them from a remote login.
To summarize: mqsideploy from automation is more reliable when local, not remote.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Sep 09, 2011 9:42 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 364
Location: Columbus, OH
|
| Quote: |
| In all cases, build automation is a priority because after the development environment, no human hands affect the configuration of QA env, Perf Test env, or Production. |
Must be nice.
I've been trying to get the Build teams (which I am part of one) out of the production deployment process.
My goal is to have the Run team (operations basically) handle all deployments from the performance test environment up to production.
Basically I want them to catch any errors when the deployments happen in performance rather than on production deployment night.
Once that happens I'll work on having the Run team do the deployments lower and lower down the environment stack. |
|
| Back to top |
|
|
|
|
