Saturday, June 20, 2015

Forking pipelines

Yesterday, in preparation for Dockercon I posted a blog summary of some things we have been working on.

A piece buried within that post is the ability to fork a project and get its delivery pipeline along with the source. GitHub has made it trivial to share and fork source but deploying another persons application has never been simple. Docker goes a long way to making this better by providing consistent packaging mechanism and environment to run an application. Now via the use of a pipeline.yml file in the projects source repository we are able share the full delivery pipeline along with the source of an application.



I hope that this gets traction. Every software project has to standup a delivery pipeline. This always seems like more work than it should be and often feels like recreating the wheel. The ability to copy or clone existing pipelines is step towards allowing teams have robust continuous delivery pipelines with little cost, and for organizations to standardize on their delivery processes in a way that is executable rather than simple a list of requirements on some wiki. I can imagine communities emerging to share job types and pipeline definitions but to start with lets take a closer look at this first step.

The application

Lets assume you have a project with a pipeline (if you don’t see the post above, and go create one). For this exercise I took a BookClub application that Steve Atkin developed. This application is used to demonstrate how a java application can use the Globalization Pipeline service. It is a good application for my purposes because deployment involves binding the application to a number of services in IBM Bluemix, as well as passing credentials to the application for services not in Bluemix. In addition it is a Cloud Foundry application so also provides me with an excuse to Dockerize it and see how things work on the Container service.



Generating a pipeline.yml file

I cloned the project into a GitHub repository. This step was not necessary I wanted to organize a few examples in one place. The BookClub project had a pipeline. If your application does not you should go ahead and build one out manually that includes at least a Build and Deploy stage:
Next I generated a pipeline.yml file from the existing Pipeline by adding /yaml to the end of my exiting pipeline definition.
I checked this into my source repository as .bluemix/pipeline.yml.

The pipelne.yml file contained information specific to my organization and space. I edit the file and replaced

    target:
      url: https://api.ng.bluemix.net
      organization: rjminsha@us.ibm.com
      space: dev
with
    target:
      url: ${CF_TARGET_URL}
      organization: ${CF_ORGANIZATION}
      space: ${CF_SPACE}
Now when someone clones my pipeline, their information will be automatically substituted. I also replaced my application name with
     
   application: ${CF_APP}
Now when the project is copied using the 'Deploy to Bluemix Button', a unique project name will be generated and passed in as ${CF_APP}. In addition a unique route or url will be created for the new instance of the application and pipeline. These can be changed later if you desire. If you are working with a pipeline that uses Containers you will want to set the IMAGE_NAME, CONTAINER_NAME, and ROUTE_HOSTNAME to ${CF_APP}.

Dealing with services

Next I had to think about services. The BookClub application leverages the Globalization, Watson Machine Translation and IBM Insights for Twitter services in Bluemix. To deploy the application into a space you need those services added to the space and bound to the application. Binding a service to an application is how an application gets credentials for a service, and knows the correct endpoint.

Rather than ask the user to manually do this as a 'one time setup' I wanted to automate the creation of the service. If the services already existed I should bind the existing instances to my application, and if not I need to add service instances into the space. I checked a deployment utility into my repository that I would use to automate this.
... 
try:
    Logger = setupLogging()
    parser = argparse.ArgumentParser()
    parser.add_argument("--app")
    args=parser.parse_args()
    if args.app:
        appName=args.app
    else: 
        appName=DEFAULT_BRIDGEAPP_NAME

    createBoundAppForService(GLOBALIZATION_SERVICE, GLOBALIZATION_SERVICE_PLAN, GLOBALIZATION_SERVICE_NAME,appName)
    createBoundAppForService(MACHINE_TRANSLATION_SERVICE, MACHINE_TRANSLATION_PLAN, MACHINE_TRANSLATION_NAME,appName)
    createBoundAppForService(TWITTER_SERVICE, TWITTER_PLAN, TWITTER_NAME,appName)

    sys.exit(0)

except Exception, e:
    Logger.warning("Exception received", exc_info=e)
    sys.exit(1)
...
To make this available to the deployment jobs I also added it to my build stage in the pipeline.yml
      #!/bin/bash
      # build the application 
      mvn -B package
      #copy deploy utility
      cp setup_services.py ${ARCHIVE_DIR}
The last step was to deal with the API Keys needed to access non-bluemix services. The application uses APIs from http://developer.nytimes.com, http://idreambooks.com/api and http://www.alchemyapi.com. I didn't want to include private information like this in the build script or pipeline.yml so moved these to secure properties on the deploy stage and once again updated the pipeline.yml.
The secure properties can be accessed as environment variables in the deploy script, but will be hidden in output.


Adding Deploy to Bluemix

Finally I added a 'Deploy to Bluemix' button to my projects README.md. For information on this 'Deploy to Bluemix' refer to Philippe Mulet's excellent blog. It boils down to adding the following markdown to your README.md :
[![Deploy to Bluemix](https://bluemix.net/deploy/button.png)](https://bluemix.net/deploy?repository=https://github.com/Puquios/bookclub-foundry.git)
After doing so if someone clicks this button it will automatically clone the project, create an Bluemix DevOps Services Project, create a pipeline, generate a unique application name and route and kick off an initial deployment.


Give it a try

Head on over to the to the project, just deploy it!, or check out some of my other examples on my Puquios project. Then see some of the other examples, or share your own.

3 comments:

  1. Great Post,really it was very helpful for us.
    Thanks a lot for sharing!
    I found this blog to be very useful!!
    Python training in Bangalore

    ReplyDelete
  2. Thanks for sharing like a wonderful blog’s learn more new information from your blog. Keep sharing the post like this
    Python training in bangalore
    Python training in Bangalore

    ReplyDelete