Build a Devops Server

Devops…

The project development cycle goes back to the beginning of time, i.e. the Unix epoc, January 1, 1970. I’ve seen a few big trends come and go in project management and also some lasting improvements. The latest wave seems to be devops. What is devops, anyway? My understanding is that it is a collection of practices and tools that streamline the development cycle. Perhaps the biggest selling point is that it drastically reduces time to market for projects. But that’s only the beginning. Since devops makes things speedier, it encourages us to try out new ideas that we might otherwise pass up.

Although devops seems to be geared to medium and large IT departments, it could also be very useful on an individual or small team level. Sometimes when you are working with a new technology the hardest thing is to set up the software stack. Devops could make this much easier by automating the grunt work. Another frequent chore is deploying small changes as you fine tune an app. This could be a simple one-click operation with devops.

This article is about setting up a devops server which you can then use for any projects that you like. Isolating your devops environment to a server has a number of advantages:

  • You don’t have to clutter your computer with devops tools.
  • Conversely, your devops environment is not cluttered with whatever is on your work computer.
  • You can use mobile devices as well as your main development machine.
  • Members of a team can access the same devops environment.
  • You can standardize on a single operating system (eg Amazon Linux) for devops.

You will need some familiarity with AWS and Ansible. Some articles on this blog that can get you up to speed are:

  1. A Project Template for AWS
  2. Ansible Quick Start
  3. The AWS CLI Command Line Tool

Overview of Project

  1. Check the requirements. These include launching a new server instance in AWS, and ensuring that you have Ansible on your local computer.
  2. Clone the Github repo that contains the Ansible scripts for the project.
  3. Use Ansible to install the software stack on the devops server.
  4. Configure Jenkins.
  5. Use the new devops server to build out an example project that creates s microservice.

The devops server will have the following software tools:

  • Ansible, for installing and configuring software on other servers.
  • Jenkins, for continuous project builds and deployment.
  • Git, for managing programs, scripts, config files, etc.
  • Maven, for compiling and packaging Java.

Future releases will probably include a C++ compiler and other tools.

The server is meant to be a devops platform, available to members of a project team, and separate from the development environment. The tool set is intended to automate the build out process to a degree where it becomes quick and reliable.

A note about public IP addresses in AWS: The devops server does not have to be up continuously; it can be shut down when it isn’t needed, thus making it very economical for a cost-conscious project. Another cost-saving measure is to forego a permanent public IP address for the server and let AWS assign one from its pool. The one drawback of using the pool IP addresses is that they change each time a server is stopped and started again.

Requirements

  1. An Amazon AWS instance running Amazon Linux, its public IP address, and the SSH key for accessing the instance. The security group for the instance should allow incoming connections on ports 22 and 8080.
  2. A local computer for running the Ansible playbook. The machine should have Linux (or Mac OSX), the Ansible package, and Git.

Build Procedure

  1. Download this repo to your local computer:

      git clone http://github.com/tomlamphier/ansible-devops
    
  2. Change the directory to ansible-devops. Copy in SSH key as devops-private-key.pem. Ensure that the permissions are set to 400.

  3. Edit the two files connect and hosts, replacing xxx.xxx.xxx.xxx with the public IP address of the AWS instance.

  4. Test your connectivity to the AWS server:

      ./connect
      # respond 'yes' to the enevitable authentication message
    
  5. Run the Ansible playbook

      ansible-playbook -i hosts playbook.yml
      # Note: The playbook displays a Jenkins initial password
      # near the end of the run. Cut and paste this password to a temp
      # file for later use.
    

Post-Build Configuration

The last step of the above build starts Jenkins on the devops server. We will log on to Jenkins from a browser to perform the post-build configuration:

  1. Go to the devops public IP address + port 8080 in a browser. You will see a page asking for the initial Jenkins password. Paste it into the appropriate field.
  2. Select the option to install the recommended plugins. Follow the instructions to create an admin user and password, continue to home page.
  3. Go to Manage Jenkins ==> Manage Plugins. Install the Publish over SSH plugin. (Install without restart)
  4. Go to Manage Jenkins ==> Global Tool Configuration. Scroll down to Maven – Maven installations, click [Add Maven], Set name to “Maven on Devops”, uncheck “Install Automatically”, set Maven Home to “/usr/local/apache-maven-3.5.2”. Click on [Save].
  5. Go to Manage Jenkins ==> Configure System, Scroll down to “Publish over SSH.” Cut and paste contents of the private key file (devops-private-key.pem) into the Key field. Click [Save].

Use Devops Server to Build a Project

Now that we have the devops server up and running, we can use it to build something. The example-devops directory in the previously cloned repo includes all you need to create a microservice that uses an embedded version of Tomcat. This project comes from Oracle’s online tutorials.

  1. Launch a new Amazon Linux instance in AWS using the same SSH key pair as before. Write down the public IP address that AWS assigns for the instance.

  2. Copy the Tomcat microservice project to the devops server.

      # in ansible-devops directory
      scp -i devops-private-key.pem -r example-devops ec2-user@xxx.xxx.xxx.xxx 
      # replace xxx's with public IP of the devops server
    
  3. Connect to devops server.

      # in ansible-devops directory
      ./connect
    
  4. Change to the example-devops directory, edit hosts and connect files, replacing xxx’s with public IP of the Tomcat microservice instance.

  5. Test connectivity from devops server to microservice server.

      ./connect
      # enter logout to exit
    
  6. Run the Ansible playbook.

      ansible-playbook -i hosts playbook.yml
    
  7. Create a Jenkins pipeline for the Java code.

    • Go to the devops server’s public IP + port 8080 in a web browser.
    • Log on with your Jenkins user ID and password.
    • Go to Manage Jenkins ==> Configure System, scroll down to “SSH Servers” under “Publish over SSH.” Click [Add]. Set name to “tomcat-microservice”, Hostname to the public IP of the microservice instance, Username to “ec2-user”. Click [Save].
    • Click “New Item”, set name to tomcat-microservice, select “Freestyle Project”, click [OK].
    • Select Git for source code management, set repository URL to http://github.com/tomlamphier/tomcat-microservice.git
    • Scroll down to “Build.” Click [Add Build Step], select “Invoke top-level Maven targets”, set Maven Version to “Maven on Devops”, Goals to “clean package”.
    • Scroll to “Post-Build Actions.” Click [Add post-build action]. Select “Send build artifact over SSH. Set server name to “tomcat-microservice”, target source files to “**/target/*.jar, remove prefix to “target/”.
    • Click [Save]
  8. Click [Build Now] to run the Jenkins build. Note the first run may take up to 5 minutes, as Maven will be doing many initial downloads. A blue icon after a build means success. You can click on an item under build history and select “Console Output” to see what happened with a given build.

  9. Now we can run the microservice. Connect to the microservice instance and start the microservice.

      # in directory example-devops
      ./connect
      ./start-tomcat  
      # note: use Ctl-c or command ./stop-tomcat to shut down service
    
  10. Go to the public IP address, port 8080 of the microservice instance in a browser. If all has gone well up to this point, you should see the following page:

Conclusion

The point of having a devops server is to automate repetitive and time-consuming project tasks. The example Tomcat microservice project illustrates how quickly you can deploy a project, once you have the infrastructure in place. There is a learning curve involved when you try to get up to speed with devops–but it’s worth the trouble as an investment in your productivity.