Jenkins as a code

Jenkins is an open-source tool built of plugins – that is the most important thing when you consider functionalities of this tool. Great place to deal with continuous integration and continuous deployment within huge community. This article is for people who are familiar with Jenkins, but they are still working with standalone instances and face problems such as backup, restore, and safety upgrades. I will describe our solution, how Jenkins deals with the maintenance process by himself.

Another team, another approach

I have worked with many different solutions regarding automation tools, Jenkins was most commonly used, but when the dust of the project subsided, I realized many different instances do the same, with the same configuration, but some of them were running on Docker container another one strict on the virtual machine. It does not matter how many different solutions exist to run Jenkins if always appear a few questions: What about backup? How to do it? Should I use a plugin for that? How to restore my instance after crash on another machine?  This is the right place to ask them when you are responsible for maintaining Jenkins instances. We decided to use Jenkins in Docker containers years ago especially in order to ensure that their dependencies are accurate and segregated from other platform’s dependencies, regardless of the infrastructure, you can run Jenkins instance on your machine.

What do we need?

My experience tells me on the sentence “as simple as possible”, is hard to understand by people who just started, they are absorbing knowledge, would like to try many different technologies, sometimes I think they just want to use all that they know. I have prepared a list of things, which I would like to have in my ideal solution :

• store configuration of Jenkins in a repository, maybe in flat files if it is possible,
  • simple restore of Jenkins on another machine,
  • easy upgrade.

There are all requirements that I want for now. I have made up with the fact that I will never find a suitable solution in one piece that is why I divided the requirements and I was looking for one solution per one single functionality.

Jenkins as a code
In the world where automation is a key to IT infrastructure and everything we can define as a code, I have found the plugin to Jenkins which generates configuration as a code. Let’s take it to test. This plugin as the rest of plugins is available, you can easily install it.

That’s it, from now on you have available new icon from Manage Jenkins site:

Couple options exist under this icon:

Example configuration with default plugins after fresh start of Jenkins instance:

jenkins:
  agentProtocols:
  - "JNLP4-connect"
  - "Ping"
  authorizationStrategy:
    loggedInUsersCanDoAnything:
      allowAnonymousRead: false
  crumbIssuer:
    standard:
      excludeClientIPFromCrumb: false
  disableRememberMe: false
  labelAtoms:
  - name: "master"
  markupFormatter: "plainText"
  mode: NORMAL
  myViewsTabBar: "standard"
  numExecutors: 2
  primaryView:
    all:
      name: "all"
  projectNamingStrategy: "standard"
  quietPeriod: 5
  remotingSecurity:
    enabled: true
  scmCheckoutRetryCount: 0
  securityRealm:
    local:
      allowsSignup: false
      enableCaptcha: false
      users:
      - id: "admin"
        name: "Admin"
        properties:
        - "apiToken"
        - "myView"
        - preferredProvider:
            providerId: "default"
        - "timezone"
        - mailer:
            emailAddress: "nobody@nowhere"
  slaveAgentPort: 50000
  updateCenter:
    sites:
    - id: "default"
      url: "https://updates.jenkins.io/update-center.json"
  views:
  - all:
      name: "all"
  viewsTabBar: "standard"
security:
  apiToken:
    creationOfLegacyTokenEnabled: false
    tokenGenerationOnCreationEnabled: false
    usageStatisticsEnabled: true
  sSHD:
    port: -1
unclassified:
  buildDiscarders:
    configuredBuildDiscarders:
    - "jobBuildDiscarder"
  buildStepOperation:
    enabled: false
  defaultFolderConfiguration:
    healthMetrics:
    - worstChildHealthMetric:
        recursive: true
  extendedEmailPublisher:
    adminRequiredForTemplateTesting: false
    allowUnregisteredEnabled: false
    charset: "UTF-8"
    debugMode: false
    defaultBody: |-
      $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS:

      Check console output at $BUILD_URL to view the results.
    defaultSubject: "$PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS!"
    maxAttachmentSize: -1
    maxAttachmentSizeMb: 0
    precedenceBulk: false
    useSsl: false
    watchingEnabled: false
  gitHubConfiguration:
    apiRateLimitChecker: ThrottleForNormalize
  gitHubPluginConfig:
    hookUrl: "http://localhost:8080/github-webhook/"
  gitSCM:
    createAccountBasedOnEmail: false
    showEntireCommitSummaryInChanges: false
    useExistingAccountWithSameEmail: false
  location:
    adminAddress: "<nobody@nowhere>"
    url: "http://localhost:8080/"
  mailer:
    charset: "UTF-8"
    useSsl: false
    useTls: false
  pollSCM:
    pollingThreadCount: 10
  timestamper:
    allPipelines: false
    elapsedTimeFormat: "'<b>'HH:mm:ss.S'</b> '"
    systemTimeFormat: "'<b>'HH:mm:ss'</b> '"
tool:
  git:
    installations:
    - home: "git"
      name: "Default"

We can easily download the current configuration, reload existing, or apply testing one.
Let’s switch to the structure of this configuration file, we will see my favorite structure – yaml. I won’t write about fields and meaning each of the fields but you should know about the main groups of configuration.

• Jenkins – here you can find everything related with Jenkins master’s setting like labels, primary view or settings of update center
  • security – especially scripts which have been approved to execute by Jenkins
  • tools – all things related to tools configuration in http://jenkins_instance/configureTools/
  • unclassified – this is a big bag for all necessary configuration but which is not supported by this plugin, in other words, the plugin does not have implementation for support Jenkins as a code.
  • credentials – I have left this section on the end because of encryption. This article shows the concept of upgrade and maintenance Jenkins, that’s why encryption is a very important section, imagine that your Jenkins should be moved or current instance of Jenkins has broken and backup contains the only configuration from plugin Jenkins as a code and job history. May it work?

Jenkins generated secret key one after the first start which is stored in this file: $JENKINS_HOME/secret.key. This key is related to plugins data encryption, connection master with various types of clients. What about your credentials? I’ve just created one with value “test”.

Now I will show you how it looks in Configuration as Code structure.

You have two options in this place, first – decrypt credentials, second – store in external place e.g. Vault. Two files are responsible for encryption and also decryption:

$JENKINS_HOME/secrets/master.key

$JENKINS_HOME/secrets/hudson.util.Secret

Secret key “master.key” is used to encrypt “hudson.util.Secret” key which encrypts your credentials.

Function to decrypt credentials is available in Jenkins scripts: hudson.util.Secret.decrypt(„{XXX=}”)

Summarizing, if you would like to apply configuration for Configuration as a code plugin on a fresh instance of Jenkins you should have decrypted data but to be more secure you should decrypt the content of repository where this configuration is. I encourage you to use Vault if you have the opportunity.

How to deal with upgrade – custom solution.

I was trying to do some upgrade process for Jenkins which will be managed by… Jenkins but I still want to review changes before upgrade – just for sure. Because BitBucket is a part of my technology stack It will prepare Pull Request after my test passed! Referring to the test, I was preparing it by 2 months, between manual upgrades I noticed the most important action which must be executed to be sure that Jenkins will work properly. I’ve described everything that I need on this schema:

The hardest thing was preparing configuration with a new base Docker image of Jenkins and when I finally did it, it was time to do the test. Thanks for that my Jenkins is dockerized so everything that I need to do is up Jenkins container and catch the result from logs. First question, how to test all plugins with the new version of Jenkins or all upgraded plugins with the current version? All Jenkins plugins inherit Harness unit test from the plugin parent POM, so when something goes wrong I will know that in the starting the process. The test is ready, time to put it to Jenkinsfile, I’ve separated all things to do in a single stage to be more readable.

I will receive notification about status of this job, but that is Jenkins functionality so I will not write about it.

After a successful build that discovers changes, Pull Request will appear and will wait for the approval.

What is the status of the current maintenance process? Right now, after the merge, we have the newest and tested version ready to deploy, so it’s time to deploy! I see only one question right now, when? I would like to do this smoothly and without any interruptions. I created some coordinator connected with the groovy script, finally, I have achieved what I wanted.
I used Jenkins job with crontab which checks from 00:00 to 2:00 every 10 minutes if Jenkins is busy. When it is finally free from others Jobs I put Jenkins in a quiet mode which prevents Jenkins to start new Jobs. That is time for a basic check which handles coordinator and answer to Jenkins job using HTTP code. When the coordinator finds some changes in the repository, the upgrade process starts with a safe exit mode in Jenkins. I do not expect any surprises because I have already tested this version! You can find more details in the schema above:

Summary

There are many different approaches to deal with upgrades. Everything depends on what is most important for you in this process, what you can afford and how can you be sure Jenkins is properly tested. Although I hope it will be an inspiration for the maintenance of your Jenkins!