Getting to grips with Jelly and Stapler

 Posted on June 22, 2011  (Last modified on January 14, 2020)  |  admin

I use the Jenkins Continuous Integration server to help control and organise my various bits of code. Much of the work I do is in one or two man teams and and some question the use of a CI server for that but I actually find that it adds an extra pair of eyes and sanity checks that could easily be missed if you just rely on your own build environment to build/release things.

One of the nice things about Jenkins is it open, flexible and pluggable architecture. It quite happily builds both Java and .NET code for me with very little fuss.

Being the lazy engineer that I am I like to automate many tasks to ensure that they get done the same way each time. One of this tasks is pushing builds out to my development/QA servers. I use the Jenkins scp-plugin to help automate this but it only allows artifacts to be sent to one site/server for each build. There is a  longstanding feature request JENKINS-2923 “Upload artifacts to more scp sites” asking for multi-site features.

Tonight I finally cracked and decided that I ought to learn a bit of Jelly and Stapler and update the plugin so that it can cope with uploading to multiple sites from one build.

On the face of it this is a fairly simple change but there are a few complications because it is important to ensure that existing instances work correctly.

Multiple Sites

The first change is to separate the site specific functionality out of SCPRepositoryPublisher to create a per-site SitePublisher instance. The SCP RepositoryPublisher then contains a list of SitePublishers.[][5]

This relatively simple refactoring is made more complex by the need to maintain backward compatibility with existing job configuration. The configuration is saved/loaded using XStream and this handles all of the serialization and deserialization work for us.

Problems arise because siteName and entries objects are no longer direct members of SCPRepositoryPublisher. To handle this XStream allows an object to implement a readResolve() method that can be used to perform any data layout conversions. Hence we leave the siteName and entries members in SCPRepositoryPublisher and if they are populated then convert them into a SitePublisher instance.

public Object readResolve() {
    if ((siteName != null) || (entries != null)) {
        // Convert from old style single site to new multi-site
        // configuration.
        if (this.publishers == null) {
            this.publishers = new ArrayList<SitePublisher>();
        }
        this.publishers.add(new SitePublisher(this.siteName, this.entries));
        this.siteName = null;
        this.entries = null;
    }
    return this;
}

Stapling the Jelly together

Stapler and Jelly act like a macro language on steroids. They take a nice verbose XML description and turn it into a combination of HTML, JavaScript, JSON and Java using a healthy dose of reflection in the process to make it difficult for the uninitiated to work out what is going on.

I finally think I’ve got to the bottom of it using a fair amount of Firebug inspection of what is travelling over the wire. On the face of it it is fairly simple. The existing repeated list of entries needs wrapping in an outer repeated list of sites. However the output and input data structures need to be impedance matched with the form definitions to ensure that the right data gets into the right places.

At this stage I should add that there is likely some magic debug switch to turn on debugging of this and allow me to see what is happening both server side and browser side. I’ll leave that for a task for another day though.

The core of the Jenkins bindings for Jelly and Stapler live in lib/form these can be found in the core jenkins artifact at src/main/resources/lib/form.

The config.jelly file that implements the job configuration form fragment has 3 main functions:

  • output data population
  • form layout
  • input data organisation

Each of these is contained within the structure of the jelly data. It is of course necessary that the individual components are nested correctly together to produce the desired output.

Output Data Population

The basic form layout contains a template that is populated when the job configuration is loaded. This configuration is a JSON representation of the ScpRepositoryPublisher data. The form repeatable elements are used to traverse the list members contained within the data.

<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define" xmlns:l="/lib/layout" xmlns:t="/lib/hudson" xmlns:f="/lib/form">

 <j:set var="helpURL" value="/plugin/scp" />

  <f:entry title="Repositories">
 <strong>  <f:repeatable name="publishers" var="p" items="${instance.publishers}" add="Add Site"></strong>

  <table width="100%">
   <f:entry title="${%SCP site}">
    <select name="scp.publishers.siteName" description="Select configured SCP host.Check global hudson config for defining connection properties for this hosts">
     <strong><j:forEach var="s" items="${descriptor.sites}"></strong>
      <f:option selected="${s.name==p.siteName}">${s.name}</f:option>
     <strong></j:forEach></strong>
    </select>
   </f:entry>

   <f:nested>
    <f:entry title="${%Files to upload}">

     <strong><f:repeatable name="publishers.entries" var="e" items="${p.entries}"></strong>
      <table width="100%">
       <f:entry title="${%Source}" help="${helpURL}/help-source.html">
        <input name="scp.publishers.entries.sourceFile"
               type="text" value="${e.sourceFile}" />
       </f:entry>
       <f:entry title="${%Destination}" help="${helpURL}/help-destination.html">
        <input name="scp.publishers.entries.filePath"
               type="text" value="${e.filePath}" />
       </f:entry>
       <f:entry title="">
        <label>
         <f:checkbox name="scp.publishers.entries.keepHierarchy" checked="${e.keepHierarchy}" />
                     ${%Keep Hierarchy}
        </label>
       </f:entry>
       <f:entry title="">
        <div align="right">
         <strong><f:repeatableDeleteButton /></strong>
        </div>
       </f:entry>
      </table>
     <strong></f:repeatable></strong>
    </f:entry>
    <f:entry title="">
     <div align="right">
      <strong><f:repeatableDeleteButton value="Delete Site" /></strong>
     </div>
    </f:entry>
   </f:nested>
  </table>
 <strong></f:repeatable></strong>
 </f:entry>
</j:jelly>

The repeatable elements use the items and var attributes to identify the elements to be iterated over and to assign individual values into a loop variable.

Repeatable also causes an add button to be included to allow the form to be extended with new entries. We also add a delete button using repeatableDeleteButton and this will be added to each entry in the iterated list.

Form Layout

The form layout translates into recognisable HTML elements. These are iterated appropriately according to the repeatable elements discussed above.

<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define" xmlns:l="/lib/layout" xmlns:t="/lib/hudson" xmlns:f="/lib/form">

 <strong><j:set var="helpURL" value="/plugin/scp" /></strong>

  <f:entry title="Repositories">
   <f:repeatable name="publishers" var="p" items="${instance.publishers}" add="Add Site">

    <strong><table width="100%"></strong>
     <f:entry title="${%SCP site}">
      <strong><select name="scp.publishers.siteName" description="Select configured SCP host.Check global hudson config for defining connection properties for this hosts"></strong>
       <j:forEach var="s" items="${descriptor.sites}">
        <strong><f:option selected="${s.name==p.siteName}">${s.name}</f:option></strong>
       </j:forEach>
      <strong></select></strong>
     </f:entry>

     <strong><f:nested></strong>
      <strong><f:entry title="${%Files to upload}"></strong>

       <f:repeatable name="publishers.entries" var="e" items="${p.entries}">
        <strong><table width="100%"></strong>
         <f:entry title="${%Source}" help="${helpURL}/help-source.html">
          <strong><input name="scp.publishers.entries.sourceFile"</strong>
<strong>                                  type="text" value="${e.sourceFile}" /></strong>
         </f:entry>
         <f:entry title="${%Destination}" help="${helpURL}/help-destination.html">
          <strong><input name="scp.publishers.entries.filePath"</strong>
                 <strong>type="text" value="${e.filePath}" /></strong>
         </f:entry>
         <f:entry title="">
          <label>
           <strong><f:checkbox name="scp.publishers.entries.keepHierarchy" checked="${e.keepHierarchy}" /></strong>
                       ${%Keep Hierarchy}
          </label>
         </f:entry>
         <f:entry title="">
          <div align="right">
           <f:repeatableDeleteButton />
          </div>
         </f:entry>
        </table>
       </f:repeatable>
      </f:entry>
      <f:entry title="">
       <div align="right">
      <f:repeatableDeleteButton value="Delete Site" />
     </div>
    <strong></f:entry></strong>
   <strong></f:nested></strong>
  <strong></table></strong>
 </f:repeatable>
 </f:entry>
</j:jelly>

Input Data Organisation

The data being passed back to the server needs to be structured the same manner as the data expected to construct a ScpRepositoryPublisher element.

<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define" xmlns:l="/lib/layout" xmlns:t="/lib/hudson" xmlns:f="/lib/form">

 <j:set var="helpURL" value="/plugin/scp" />

 <strong><f:entry title="Repositories"></strong>
  <f:repeatable <strong>name="publishers"</strong> var="p" items="${instance.publishers}" add="Add Site">

   <table width="100%">
    <f:entry title="${%SCP site}">
     <select <strong>name="scp.publishers.siteName"</strong> description="Select configured SCP host.Check global hudson config for defining connection properties for this hosts">
      <j:forEach var="s" items="${descriptor.sites}">
       <f:option selected="${s.name==p.siteName}">${s.name}</f:option>
      </j:forEach>
     </select>
    </f:entry>

    <f:nested>
     <strong><f:entry title="${%Files to upload}"></strong>

      <f:repeatable name="publishers.entries" var="e" items="${p.entries}">
       <table width="100%">
        <f:entry title="${%Source}" help="${helpURL}/help-source.html">
         <input <strong>name="scp.publishers.entries.sourceFile"</strong>
                type="text" value="${e.sourceFile}" />
        </f:entry>
        <f:entry title="${%Destination}" help="${helpURL}/help-destination.html">
         <input <strong>name="scp.publishers.entries.filePath"</strong>
                type="text" value="${e.filePath}" />
        </f:entry>
        <f:entry title="">
         <label>
          <f:checkbox <strong>name="scp.publishers.entries.keepHierarchy"</strong> checked="${e.keepHierarchy}" />
                      ${%Keep Hierarchy}
         </label>
        </f:entry>
        <f:entry title="">
         <div align="right">
          <f:repeatableDeleteButton />
         </div>
        </f:entry>
       </table>
      </f:repeatable>
     <em></f:entry></em>
     <f:entry title="">
      <div align="right">
       <f:repeatableDeleteButton value="Delete Site" />
      </div>
     </f:entry>
    </f:nested>
   </table>
  </f:repeatable>
 <em></f:entry></em>
</j:jelly>

It is not immediately obvious when developing the form but the entry elements that surround the repeatable elements are essential. These inform the JSON constructor to create a new list element that contains the sub-entries that are repeated within the form.

The names of individual entries is important because these are used to name the JSON data components and these are then used to identify appropriate parameters when constructing the Java objects.

The Java objects are built by the newInstance method from the plugin descriptor boilerplate code.

@Override
public Publisher newInstance(StaplerRequest req, JSONObject formData) {
    return req.bindJSON(SCPRepositoryPublisher.class, formData);
}

This in turn causes the appropriate Java object to be built by the bindJSON call.  bindJSON looks for Constructors that are annotated with @DataBoundConstructor. This is where the names of the JSON objects must match exactly with the constructor parameters. Any differences will result in the data being silently dropped.

All of this put together gives us our original aim of utilising Stapler and Jelly to add an extra level of configuration into each job. What at first seems complex turns into some fairly simple components that fit together to create the whole. Now on with deploying my build artifacts to multiple sites.