Merge "Jenkins Job Builder 2.0.0 API Changes/Rewrite"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
fe22ae841d
@ -41,6 +41,7 @@ permits.
|
||||
specs/deploy-ci-dashboard
|
||||
specs/doc-publishing
|
||||
specs/infra-cloud
|
||||
specs/jenkins-job-builder_2.0.0-api-changes
|
||||
specs/nodepool-launch-workers
|
||||
specs/nodepool-workers
|
||||
specs/public_hiera
|
||||
|
||||
300
specs/jenkins-job-builder_2.0.0-api-changes.rst
Normal file
300
specs/jenkins-job-builder_2.0.0-api-changes.rst
Normal file
@ -0,0 +1,300 @@
|
||||
|
||||
::
|
||||
|
||||
Copyright (c) 2015 Wayne Warren
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0
|
||||
Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
===============================
|
||||
JJB 2.0.0 API Changes
|
||||
===============================
|
||||
|
||||
https://storyboard.openstack.org/#!/story/2000258
|
||||
|
||||
The goal of this spec is to define a series of backwards-incompatible API
|
||||
changes that will necessitate a 2.x release of Jenkins Job Builder. The
|
||||
high-level goals of these changes are:
|
||||
|
||||
* to ensure that the various classes are useful outside of JJB itself where
|
||||
appropriate
|
||||
* to clearly define public interfaces for various classes and document the
|
||||
purpose/role of these classes in the API
|
||||
* to eliminate unuseful class relationships in favor of tiered delegation of
|
||||
program behavior
|
||||
* to reduce the importance of the YamlParser as the only source of the data
|
||||
structures necessary for XML generation
|
||||
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
Jenkins Job Builder has history of organic, piecemeal development wherein
|
||||
patches tend to be highly focused on improving one or another aspect of the
|
||||
program. For the most part this is understandable, expected even--the modular
|
||||
implementation of XML generation on a per-Jenkins-plugin basis lends itself well
|
||||
to this style of development. However when it comes to changes to the core
|
||||
abstractions, in particular the Builder and YamlParser clases, many ad hoc
|
||||
design decisions have led to tightly bound implementations that make the overall
|
||||
API difficult if not impossible to reuse as a library.
|
||||
|
||||
It's true that yes, we can import jenkins_jobs.Builder or
|
||||
jenkins_jobs.parser.YamlParser to use them programmatically outside of JJB
|
||||
proper, but it is difficult to imagine anyone actually doing this given the
|
||||
current constructors for these classes as well as their relationships--Builder
|
||||
objects actually own references to YamlParser objects when there are no clear
|
||||
benefits to this organization save for expediency during JJB development.
|
||||
|
||||
The YamlParser itself actually does more than "parse" YAML, it is also
|
||||
responsible for generating XmlJob objects from the yaml data it handles. This
|
||||
means that all the lovely XML building code that makes JJB truly special cannot
|
||||
reliably be accessed using JJB as a library since it depends on implementation
|
||||
details of the YamlParser (specifically the xml_jobs attribute).
|
||||
|
||||
In addition to the tight binding between classes that make them practically
|
||||
unusable, there is an almost impossibly tight binding between code in
|
||||
jenkins_jobs.execute() and jenkins_jobs.Builder, which depends on code in
|
||||
jenkins_jobs.execute() to munge various command line options and configuration
|
||||
data, ie to obtain user, password, plugins_list, cache settings, etc. To top it
|
||||
off, jenkins_jobs.Builder ends up taking the "config" object *anyway*, which
|
||||
should make us question to what purpose jenkins_jobs.execute() is even handling
|
||||
various pieces of configuration logic at all--arguably, this logic ought to live
|
||||
in a separate "Config" class.
|
||||
|
||||
The problems described up to this point indicate a need to clean up the API, but
|
||||
one issue that really deserves to be driven home is the essential character of
|
||||
the YamlParser to everything that JJB does. Because of the tight bindings
|
||||
between classes described above, understanding the tricky behavior of the
|
||||
YamlParser is absolutely necessary to construct anything more simple than a
|
||||
collection of "job" objects or even a simple set of "project"s containing lists
|
||||
of "job-template"s.
|
||||
|
||||
This is primarily problematic because if JJB is going to offer a programmatic
|
||||
API, it should be possible to use it to do something more interesting than the
|
||||
exact same thing that is being done in jenkins_jobs.execute(). For example, it
|
||||
should be possible to define one's own set of abstraction classes in Python that
|
||||
can be used to generate the data used by jenkins_jobs.Builder to POST jobs to a
|
||||
Jenkins server.
|
||||
|
||||
JJB plugin modules all take a "parser" argument, but rarely if ever is this
|
||||
parser argument used in these modules. This parser is intended to be a reference
|
||||
to a YamlParser instance but the benefit of this relationship does not really
|
||||
outweigh its cost in terms of tight binding between plugin module and the means
|
||||
used to obtain JJB data structures actually necessary for generating XML; its
|
||||
primary use case most of the time is to provide access to the ModuleRegitry
|
||||
object owned by the YamlParser. It would make more sense to replace this with an
|
||||
explicit reference to the ModuleRegistry itself.
|
||||
|
||||
A Side Note on Macros
|
||||
---------------------
|
||||
|
||||
A problematic JJB feature when it comes to cleaning up the API is the "macro".
|
||||
The current implementation leads to an unnecessarily tight binding between
|
||||
the YamlParser and the ModuleRegistry because the ModuleRegistry and XML
|
||||
generation code require the YamlParser's xml_jobs attribute in order to
|
||||
determine whether or not a component is a macro. This tight binding is
|
||||
problematic because it means we need to mock the YamlParser's xml_jobs data
|
||||
structure in order to use any other means of producing JJB data structures than
|
||||
the YamlParser itself. As of right now, it is fine to mock YamlParser's xml_jobs
|
||||
data structure with an empty dictionary but some detail of the way it is
|
||||
accessed in the ModuleRegistry or XML generation code could easily change that
|
||||
would cause unforeseen problems when attempting to avoid using the YamlParser.
|
||||
|
||||
Proposed Changes
|
||||
================
|
||||
|
||||
Modularize Command Line Parsing
|
||||
-------------------------------
|
||||
|
||||
Since so much of JJB's API depends on JJB's configuration file and command line
|
||||
options, it makes sense to modularize these in such a way that they can easily
|
||||
be used by third party scripts/tools. This will be done using a dedicated
|
||||
"Config" class.
|
||||
|
||||
This will allow constructors for other JJB classes such as the Builder and
|
||||
YamlParser to be simplified--rather than taking both a "config" object and a
|
||||
number of other parameters they will instead take a single "config" parameter.
|
||||
|
||||
Redefine Programmatic APIs
|
||||
--------------------------
|
||||
|
||||
The following classes will need their APIs redesigned to make them usable by
|
||||
third parties:
|
||||
|
||||
* YamlParser
|
||||
* ModuleRegistry
|
||||
* Builder
|
||||
|
||||
These rewrites should be accompanied by docstrings that explicitly define the
|
||||
purpose of these classes so that these purposes can be considered in the context
|
||||
of future patches against JJB that attempt to make foundational changes to their
|
||||
behavior.
|
||||
|
||||
YamlParser
|
||||
''''''''''
|
||||
|
||||
The current YamlParser exposes all of its methods as "public" methods even when
|
||||
there is no obvious or implicit value in doing so. The goal for rewriting its
|
||||
API will be to provide explicit "public" methods that other classes can call to
|
||||
get information from it.
|
||||
|
||||
ModuleRegistry
|
||||
''''''''''''''
|
||||
|
||||
The new ModuleRegistry will serve a dual role; in addition to registering JJB
|
||||
modules for use during XML generation it will also provide an API for modules to
|
||||
use to access information other than the given JJB data. For example, it should
|
||||
provide access to Jenkins plugin version information to allow modules to
|
||||
implement plugin version dependent XML.
|
||||
|
||||
|
||||
Add XmlBuilder class
|
||||
--------------------
|
||||
|
||||
The XmlBuilder class should define a method that takes as input a dictionary
|
||||
where:
|
||||
|
||||
* keys are job names, and
|
||||
* values are all the key/value pairs necessary to produce an XmlJob for the
|
||||
named job
|
||||
|
||||
This method should produce as output a list of XmlJobs that can then be passed
|
||||
to a Builder method either to produce test output or to update jobs on the
|
||||
configured Jenkins instance.
|
||||
|
||||
Alternate names for this class might be JenkinsXmlBuilder,
|
||||
JenkinsJobConfigBuilder.
|
||||
|
||||
Replace references to YamlParser with references to ModuleRegistry in modules
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
This addresses the need to decouple plugin modules from the YamlParser while
|
||||
also allowing us to access components of the ModuleRegistry when necessary (ie,
|
||||
accessing Jenkins plugins_info to generate different XML for different versions
|
||||
of plugins).
|
||||
|
||||
Rewrite JJB Macro Implementation
|
||||
--------------------------------
|
||||
|
||||
The goal of this rewrite should be primarily to remove the need for referencing
|
||||
the YamlParser's raw yaml data structure while generating XML. This way the
|
||||
proposed XmlBuilder class does not have to be concerned with any particular
|
||||
details of the YamlParser--reducing the risk that implementation of one of these
|
||||
classes has unforeseen side effects on the other's behavior.
|
||||
|
||||
One method that has been proposed is to implement a secondary module registry
|
||||
that specifically handles macros.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Wayne Warren, wayne+launchpad@sdf.org
|
||||
|
||||
Gerrit Topic
|
||||
------------
|
||||
|
||||
Use Gerrit topic "jjb-2.0.0-api" for all patches related to this spec.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git-review -t jjb-2.0.0-api
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
Development Process
|
||||
'''''''''''''''''''
|
||||
|
||||
All work for this spec will target a feature branch of JJB called "2.0.x" which
|
||||
will contain all backwards-incompatible changes targeted for this release,
|
||||
including any work covered by other specs targeting the same release.
|
||||
|
||||
Once the work is complete, we will release Jenkins Job Builder 2.0.0.
|
||||
|
||||
Code Changes
|
||||
''''''''''''
|
||||
|
||||
* Break command line parsing function into dynamically-loaded, modular
|
||||
components for consumption outside of the primary JJB command line tool.
|
||||
* Create function that specifically handles pre-Builder processing
|
||||
of command line options and configuration settings; this will be necessary to
|
||||
make the Builder class usable as an API to be consumed externally.
|
||||
(Alternatively, consider ways to clean up the Builder constructor in such a
|
||||
way that doesn't need to take the full "config" object)
|
||||
* Separate XML generation code from YamlParser, move into new class "XmlBuilder"
|
||||
that takes a ModuleRegistry object in its constructor and contains various
|
||||
methods that take one or more JJB job dictionaries and output a corresponding
|
||||
number of XmlJob objects.
|
||||
* Separate YamlParser from YamlInterpreter:
|
||||
* YamlParser will be responsible for loading a given list of YAML files
|
||||
* YamlInterpreter will be responsible for applying JJB's DSL semantics to the
|
||||
loaded lists of dictionaries.
|
||||
* Refactor Builder methods and their calling contexts to simplify the interface;
|
||||
the "update_jobs" method, for instance, should simply take a list of XmlJobs
|
||||
to update.
|
||||
|
||||
Repositories
|
||||
------------
|
||||
|
||||
None
|
||||
|
||||
Servers
|
||||
-------
|
||||
|
||||
None
|
||||
|
||||
DNS Entries
|
||||
-----------
|
||||
|
||||
None
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
Documentation changes may be necessary as this work is primarily intended to
|
||||
make existing JJB API usable outside of the jenkins_jobs.cmd module.
|
||||
|
||||
Security
|
||||
--------
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
Backwards Compatibility
|
||||
'''''''''''''''''''''''
|
||||
|
||||
No existing test scenarios will be changed to support this spec; however, there
|
||||
will need to be changes to test fixture base classes to support the new API
|
||||
changes.
|
||||
|
||||
We will also monitor openstack-infra's JJB output to ensure that it does not
|
||||
change because of this spec.
|
||||
|
||||
New Tests
|
||||
'''''''''
|
||||
|
||||
Unit tests will explicitly validate the behavior of the following:
|
||||
|
||||
* the Builder class' "public" methods
|
||||
* the YamlParser class' "public" methods
|
||||
* the XmlBuilder class' "public" methods
|
||||
* the Config class' "public" methods
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
Python Support
|
||||
--------------
|
||||
|
||||
We intend to continue supporting the same Python versions supported by JJB 1.x:
|
||||
|
||||
* Python 2.7
|
||||
* Python 3.4
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user