diff --git a/doc/source/index.rst b/doc/source/index.rst index dcf24eb..f515042 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -41,6 +41,7 @@ permits. specs/deploy-ci-dashboard specs/doc-publishing specs/infra-cloud + specs/jenkins-job-builder_2.0.0-api-changes specs/nodepool-launch-workers specs/nodepool-workers specs/public_hiera diff --git a/specs/jenkins-job-builder_2.0.0-api-changes.rst b/specs/jenkins-job-builder_2.0.0-api-changes.rst new file mode 100644 index 0000000..50f6eea --- /dev/null +++ b/specs/jenkins-job-builder_2.0.0-api-changes.rst @@ -0,0 +1,300 @@ + +:: + + Copyright (c) 2015 Wayne Warren + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode + +=============================== +JJB 2.0.0 API Changes +=============================== + +https://storyboard.openstack.org/#!/story/2000258 + +The goal of this spec is to define a series of backwards-incompatible API +changes that will necessitate a 2.x release of Jenkins Job Builder. The +high-level goals of these changes are: + +* to ensure that the various classes are useful outside of JJB itself where + appropriate +* to clearly define public interfaces for various classes and document the + purpose/role of these classes in the API +* to eliminate unuseful class relationships in favor of tiered delegation of + program behavior +* to reduce the importance of the YamlParser as the only source of the data + structures necessary for XML generation + + +Problem Description +=================== + +Jenkins Job Builder has history of organic, piecemeal development wherein +patches tend to be highly focused on improving one or another aspect of the +program. For the most part this is understandable, expected even--the modular +implementation of XML generation on a per-Jenkins-plugin basis lends itself well +to this style of development. However when it comes to changes to the core +abstractions, in particular the Builder and YamlParser clases, many ad hoc +design decisions have led to tightly bound implementations that make the overall +API difficult if not impossible to reuse as a library. + +It's true that yes, we can import jenkins_jobs.Builder or +jenkins_jobs.parser.YamlParser to use them programmatically outside of JJB +proper, but it is difficult to imagine anyone actually doing this given the +current constructors for these classes as well as their relationships--Builder +objects actually own references to YamlParser objects when there are no clear +benefits to this organization save for expediency during JJB development. + +The YamlParser itself actually does more than "parse" YAML, it is also +responsible for generating XmlJob objects from the yaml data it handles. This +means that all the lovely XML building code that makes JJB truly special cannot +reliably be accessed using JJB as a library since it depends on implementation +details of the YamlParser (specifically the xml_jobs attribute). + +In addition to the tight binding between classes that make them practically +unusable, there is an almost impossibly tight binding between code in +jenkins_jobs.execute() and jenkins_jobs.Builder, which depends on code in +jenkins_jobs.execute() to munge various command line options and configuration +data, ie to obtain user, password, plugins_list, cache settings, etc. To top it +off, jenkins_jobs.Builder ends up taking the "config" object *anyway*, which +should make us question to what purpose jenkins_jobs.execute() is even handling +various pieces of configuration logic at all--arguably, this logic ought to live +in a separate "Config" class. + +The problems described up to this point indicate a need to clean up the API, but +one issue that really deserves to be driven home is the essential character of +the YamlParser to everything that JJB does. Because of the tight bindings +between classes described above, understanding the tricky behavior of the +YamlParser is absolutely necessary to construct anything more simple than a +collection of "job" objects or even a simple set of "project"s containing lists +of "job-template"s. + +This is primarily problematic because if JJB is going to offer a programmatic +API, it should be possible to use it to do something more interesting than the +exact same thing that is being done in jenkins_jobs.execute(). For example, it +should be possible to define one's own set of abstraction classes in Python that +can be used to generate the data used by jenkins_jobs.Builder to POST jobs to a +Jenkins server. + +JJB plugin modules all take a "parser" argument, but rarely if ever is this +parser argument used in these modules. This parser is intended to be a reference +to a YamlParser instance but the benefit of this relationship does not really +outweigh its cost in terms of tight binding between plugin module and the means +used to obtain JJB data structures actually necessary for generating XML; its +primary use case most of the time is to provide access to the ModuleRegitry +object owned by the YamlParser. It would make more sense to replace this with an +explicit reference to the ModuleRegistry itself. + +A Side Note on Macros +--------------------- + +A problematic JJB feature when it comes to cleaning up the API is the "macro". +The current implementation leads to an unnecessarily tight binding between +the YamlParser and the ModuleRegistry because the ModuleRegistry and XML +generation code require the YamlParser's xml_jobs attribute in order to +determine whether or not a component is a macro. This tight binding is +problematic because it means we need to mock the YamlParser's xml_jobs data +structure in order to use any other means of producing JJB data structures than +the YamlParser itself. As of right now, it is fine to mock YamlParser's xml_jobs +data structure with an empty dictionary but some detail of the way it is +accessed in the ModuleRegistry or XML generation code could easily change that +would cause unforeseen problems when attempting to avoid using the YamlParser. + +Proposed Changes +================ + +Modularize Command Line Parsing +------------------------------- + +Since so much of JJB's API depends on JJB's configuration file and command line +options, it makes sense to modularize these in such a way that they can easily +be used by third party scripts/tools. This will be done using a dedicated +"Config" class. + +This will allow constructors for other JJB classes such as the Builder and +YamlParser to be simplified--rather than taking both a "config" object and a +number of other parameters they will instead take a single "config" parameter. + +Redefine Programmatic APIs +-------------------------- + +The following classes will need their APIs redesigned to make them usable by +third parties: + +* YamlParser +* ModuleRegistry +* Builder + +These rewrites should be accompanied by docstrings that explicitly define the +purpose of these classes so that these purposes can be considered in the context +of future patches against JJB that attempt to make foundational changes to their +behavior. + +YamlParser +'''''''''' + +The current YamlParser exposes all of its methods as "public" methods even when +there is no obvious or implicit value in doing so. The goal for rewriting its +API will be to provide explicit "public" methods that other classes can call to +get information from it. + +ModuleRegistry +'''''''''''''' + +The new ModuleRegistry will serve a dual role; in addition to registering JJB +modules for use during XML generation it will also provide an API for modules to +use to access information other than the given JJB data. For example, it should +provide access to Jenkins plugin version information to allow modules to +implement plugin version dependent XML. + + +Add XmlBuilder class +-------------------- + +The XmlBuilder class should define a method that takes as input a dictionary +where: + +* keys are job names, and +* values are all the key/value pairs necessary to produce an XmlJob for the + named job + +This method should produce as output a list of XmlJobs that can then be passed +to a Builder method either to produce test output or to update jobs on the +configured Jenkins instance. + +Alternate names for this class might be JenkinsXmlBuilder, +JenkinsJobConfigBuilder. + +Replace references to YamlParser with references to ModuleRegistry in modules +----------------------------------------------------------------------------- + +This addresses the need to decouple plugin modules from the YamlParser while +also allowing us to access components of the ModuleRegistry when necessary (ie, +accessing Jenkins plugins_info to generate different XML for different versions +of plugins). + +Rewrite JJB Macro Implementation +-------------------------------- + +The goal of this rewrite should be primarily to remove the need for referencing +the YamlParser's raw yaml data structure while generating XML. This way the +proposed XmlBuilder class does not have to be concerned with any particular +details of the YamlParser--reducing the risk that implementation of one of these +classes has unforeseen side effects on the other's behavior. + +One method that has been proposed is to implement a secondary module registry +that specifically handles macros. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Wayne Warren, wayne+launchpad@sdf.org + +Gerrit Topic +------------ + +Use Gerrit topic "jjb-2.0.0-api" for all patches related to this spec. + +.. code-block:: bash + + git-review -t jjb-2.0.0-api + +Work Items +---------- + +Development Process +''''''''''''''''''' + +All work for this spec will target a feature branch of JJB called "2.0.x" which +will contain all backwards-incompatible changes targeted for this release, +including any work covered by other specs targeting the same release. + +Once the work is complete, we will release Jenkins Job Builder 2.0.0. + +Code Changes +'''''''''''' + +* Break command line parsing function into dynamically-loaded, modular + components for consumption outside of the primary JJB command line tool. +* Create function that specifically handles pre-Builder processing + of command line options and configuration settings; this will be necessary to + make the Builder class usable as an API to be consumed externally. + (Alternatively, consider ways to clean up the Builder constructor in such a + way that doesn't need to take the full "config" object) +* Separate XML generation code from YamlParser, move into new class "XmlBuilder" + that takes a ModuleRegistry object in its constructor and contains various + methods that take one or more JJB job dictionaries and output a corresponding + number of XmlJob objects. +* Separate YamlParser from YamlInterpreter: + * YamlParser will be responsible for loading a given list of YAML files + * YamlInterpreter will be responsible for applying JJB's DSL semantics to the + loaded lists of dictionaries. +* Refactor Builder methods and their calling contexts to simplify the interface; + the "update_jobs" method, for instance, should simply take a list of XmlJobs + to update. + +Repositories +------------ + +None + +Servers +------- + +None + +DNS Entries +----------- + +None + +Documentation +------------- + +Documentation changes may be necessary as this work is primarily intended to +make existing JJB API usable outside of the jenkins_jobs.cmd module. + +Security +-------- + +None + +Testing +------- + +Backwards Compatibility +''''''''''''''''''''''' + +No existing test scenarios will be changed to support this spec; however, there +will need to be changes to test fixture base classes to support the new API +changes. + +We will also monitor openstack-infra's JJB output to ensure that it does not +change because of this spec. + +New Tests +''''''''' + +Unit tests will explicitly validate the behavior of the following: + +* the Builder class' "public" methods +* the YamlParser class' "public" methods +* the XmlBuilder class' "public" methods +* the Config class' "public" methods + +Dependencies +============ + +Python Support +-------------- + +We intend to continue supporting the same Python versions supported by JJB 1.x: + +* Python 2.7 +* Python 3.4 +