infra-manual/doc/source/creators.rst

title

Repository Creator's Guide

Repository Creator's Guide

Before You Start

This is a long document. It's long because it has to be, not because we want it to be. If you follow it, everything will be fine.

It is important that you perform all of the steps, in the order they are given here. Don't skip any steps. Don't try to do things in parallel. Don't jump around.

Choosing a Good Name for Your Repository

It is important to choose a descriptive name that does not conflict with other repositories. There are several places you'll need to look to ensure uniqueness and suitability of the name.

Note

If you encounter any issues establishing a valid unique name across all of the tools we use, consult with the Release Manager before going any further.

Character Set

We prefer to use names made up of lower case ASCII letters and the - punctuation symbol to avoid issues with various installation tools.

git repository

The base name of the repository should be unique across all of the namespace directories for git repositories under https://git.openstack.org/cgit. That is, it is not sufficient to have openstack/foo and openstack-dev/foo because that prevents us from moving those two repositories into the same namespace at some point.

Launchpad

It is preferred but not absolutely necessary for your project name on https://launchpad.net to be the same as your git repository name. Try "python-" as a prefix if necessary (for example, "python-stevedore").

PyPI

Python packages need to have a unique name on the Python Package Index (https://pypi.python.org) so we can publish source distributions to be installed via pip.

It is best to name the repository and the top level Python package the same when possible so that the name used to install the dist and the name used to import the package in source files match. Try "python-" as a prefix if necessary (for example, "python-stevedore").

Project Team Rules

Some OpenStack project teams have naming conventions that must be followed. For example, the Oslo team has instructions for choosing a name for new Oslo libraries.

Set up Launchpad

OpenStack uses https://launchpad.net for project management tasks such as release planning and bug tracking. The first step to importing your repository is to make sure you have the right project management tools configured.

Create a new Launchpad Project

1. Visit https://launchpad.net/projects/+new and fill in the details.
2. Name your project using the same name you plan to use for the git repository, unless that is taken. Try "python-" as a prefix if necessary (for example, "python-stevedore"). If that name is also taken, consult with the Release Manager before going any further.

Put Your New Project in the Correct Project Group

1. From the Overview page of your project, select "Change Details" from the right sidebar (e.g., http://launchpad.net/oslo.foo/+edit).
2. Find the "Part of" field and set the value to "openstack" for integrated projects and "oslo" for Oslo libraries.
3. Save your changes.

Create Bug Tracker

1. From the Overview page for your project, click the "Bugs" link at the top of the page. Launchpad should suggest that you set up bug tracking.

2. Choose "In launchpad".

3. Check the box labeled "Expire 'Incomplete' bug reports when they become inactive"

4. Check the box labeled "Search for possible duplicate bugs when a new bug is filed"

5. Set the "Bug supervisor" field to "<projectname>-bugs" (for example, "oslo-bugs").

   Note

   You may need to create the bug management team in Launchpad. If you do so, set the owner of the team to the "OpenStack Administrators team" called "openstack-admins".

6. Save your changes.

Create Blueprint Tracker

If your project uses Launchpad blueprints to track new feature work, you should set up the blueprint tracker now. Otherwise, skip this step.

1. From the Overview page for your project, click the "Blueprints" link at the top of the page. Launchpad should suggest that you set up blueprint tracking.
2. Choose "Launchpad".
3. Save your changes.

Set up Supervisors for your Project

From the Overview page for your project, click the pencil "edit" icon next to the Maintainer field. Replace your name with the <projectname>-drivers team (for example, "oslo-drivers").

Note

You may need to create the drivers team. If you do:

1. Set the owner of the team to 'openstack-administrators'
2. Add 'openstack-release' as a sub-team so the release team can help keep milestones updated as part of the release process.

From the Overview page for your project, click the pencil "edit" icon next to the Drivers field. Replace your name with the project drivers team.

Note

If either of these steps makes it so you cannot edit the project, stop and ask someone in the drivers group to help you before proceeding.

Give OpenStack Permission to Publish Releases

New packages without any releases need to be manually registered on PyPI.

If you do not have PyPI credentials, you should create them at https://pypi.python.org/pypi?%3Aaction=register_form as they are required for the next step.

Once you have PyPI credentials visit https://pypi.python.org/pypi?%3Aaction=submit_form and fill in only the required fields.

Next your package needs to be updated so the "openstackci" user has "Owner" permissions.

Visit https://pypi.python.org/pypi?:action=role_form&package_name=<packagename> and add "openstackci" in the "User Name" field, set the role to "Owner", and click "Add Role".

Adding the Repository to the CI System

To add a repository to the CI System, you need to modify some infrastructure configuration files using git and the OpenStack gerrit review server.

All of the changes described in this section should be submitted together as one patchset to the openstack-infra/project-config repository.

Add the repository to the master repositories list

1. Edit gerrit/projects.yaml to add a new section like:

   - project: openstack/<repositoryname>
     description: Latest and greatest cloud stuff.

2. Provide a very brief description of the library.

3. If you have an existing repository that you want to import (for example, when graduating an Oslo library or bringing a repository into gerrit from github), set the "upstream" field to the URL of the publicly reachable repository and also read the information in setup_review:

   - project: openstack/<repositoryname>
     description: Latest and greatest cloud stuff.
     upstream: git://github.com/awesumsauce/<repositoryname>.git

   Note

   If the git repository short name does not match the Launchpad project name, you need to add a "groups" list to provide the mapping. The groups list is also used by Storyboard to be able to present grouped views of stories and tasks across multiple related repositories.

   For example, Oslo repositories should use "oslo" to ensure that they are associated with the https://launchpad.net/oslo project group for tracking bugs and milestones:

   - project: openstack/<repositoryname>
     description: Latest and greatest cloud stuff.
     upstream: git://github.com/awesumsauce/<repositoryname>.git
     groups:
        - oslo

Add Gerrit permissions

Each repository should have two gerrit groups. The first, "<projectname>-core", is the normal core group, with permission to +2 changes. The second, "<projectname>-release" is a small group of the primary maintainers with permission to push tags to trigger releases. Libraries for projects managed by the release team do not need this group, and should use library-release instead.

Create a gerrit/acls/openstack/<repositoryname>.config as explained in the following sections.

Minimal ACL file

The minimal ACL file allows working only on master and requires a change-ID for each change:

[access "refs/heads/*"]
abandon = group <projectname>-core
label-Code-Review = -2..+2 group <projectname>-core
label-Workflow = -1..+1 group <projectname>-core

[receive]
requireChangeId = true

[submit]
mergeContent = true

Request Signing of ICLA

If your repository requires signing of the Individual Contributor License Agreement (ICLA), change the receive section to:

[receive]
requireChangeId = true
requireContributorAgreement = true

Note that this is mandatory for all official projects in the openstack namespace and should also be set for repositories that want to become official.

Creation of Tags

For library repositories managed by the release team, allow the library-release team to create tags by adding a new section containing:

[access "refs/tags/*"]
pushSignedTag = group library-release

For non-library projects, or projects not managed by the release team, you can allow the project-specific release team to create tags by adding a new section containing:

[access "refs/tags/*"]
pushSignedTag = group <projectname>-release

Creating of Branches

To allow creation of branches to the release team, add a create rule to it the refs/heads/* section:

[access "refs/heads/*"]
abandon = group <projectname>-core
create = group <projectname>-release
label-Code-Review = -2..+2 group <projectname>-core
label-Workflow = -1..+1 group <projectname>-core

Extended ACL File

So, if your repository requires the ICLA signed, has a release team that will create tags and branches, create a gerrit/acls/openstack/<repositoryname>.config like:

[access "refs/heads/*"]
abandon = group <projectname>-core
create = group <projectname>-release
label-Code-Review = -2..+2 group <projectname>-core
label-Workflow = -1..+1 group <projectname>-core

[access "refs/tags/*"]
pushSignedTag = group <projectname>-release

[receive]
requireChangeId = true
requireContributorAgreement = true

[submit]
mergeContent = true

See other files in the same directory for further examples.

Add Basic Jenkins Jobs

Test jobs run through Jenkins, and the jobs are defined using jenkins-job-builder configuration files.

Note

Different repositories will need different jobs, depending on their nature, implementation language, etc. This example shows how to set up a new Python code repository because that is our most common case. If you are working on another type of repository, you will want to choose different jobs or job templates to include in the "jobs" list.

Edit jenkins/jobs/projects.yaml to add your repository. There are several sections, designated in comments, for different types of repositories. Find the right section and then add a new stanza like:

- project:
   name: <repositoryname>
   node: 'bare-precise || bare-trusty'
   tarball-site: tarballs.openstack.org
   doc-publisher-site: docs.openstack.org
   jobs:
     - python-jobs
     - openstack-publish-jobs
     - pypi-jobs

Configure Zuul to Run Jobs

Zuul is the gate keeper. It watches for changes in gerrit to trigger the appropriate jobs. To start, establish the rules for the jobs you need.

Note

Different repositories will need different jobs, depending on their nature, implementation language, etc. This example shows how to set up the full set of gate jobs for a new Python code repository because that is our most common case. If you are working on another type of repository, you will want to choose different jobs or job templates to include here.

Edit zuul/layout.yaml to add your repository. There are several sections, designated in comments, for different types of repositories. Find the right section and then add a new stanza like:

- name: openstack/<repositoryname>
  template:
    - name: merge-check
    - name: python-jobs
    - name: openstack-server-publish-jobs
    - name: check-requirements
    - name: integrated-gate
    - name: publish-to-pypi
    - name: python3-jobs
    - name: translation-jobs

You can find more info about job templates in the beginning of zuul/layout.yaml in the section starting with "project-templates:".

Note

If you use pypi-jobs and publish-to-pypi, please ensure your repository's namespace is registered on https://pypi.python.org as described in register-pypi. This will be required before your change is merged.

If you are not ready to run any tests yet and did not configure python-jobs in jenkins/jobs/projects.yaml, the entry for zuul/layout.yaml should look like this instead:

- name: openstack/<repositoryname>
  template:
    - name: merge-check
    - name: noop-jobs

Configure GerritBot to Announce Changes

If you want changes proposed and merged to your repository to be announced on IRC, edit gerritbot/channels.yaml to add your new repository to the list of repositories. For example, to announce changes related to an Oslo library in the #openstack-oslo channel, add it to the openstack-oslo section:

openstack-oslo:
  events:
    - patchset-created
    - x-vrif-minus-2
  projects:
    - openstack/cliff
    - openstack/oslo.config
    - openstack/oslo-incubator
    - openstack/oslo.messaging
    - openstack/oslo.rootwrap
    - openstack/oslosphinx
    - openstack/oslo-specs
    - openstack/oslo.test
    - openstack/oslo.version
    - openstack/oslo.vmware
    - openstack/stevedore
    - openstack/taskflow
    - openstack-dev/cookiecutter
    - openstack-dev/hacking
    - openstack-dev/oslo-cookiecutter
    - openstack-dev/pbr
  branches:
    - master

If you're adding a new IRC channel, see the IRC services documentation.

Submitting Infra Change for Review

When submitting the change to openstack-infra/project-config for review, use the "new-project" topic so it receives the appropriate attention:

$ git review -t new-project

Note the Change-Id in your commit message for the next step.

Add New Repository to the Governance Repository

Each repository managed by an official OpenStack project team needs to be listed in reference/projects.yaml in the openstack/governance repository to indicate who owns the repository so we know where ATCs voting rights extend.

If your repository is under the stackforge section of the git repository structure, you can skip this step.

Find the appropriate section in reference/projects.yaml and add the new repository to the list. For example, to add a new Oslo library edit the "Oslo" section:

Oslo:
  ptl: Doug Hellmann (dhellmann)
  service: Common libraries
  mission:
    To produce a set of python libraries containing code shared by OpenStack
    projects. The APIs provided by these libraries should be high quality,
    stable, consistent, documented and generally applicable.
  url: https://wiki.openstack.org/wiki/Oslo
  tags:
    - name: team:diverse-affiliation
  projects:
    - repo: openstack/oslo-incubator
      tags:
        - name: release:has-stable-branches
    - repo: openstack/oslo.config
      tags:
        - name: release:independent
        - name: release:has-stable-branches
    - repo: openstack/oslo.messaging
      tags:
        - name: release:independent
        - name: release:has-stable-branches
    - repo: openstack/oslo.rootwrap
      tags:
        - name: release:independent
        - name: release:has-stable-branches
    - repo: openstack/oslosphinx
      tags:
        - name: release:independent
        - name: release:has-stable-branches
    - repo: openstack-dev/cookiecutter
    - repo: openstack-dev/pbr
      tags:
        - name: release:independent

When writing the commit message for this change, make this change depend on the repository creation change by including a link to its Change-ID (from the previous step):

Depends-On: <Gerrit Change-Id>

Then, go back to the project-config change and add a review comment that includes a link to the change in the governance repository, so that reviewers know that the governance change has been created.

However, if you are creating an entirely new OpenStack project team (i.e., adding a new top-level entry into reference/projects.yaml), you should reverse the dependency direction (the repository creation change should depend on the governance change because the TC needs to approve the new project team application first).

Wait Here

The rest of the process needs this initial import to finish, so coordinate with the Infra team, and read ahead, but don't do any of these other steps until the import is complete and the new repository is configured.

The Infra team can be contacted via IRC on Freenode in the #openstack-infra channel or via email to the openstack-infra mail list.

Update the Gerrit Group Members

After the review is approved and groups are created, ask the Infra team to add you to both groups in gerrit, and then you can add other members.

The project team lead (PTL), at least, should be added to "<projectname>-release", and other developers who understand the release process can volunteer to be added as well.

Updating devstack-vm-gate-wrap.sh

The devstack-gate tools let us install OpenStack repositories in a consistent way so they can all be tested with a common configuration. If your repository will not need to be installed for devstack gate jobs, you can skip this step.

Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new repository:

PROJECTS="openstack/<repositoryname> $PROJECTS"

Keep the list in alphabetical order.

Add Repository to the Requirements List

The global requirements repository (openstack/requirements) controls which dependencies can be added to a repository to ensure that all of OpenStack can be installed together on a single system without conflicts. It also automatically contributes updates to the requirements lists for OpenStack repositories when the global requirements change.

If your repository is not going to participate in this requirements management, you can skip this step.

Edit the projects.txt file to add the new library, adding "openstack/<repositoryname>" in the appropriate place in alphabetical order.

Preparing a New Git Repository using cookiecutter

All OpenStack repositories should use one of our cookiecutter templates for creating an initial repository to hold the source code.

If you had an existing repository ready for import when you submitted the change to project-config, you can skip this section.

Start by checking out a copy of your new repository:

$ git clone git://git.openstack.org/openstack/<repositoryname>

$ pip install cookiecutter

Choosing the Right cookiecutter Template

The template in openstack-dev/cookiecutter is suitable for most repositories.

$ cookiecutter https://git.openstack.org/openstack-dev/cookiecutter

The template in openstack-dev/oslo-cookiecutter should be used for Oslo libraries.

$ cookiecutter https://git.openstack.org/openstack-dev/oslo-cookiecutter

Applying the Template

Running cookiecutter will prompt you for several settings, based on the template's configuration. It will then update your repository with a skeleton, ready to have your other files added.

$ cd <repositoryname>
$ git review

If you configured all of the tests for the repository when it was created in the previous section, you will have to ensure that all of the tests pass before the cookiecutter change will merge. You can run most of the tests locally using tox to verify that they pass.

Verify That Gerrit and the Test Jobs are Working

The next step is to verify that you can submit a change request for the repository, have it pass the test jobs, approve it, and then have it merge.

Configure git review

If the new repository you have added has a specified upstream you will need to add a .gitreview file to the repository once it has been created. This new file will allow you to use git review.

The basic process is clone your new repository, add file, push to Gerrit, review and approve:

$ git clone https://git.openstack.org/openstack/<repositoryname>
$ cd <repositoryname>
$ git checkout -b add-gitreview
$ cat > .gitreview <<EOF
[gerrit]
host=review.openstack.org
port=29418
project=openstack/<repositoryname>.git
EOF
$ git review -s
$ git add .gitreview
$ git commit -m 'Add .gitreview file'
$ git review

Verify that the Tests Pass

If you configure tests for an imported repository, ensure that all of the tests pass successfully before importing. Otherwise your first change needs to fix all test failures. You can run most of the tests locally using tox to verify that they pass.

Verify the Gerrit Review Permissions

When your repository is added to gerrit, the groups defined in the ACLs file (see add-gerrit-permissions) are created, but they are empty by default. Someone on the infrastructure team with gerrit administrator privileges will need to add you to each group. After that point, you can add other members.

To check the membership of the groups, visit https://review.openstack.org/#/admin/projects/openstack/<repositoryname>,access -- for example, https://review.openstack.org/#/admin/projects/openstack-infra/infra-manual,access -- and then click on the group names displayed on that page to review their membership.

Prepare an Initial Release

Make Your Repository Useful

Before going any farther, make the repository do something useful.

If you are importing an existing repository with features, you can go ahead.

If you are creating a brand new repository, add some code and tests to provide some minimal functionality.

Provide Basic Developer Documentation

Update the README.rst file to include a paragraph describing the new repository.

Update the rest of the documentation under doc/source with information about the public API, tips on adopting the tool, instructions for running the tests, etc.

Tagging a Release

To verify that the release machinery works, push a signed tag to the "gerrit" remote. Use the smallest version number possible. If this is the first release, use "0.1.0". If other releases of the repository exist, choose an appropriate next version number.

Note

You must have GnuPG installed and an OpenPGP key configured for this step.

Run:

$ git tag -s -m "descriptive message" $version
$ git push gerrit $version

Wait a little while for the pypi job to run and publish the release.

If you need to check the logs, you can use the git-os-job command:

$ git os-job $version

Allowing Other OpenStack Repositories to Use Your Library

OpenStack repositories share a common global requirements list so that all components can be installed together on the same system. If you are importing a new library repository, you need to update that list to allow other repositories to use your library.

Update the Global Requirements List

Check out the openstack/requirements git repository and modify global-requirements.txt to:

1. add the new library
2. add any of the library's direct dependencies that are not already listed

Setting up Gate Testing

The devstack gate jobs install all OpenStack repositories from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of repositories in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the repository needs to be added to the appropriate library file of devstack.

Updating devstack

1. Check out openstack-dev/devstack.

2. Edit the appropriate repository file under lib to add a variable defining where the source should go. For example, when adding a new Oslo library add it to lib/oslo:

   <REPOSITORYNAME>_DIR=$DEST/<repositoryname>

3. Edit the installation function in the same file to add commands to check out the repository. For example, when adding an Oslo library, change install_oslo in lib/oslo.

   When adding the new item, consider the installation order. Dependencies installed from source need to be processed in order so that the lower-level packages are installed first (this avoids having a library installed from a package and then re-installed from source as a dependency of something else):

   function install_oslo() {
     ...
     _do_install_oslo_lib "<repositoryname>"
     ...
   }

4. Edit stackrc to add the other variables needed for configuring the new library:

   # new-repository
   <REPOSITORYNAME>_REPO=${<REPOSITORYNAME>_REPO:-${GIT_BASE}/openstack/<repositoryname>.git}
   <REPOSITORYNAME>_BRANCH=${<REPOSITORYNAME>_BRANCH:-master}

Add Link to Your Developer Documentation

Update the http://docs.openstack.org/developer/openstack-projects.html page with a link to your documentation by checking out the openstack/openstack-manuals repository and editing www/developer/openstack-projects.html.

Skip this step if your repository is under stackforge.

Project Renames

The first step of doing a rename is understanding the required governance changes needed by the rename. You can use the following criteria:

For new project moving from Stackforge into the big tent: Add a "Depends-On: GovernanceID" of the openstack/governance change that accepted the project into the big tent to the commit message.

For a project being added to existing non-Stackforge project: Create an openstack/governance change and add a "Depends-On: project-changeID" of the change you make in the following steps to the commit message, and add a comment in the openstack-infra/project-config change that references the governance change. You will also make sure the PTL has expressed approval for the addition in some way.

When preparing to rename a project, begin by making changes to the files in the openstack-infra/project-config repository related to your project.

When uploading your change, make sure the topic is "project-rename" which can be done by submitting the review with the following git review command:

$ git review -t project-rename

Members of the infrastructure team will review your change.

Finally, add it to the Upcoming Project Renames section of the Infrastructure Team Meeting page to make sure it's included in the next rename window.

Note

Renames have to be done during a Gerrit maintenance window scheduled by the Infrastructure team, so it may take a few weeks for your rename to be completed.

Post rename, a member of the Infrastructure team will submit a patch to update the .gitreview file in the renamed project to point to the new project name.

Other projects you may need to update post-rename:

• projects.txt in openstack/requirements