- force review method to 'default' if --bypass-gerrit is
selected.
- Allow local branch creation even if remote tracvking
branch is undetermined.
- allow git-utils to call into git-repo-utils to determine
the prefered remote rather than guessing without repo's help.
- allow git_local_branch to return 'HEAD' if there is
no better answer.
- git_remote must never return multiple remotes when
falling back to git_list_remotes.
- git_remote_branch must never return a tag.
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ib1ea9d4ced601e48ac6b5f41c629c9dae82e8108
Github now requires the use of an access token for the
automated push of branches and tags. The alternative
is a password prompt for each git in the middle of the
tool run.
Closes-bug: 1997483
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ife396dfed48a941df4c1e82eef7860017a4ff213
Review.opendev.org seem unreliable when hit
with a lot of transactions in a short time.
This means that an attempt to create a release branch,
e.g. r/stx.6.0 failes repeatedly, and requires a lot
of manual repair.
By adding retires and delays, we greatly improve
our odds of success.
Closes-bug: 1954536
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ia03ed58b67c02e6984a7e0ee01c6666027df6cbe
This content was intended for inclusion vs LP 1924762.
Improve create_branches_and_tags.sh. by adding options to alter
.gitreview settings, including host, port, and project, as part
of branch creation.
Also add improved help/usage text for create_branches_and_tags.sh.
Improve git_remote_branch by adding a fallback method for finding
the remote branch. It finds the nearest branch on the current remote
and reports that. I only expect this to apply when the local branch
hasn't been set to track a remote branch.
Closes-Bug: 1924762
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I790aae159df0a2360d5e50d930fa3d3be038bcc1
* new option --safe-gerrit-host in branching scripts for specifying
gerrit hosts that are safe for pushing reviews to. Allows us to create
gerrit reviews for repos not hosted on opendev.org
* new option --manifest-file for overriding default.xml location, in
case the manifest <include>s a file whose name can't be deduced
automatically
* new option --dry-run in push_* scripts
Story: 2008998
Task: 42661
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I8e7395f7d64c5dfb8936f747d8bea65be22120fd
1) Add default revision and soft/hard lockdown options to create_tags.sh.
These options already appear in the create_branches_and_tags.sh script.
Default revision set the default revision field of the manifest,
removing the need to specify a revision for each project.
Soft lockdown is content with tags already specified in the manifest.
Hard lockdown forces the use of sha's.
2) Improved determination of the local branch.
Need to work around inconsistencies of various git versions.
Also need to deal with 'repo' leaving us in detached head state and
local updates on a detached state, in which case we try
to find the 'best fit' local branch from which we diverged.
3) Ability to ask the branch/tag push scripts to bypass gerrit.
Successful use requires the git repositories and gerrit to
to permit direct push. No true of the current opendev repo setup,
but might be useful for mirrors.
Closes-Bug: 1924762
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I2ba01e11ef27c1d10c816b5af6104afab440b3b3
Previous solution included in incorrect 's' in the sed line
intended to remove .repo/repo git from the build context.
Closes-bug: 1893243
Change-Id: I902844f7e3b0f487e4f062d804880e7ad7363b09
Signed-off-by: Scott Little <scott.little@windriver.com>
Build avoidance does not work reliably because directory
.repo/repo is being included in the build context.
This directory is the repo tool itself, not any of the
gits managed by the repo manifest. The sha reported for
.repo/repo might not be available in users environmnet,
causing build avoidance to reject an otherwise valid
reference context.
Fix is in two parts.
1) Do not include .repo/repo in future build contexts.
2) Filter .repo/repo out of downloaded contexts before
use in build avoidance calculations. This addresses
contexts that are already published.
Closes-Bug: 1893243
Change-Id: I7bd20597cb7bc5ee93ae49728176791c51e89c53
Signed-off-by: Scott Little <scott.little@windriver.com>
Changes in brief:
1) TIS_PATCH_VER will now accept new automatic variables
'OTHER_GITREVCOUNT' and 'SRC_GITREVCOUNT'.
2) New BASE_SRCREV variables are supported, 'SRC_BASE_SRCREV' and
'BASE_SRCREV_FOR_PATH'
3) OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>] joins OPT_DEP_LIST as
a way to list files that affect the build, but are not included
in the assembled src.rpm.
4) Some code relating to interpreting build_srpm.data files,
and producing a list of files that affect the build, are relocated
to 'srpm-util' and restructured for greater utility.
The details:
TIS_PATCH_VER=<expression>
# An integer, or one of the supported
# variables listed below, or the sum of
# variables and integers.
# e.g.
# TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5
PKG_GITREVCOUNT # Count git revisions relative to PKG_BASE.
# Optionally only count from PKG_BASE_SRCREV
SRC_GITREVCOUNT # Count git revisions relative to SRC_DIR.
# Optionally only count from SRC_BASE_SRCREV
GITREVCOUNT # Deprecated, please use SRC_GITREVCOUNT instead.
# Count git revisions relative to SRC_DIR.
# Optionally only count from TIS_BASE_SRCREV
OTHER_GITREVCOUNT # count git revisions from all sources excluding
# PKG_BASE and SRC_DIR
# Optionally only count from
# BASE_SRCREV_FOR_PATH[<path>]
PKG_BASE_SRCREV=<sha> # Limit PKG_GITREVCOUNT revision count to
# commits since <sha>
SRC_BASE_SRCREV=<sha> # Limit SRC_GITREVCOUNT revision count to
# commits since <sha>
TIS_BASE_SRCREV=<sha> # Deprecated, please use SRC_BASE_SRCREV
# instead
# Limit GITREVCOUNT revision count to commits
# since <sha>
BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
# Limit OTHER_GITREVCOUNT revision count for
# commits under <path> to commits since <sha>.
# If <path> is the PKG_BASE of another package
# (not the current package) then the keyword
# 'OTHER_PKG_BASE_SRCREV' can be used to extract
# the 'PKG_BASE_SRCREV' value of the other
# package.
#
# The <path> can reference variables like
# $STX_BASE and $GIT_BASE.
OPT_DEP_LIST=<path-list> # Add a space separated list of paths that
# don't contribute to the content of a src.rpm
# but do contribute to triggering a rebuild,
# and possibly modifying the TIS_PATCH_VER via
# use of OTHER_GITREVCOUNT.
OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
# For a specific build type only, add a space
# separated list of paths that don't
# contribute to the content of src.rpm,
# but do contribute to triggering a
# rebuild, and possibly modifying the
# TIS_PATCH_VER via use of OTHER_GITREVCOUNT.
Story: 2006166
Task: 39765
Change-Id: I9e7d409d4eefdb41a7083db1b801d531c443b678
Signed-off-by: Scott Little <scott.little@windriver.com>
Update branching and tagging tools, used by patching, to do things
the 'repo' way. Repo has replaced wrgit.
This is complicated by ...
1) Not all repos are under our control, and so can't be branched or
tagged by us. The tools now accepts arguments that list 'remotes' or
'projects' to which branches or tags are to be applied
2) The manifest requires an update to reference the new branch or tag
as it applies to affected projects. Non-affected projects can
optionally be locked down by converting the revision to a sha.
The tools must now try to generate that manifest. It might be
sub-optimal. i.e. a revision is explicitly added to each project,
rather than using the 'default' mechanism. Perhaps something to address
in a future update.
ORIGINATING_BRANCH=master
ORIGINATING_BUILD=20200220T023000Z
NEW_BRANCH=r/stx.4.0
STX_REMOTES="starlingx"
repo init -u https://opendev.org/starlingx/manifest -b $ORIGINATING_BRANCH
repo sync --force-sync
curl http://mirror.starlingx.cengn.ca/mirror/starlingx/${ORIGINATING_BRANCH}/centos/${ORIGINATING_BUILD}/outputs/CONTEXT.sh > ${ORIGINATING_BUILD}.context
source ${ORIGINATING_BUILD}.context
create_branches_and_tags.sh --branch=$NEW_BRANCH --remotes=$STX_REMOTES --manifest --lockdown
push_branches_tags.sh --branch=$NEW_BRANCH --remotes=$STX_REMOTES --manifest
ORIGINATING_BRANCH=r/stx.4.0
ORIGINATING_BUILD=20200220T023000Z
NEW_STX_TAG=4.0.0
NEW_GIT_HUB_TAG=stx.4.0.0
STX_REMOTES="starlingx"
GIT_HUB_REMOTES="stx-staging"
repo init -u https://opendev.org/starlingx/manifest -b $ORIGINATING_BRANCH
repo sync --force-sync
curl http://mirror.starlingx.cengn.ca/mirror/starlingx/${ORIGINATING_BRANCH}/centos/${ORIGINATING_BUILD}/outputs/CONTEXT.sh > ${ORIGINATING_BUILD}.context
source ${ORIGINATING_BUILD}.context
TAG=TEST_19.10_PATCH_0000
create_tags.sh --tag=$NEW_STX_TAG --remotes=$STX_REMOTES,$GIT_HUB_REMOTES --manifest --manifest-prefix=stx. --lock-down
create_tags.sh --tag=$NEW_GIT_HUB_TAG --remotes=$GIT_HUB_REMOTES
push_tags.sh --tag=$NEW_STX_TAG --remotes=$STX_REMOTES
push_tags.sh --tag=$NEW_GIT_HUB_TAG --remotes=$STX_REMOTES
cd .repo/manifests
git mv $TAG-default.xml stx.$TAG.xml
vi stx.$TAG.xml
# set revision for all stx-staging projects to refs/tags/stx.4.0.0
<project remote="stx-staging" ... revision="refs/tags/stx.4.0.0" .
vi stx.$TAG.xml
# set default revision
<default remote="starlingx" revision="refs/tags/4.0.0"/>
# delete 'revision' for all starlingx projects
git add stx.$TAG.xml
git commit -s --amend
# set title to
Locked down manifest for StarlingX $TAG
git tag -d $TAG
git review
git tag -s -m "Tag $TAG" $TAG
git push gerrit $TAG
Change-Id: I5ac0c3e44ffda262edb416e877d46c9036cd6e92
Signed-off-by: Scott Little <scott.little@windriver.com>
Build tools need to be made ready for the code
restructuring:
1) new repos
2) relocation of release-info.inc and bsp files
3) Removal of the stx- prefix from sub-repos.
Changes:
1) Create new functions to find the release-info.inc and
bsp file under both pre and post restructure paths.
2) Update .gitignore to ignore all the new repos that
are to be created.
3) Remove an argument sanity check from build-helm-charts.sh.
It is making invalid assumptions about where applications
can be found in the directory structure.
4) build-iso will need to look to additional repos to find
packages from lower layers.
Change-Id: If62444390d0a5a363974c6ff8cdaceca35978bda
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
Fix below issues:
E003 Indent not multiple of 4
E011 Then keyword is not on same line as if or elif keyword
E020: Function declaration not in format "^function name {$":
'function check_all_explicit_deps_installed'
add ignore in tox env
E006 Line too long
E010: Do not on same line as command
Change-Id: I13fb83452324c0db74c485f304827a6c9aacf9ae
Story: 2003370
Task: 24431
Signed-off-by: Sun Austin <austin.sun@intel.com>
First step in build avoidance is the creation of a reference build.
We need to capture the state of all the git trees that contribute
to the reference build.
This update will save the git context to $MY_WORKSPACE/CONTEXT
The context file format is actually a shell script that can be run to
recreate the build context.
repo init ...
repo sync
cd $MY_REPO/..
source .../CONTEXT
Sample CONTEXT file ....
(cd ./cgcs-root && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./cgcs-root/stx/git/calico && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./cgcs-root/stx/git/ceilometer && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
...
(cd ./stx-tools && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./.repo/manifests && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./.repo/repo && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
This update also the git_list function, which populates the previously
introduced GIT_LIST variable, and can be used in several other code
paths where we want to find all gits in a directory tree.
Tested build/clean/edit parallel/serial
Story: 2002835
Task: 22754
Change-Id: I5009b8a360bdb4a03e5a4e83430b7f2ef135115e
Signed-off-by: Scott Little <scott.little@windriver.com>
Currently compiling a new package and adding it
to the iso still requires a multi-git update because
image.inc is a single centralized file in the root git.
It would be better to allow a single git update to add
a package. Too allow this, image.inc must be split across
the git repos and the build tools must be changed to
read/merge those files to arrive at the final package list.
Current scheme is to name the image.inc files using this
schema.
${distro}_${build_target}_image_${build_type}.inc
distro = centos, ...
build_target = iso, guest ...
build_type = std, rt ...
Traditionally build_type=std is omitted from config files,
so we instread use ${distro}_${build_target}_image.inc.
Story: 2003447
Task: 24650
Change-Id: Ib39b8063e7759842ba15330c68503bfe2dea6e20
Signed-off-by: Scott Little <scott.little@windriver.com>
