SQuaRE instructions for making official releases
1 The LSST Science Pipelines Release Process¶
The LSST stack is a large mixed Python/C++ codebase split among many repos with multiple dependencies between them built by a niche build system. The release process is therefore a bit complex and can be protracted. In an attempt not to have to co-ordinate the state of the release with 80+ developers, or worse, have them down tools while a release is being minted, this process has been optimized to impact development as little as possible. Unless a developer needs to be called upon to fix a failing integration or platform test, she can carry on as normal.
In gross, the conceptual steps are:
- A successful weekly release is identified that forms the basis for the official release.
- The first release candidate (
rc1) is created using that weekly build as a seed. This means that it does not matter if the codebase’s master has moved on in the meanwhile.
rcis announced for developer testing
rcare created and announced, if/as necessary to resolve any release blocking issues that may arise.
- Documentation is updated on a branch (release notes, installation instructions, metrics report)
- The final release is created using the last
rcverbatim (ie., the accepted
rcand final release code should be identical except for the name of git/eups tags) and the documentation merged to
1.1 Kicking-off the process¶
The first step is to establish whether it is a good time to make a release, eg.
if many developers are about to push significant features, it may be better to
wait for them to finish. Announce the intent to make a release in the
channel and inform all DM T/CAMs and the DM System Engineer.
If it is generally a good time, identify the nearest weekly release, this will be the seed for the release candidate.
1.1.1 Announce start of release process¶
1.1.2 Branching the docs¶
At this point you should branch pipelines_lsst_io. so as not to capture any
changes on the
master branch that may occur during the release process.
Note that the branch does not have a
git clone https://github.com/lsst/pipelines_lsst_io.git cd pipelines_lsst_io git checkout -b 888.0.x git push -u origin 888.0.x
1.1.3 Identify base weekly build¶
Identify the git tag of the weekly build you wish to base the release release
candidate upon, say
w.9999.52. This should be determined by discussion
with the product owner and team developer.
Example method for listing weekly git tags:
git clone https://github.com/lsst/lsst_distrib.git cd lsst_distrib git tag -l w.*
1.2 1st Release Candidate¶
1.2.1 Build and Publish¶
Run the jenkins official-release job. The source git refs should be only the
tag of the “seed” weekly release. The release tag must start with a
and end with
See git-tags for details on the formatting of git tags.
SOURCE_GIT_REFS: w.9999.52 RELEASE_GIT_TAG: v888.0.0.rc1 O_LATEST: false
1.3 2nd+ Release Candidate(s)¶
.rcX, where X is
> 1, is only required if a problem is found with
rc differs slightly from the initial
because it inherently is not identical to a previous
git tag (if it was,
there would be no reason to produce another
rc). The creation of a git
release branch prior to
rc1 would eliminate the differences.
1.3.1 Branch, Merge¶
Any git repository that needs to be modified for additional
should be branched and have the necessary changes merged to a release
branch. Eg., if changes were needed in
v888.0.0.rc1 a “release branch”
along the lines of
v888.0.x should be created in the repos that need
(TBD: merge to master and cherry-pick to release branch or merge to release
branch and merge to
1.3.2 Test Release Branch(es)¶
As a sanity check, run the jenkins stack-os-matrix job to verify that the
release branch + previous
rcX tag combination is buildable prior to
attempting to build and publish the new
REFS: v888.0.x v888.0.0.rc1 PRODUCTS: lsst_distrib lsst_ci
1.3.3 Build and Publish¶
Run the jenkins official-release job.
For input source git refs use the previous
rc tag along with the release
branch(es). Ensure that the release branch is specified to the left of
rcX tag in the listing of git refs.
SOURCE_GIT_REFS: v888.0.x v888.0.0.rc1 RELEASE_GIT_TAG: v888.0.0.rc2 O_LATEST: false
SOURCE_GIT_REFS: v888.0.x v888.0.0.rc5 RELEASE_GIT_TAG: v888.0.0.rc6 O_LATEST: false
1.4 Final Release¶
Note that a Final Release differs from a Release Candidate in that the DM
internal/first party git repositories receive a git tag that does not have
an alphabetic prefix (eg.,
v). This has the effect of changing the eups
version strings as
lsst-build sets the eups product version based on the
most recent git ref that has an integer as the first character.
A consequence of this behavior is that the final git tag must be present
prior to the production of
1.4.1 Build and Publish¶
Run the jenkins official-release job.
The source input must only be the latest
O_LATEST flag controls if the produced science pipelines docker image
-o_latest docker tags applied to it. This should only be set on a
final release AND only if the release is the highest version release. For
99.0.0 has been release and a
98.0.1 bugfix release is
O_LATEST should not be set.
SOURCE_GIT_REFS: v888.0.0.rc6 RELEASE_GIT_TAG: 888.0.0 O_LATEST: false
In this process we make use of the fact that git doesn’t care whether a ref is
a tag or a branch to constrain the number of branches to repositories that need
retroactive maintenance or need to be available in more than one cadence. One
such example is the
lsst repo since it contains newinstall.sh which
sets the version of eups, and that may be different for an official release
than the current bleed.
Note that the branch does not have
Branch the lsst repo:
git clone https://github.com/lsst/lsst.git cd lsst git checkout -b 888.0.x git push -u origin 888.0.x
lsst/scripts/newinstall.sh change the canonical reference for this
newinstall to be one associated with the current branch:
and commit and push.
This means that if you need to update
newinstall.sh for bleed users,
official-release users will not be prompted to update to the latest version,
but will phone home against their official-release branch for hotfixes.
Also double-check for other things that might need to be updated, like the documentation links (though these should really be fixed on master prior to branching or cherry-picked back).
Documentation to be collected for the release notes in pipelines_lsst_io is:
- Release notes from the T/CAMs for Pipelines, SUI, and DAX
- Characterization report from the DM or SQuaRE scientist
- Known issues and pre-requisites from the T/CAM for SQuaRE
- Before merging to master, ask the Documentation Engineer to review
- Update the
newinstall.rstpage on your release branch of pipelines_lsst_io with the new download location of the
1.5 Other OS checking¶
While we only officially support the software on certain platforms (RHEL/CentOS 7 is the reference, and we CI MacOS and RHEL 6), we check in a number of other popular platforms (eg Ubuntu, newer versions of CentOS etc) by spinning up machines on Digital Ocean (typically) and following the user install instructions. This also allows us to check the user from-scratch installation instructions including the pre-requisites.
2 c.l.o stubb¶
Here is where we currently are in the release process. Current step in bold. Precursor Steps --------------------------------- 1. Identify any pre-release blockers ("must-have features") :tools: 2. Wait for them to clear Release Engineering Steps ------------------------------- 1. Branch 888.0.x of newinstall.sh 1. Build and publish rc1 release candidate (based on w.9999.52) 1. **Wait for first round of bugs to clear** 1. Build and publish additional rcX releases if/as necessary 1. Full OS testing (see https://ls.st/faq ) 1. Build and publish final release Documentation Steps ------------------------- 1. Update Prereqs/Install 1. Update Known Issues 1. Gather Release notes 1. Gather Metrics report 1. **Email announcement**
3 Github teams¶
There are three “special” teams in the LSST Github org:
These are used in the release process in the following way:
Data Managementrepos are a dependency of
lsst_distriband should be tagged with the bare release version, eg.
888.0.0, unless the repo is also a member of the
DM Externalsteam. All repos tagged as part of a release should be members of the
Data Managementteam to ensure that DM developers are able to modify all components of a release.
DM Externalsalso indicates a dependency of
lsst_distribbut one that is tagged with a
vprefix in front of the release version. Eg.,
v888.0.0This is required because
lsst-buildderives the eups product version string from git tags that begin with a number. DM developers prefer that eups display external packages version string rather than of a DM composite release. Thus the
vprefix causes the git tag to be ignored by
lsst_distrib. “External” repos must not also be members of
DM Auxilliariesare repos that we want to snapshot as part of a release but are not an eups dependency of
lsst_distrib. “Aux” repos must not also be members of
5 Conda Environment/Packages Update¶
There are conflicting pressures of updating the conda package list frequently
to minimize the amount of [likely] breakage at one time and resisting changes
as the git
sha1 of the conda environment files is used to defined the
ABI of the eups
5.1 Adding a new Conda package¶
The name of the package needs to “bleed” or un-versioned environment files in the
lsst/scipipe_conda_envrepo. Which are:
After the implementation of DM-17457, the conda environments have been migrated to
yamlformat. This permits to add pip packages to the environment definition.
The bleed env files should be keep in sync with the exception of the
nomklpackage, which is required on
linux. Also note that the env files should be kept sorted to allow for clean
The regular conda env files need to be updated by running a fresh install with
deploy -b`(bleed install) and then manually exporting the env to a file. A side effect of this is other package versions will almost certainly change and this is an ABI breaking event. The existing env files are:
conda list -eshould be run on
osxinstalls and the results committed for both platforms as a single commit so that the the abbrev sha1 of the latest commit for both files will be the same.
As an abbreviated sha1 of the
lsst/lsstswrepo is used to select which [version of] conda env files are used and to define the eups binary tarball “ABI”, jenkins needs to know this value to ensure that
newinstall.shis explicitly using the correct ref and to construct the paths of the tarball
LSST_SPLENV_REFneeds to be updated at:
The ~last major release should be rebuilt in the new “ABI”
EUPS_PKGROOTso that that newinstall.sh from master will still be able to do a binary install of the current major release. This may be done by triggering a Jenkins