The Apache Software Foundation
Apache Incubator

A Guide To Release Management During Incubation (DRAFT)

Contents

Status - DRAFT

This document is under active development. This is a first draft intended to allow public review.

Abstract

This guide is descriptive, not normative. It has three main purposes.

  1. Summarize and provide references to normative release policy.
  2. Describe the process of creating an incubating release.
  3. Serve as a repository for non-binding best-practice guidelines.

Releases are important:

  • Publishing software has legal consequences.
  • Most users interact with a project only through its releases. They are a public face of the project and are often the first chance to make a good impression.
  • Poor quality releases reflect badly not only on the project but also the foundation as a whole.

Incubating releases, despite the "incubating" label and disclaimer, are official Apache releases. (Legally, they are the product of the Incubator PMC acting on behalf of the Foundation.) As such, they are expected to adhere to all Foundation-wide release policies. While deviations are occasionally allowed, each exception must be documented and approved by the IPMC.

TODO: Move the remaining paragraphs in this section elsewhere.

Apache release policy is minimal (though its principles have some subtle consequences). However, most projects have a lot more ceremony, rules and traditions to ensure high quality releases. These often evolve over time. Often projects realise too late that these need to be documented.

Podlings can short circuit this process by starting out with written release documentation. It is strongly recommended that Podlings invest time looking at the best practices recommended in this document. By selection and modification, sections of this document can be used to quickly and easily bootstrap a release guide.

References

For normative release policy, consult the following resources:

Release Managers and PPMC members should also familiarize themselves with the following:

See Also

This page spells out requirements and best practices which apply equally to all Incubator podlings. Additional information which applies only to specific podlings is available on auxiliary pages.

Java-specific Release Management Issues

The Java-specific Release Management Issues page covers topics pertaining to Java, Maven, Ant, and so on.

Release Check List

Each review item in this list is either required by Foundation-wide policy and would block a release by any Apache top-level project, or is required by Incubator policy.

1.1 Checksums and PGP signatures are valid.
See the Release Signing dev documentation.
2.1 Build is successful including automated tests.
The expanded source archive is expected to build and pass tests.
3.1 DISCLAIMER is correct, filenames include "incubating".
See the Podling Branding Guide.
3.2 Top-level LICENSE and NOTICE are correct for each distribution.
See the Licensing How-To, plus various pages under Legal Affairs.
3.3 All source files have license headers where appropriate.
See the ASF Source Header and Copyright Notice Policy.
3.4 The provenance of all source files is clear (ASF or software grants).
See the IP clearance section of the Mentor's guide, as well as the Releases section of the Incubator's policy page.
3.5 Dependencies licenses are ok as per http://apache.org/legal/
See ASF Legal Previously Asked Questions.
3.6 Release consists of source code only, no binaries.
Each Apache release must contain a source package. This package may not contain compiled components (such as "jar" files) because compiled components are not open source, even if they were built from open source.

A list of possible additional items is maintained on the ReleaseChecklist wiki page.

Guidelines

Distributing Releases

Once a release has been approved by the Incubator PMC, it needs to be uploaded to the servers for wider distribution. A description of this process and the policy governing it follows.

Attention! Sometimes you need to talk to Infra before distributing a release, esp. when your release artifacts are huge.

Incubator Distribution Directory

All Incubator authorised by the incubator are contained within the Incubator distribution directory:

https://dist.apache.org/repos/dist/release/incubator

Podling Distribution Directory

All releases created by a podling are contained within a sub-directory of the Incubator distribution directory. This is www.apache.org/dist/incubator/{podling} where {podling} is the name of the podling.

For example, the podling distribution directory for podling foo is https://dist.apache.org/repos/dist/release/incubator/foo.

Policy Overview

Distribution policy with regard to releases is simple:

  • The Incubator insists that artifacts for {podling} are contained within www.apache.org/dist/incubator/{podling}.
  • Infrastructure insists that releases are mirrored and that signatures+checksums are uploaded for every artifact.

All non-release software distributions (e.g. links to the subversion repository, automated builds, etc.) are reserved for folks participating in product development on the project's developer list. See this description for details on the policy.

The rest is up to the community to decide and that's quite a lot. Documenting the release management processes is important and often neglected. It is strongly recommended that the rest of this section is used as the basis for a written release guide for the podling.

Understanding Release Distribution

Uploading Artifacts

Released artifacts are distributed at (www.apache.org/dist) for all Apache projects. New files can be added to this site via the svn repository at (https://dist.apache.org/repos/dist/release). Each project (including the Incubator) owns a directory within /release.

The podling distribution directory is contained within this Incubator distribution directory. The Incubator Project Management Committee (IPMC) is responsible for all releases. Arrangement and management of releases within each podling distribution directory is delegated to the appropriate podling.

Release artifacts can be published to the podling distribution directory using subversion. If you do not see a subversion directory corresponding to your project, make an INFRA JIRA request. Additional information is available at the Publishing Releases page.

Mirroring

To avoid excessive use of bandwidth and to increase download speeds, official releases are made available through a global network of volunteer mirrors. Using these mirrors has some notable differences from unmirrored downloads. In particular, a script must be used to direct the download to an appropriate URL.

Users will download the mirrored release artifacts from machines outside Apache control. Users need to verify that the copy downloaded is identical to the original. Mirrored copies of checksums, KEYS and signature files (.asc and .md5 files) will be present on the mirrors but must never be used for verification. So, all links from the podling website to signatures, sums and KEYS need to refer to the original documents on www.apache.org. See release signing guide for more information.

Archiving

All Apache releases form an important part of the history of a project. They are therefore archived with the aim of preserving them indefinitely for future reference.

http://archive.apache.org is the archive host. Releases are archived under http://archive.apache.org/dist. Please remember that these archives are served from Apache bandwidth. Anyone who wants to obtain a large quantity of data from the archives should contact the Infrastructure Team.

All artifacts within www.apache.org/dist will be automatically archived. When a new artifact is uploaded, it will be sync'd to the archive. The sync'ing is scheduled to operate several times a day. So it may be some hours before an added artifact is archived.

When an (archived) artifact is deleted from the live distribution, it will remain in the archives. See how to archive.

Security

Sums And Signatures

Start by reading the guide and policy.

Sums are a convenient and simple way for users to verify releases. Signatures play a critical role in ensuring security for releases.

If a release is signed by a key that is strongly connected to the Apache web of trust then it can be verified (by anyone with access to that web of trust) that a file has not been modified (by anyone without access to the corresponding private key). This allows the infrastructure team to check that releases have not been tampered with.

So, it is crucial that new release managers ensure:

  • that the code signing private key is kept safe.
  • that the KEYS file contains the public key. (Storing public keys in a KEYS file is recommended but is not policy.)

It is important that the key is linked to the Apache web of trust.

Always upload the checksums and signatures at the same time as the artifact. This will ensure no false alarms from the infrastructure team.

Modifications

Once an artifact has been uploaded, it must never be modified. Not only is there no guarantee that the modified artifact will be correctly archived or mirrored but a change to an existing artifact is the signature of an attack.

This applies only to release artifacts. It's expected that README's, NOTES, KEYS, and so on may be updated.

Distribution Check List

  • Directory is www.apache.org/dist/incubator/podling
  • Group is incubator
  • All artifacts have matching checksums and signatures
  • Old releases archived
  • Links to KEYS, signature and checksum documents refer to the originals on www.apache.org

Best Practice

Layout

A podling is free to choose a suitable layout for its released artifacts within the podling distribution directory . It is recommended that standard layouts (commonly used by other projects) be studied and that the layout adopted is documented in the podling's release documentation. The descriptions which follow can be used as a useful basis for this documentation

The choice between different layouts is mostly subjective branding. The mirroring and archiving infrastructure should work well with most choices.

Plain Artifacts

All artifacts, checksums and signatures placed directly into the podling distribution directory. No subdirectories are created. No release documentation is placed within the distribution directory.

Miscellaneous Documents

All files contained in www.apache.org/dist will be mirrored. So, in addition in archives other documents placed in the distribution directories will be available on the mirrors for casual browsing.

As is traditional in some communities, some like to add notices (such as RELEASE_NOTES, README and CHANGES) for each release. Note that from publicity perspective it may be effective to include this information on the download page as well as in the release itself.

The HTTPD runs with fancy indexing enabled. So, creating HEADER.html and FOOTER.html documents allows the text of the index page to be customised.

A Structured Layout

Many projects use structured layouts. For example:

The common theme is that each type of artifact is grouped into a subdirectory. For example, binary packages into binaries and source packages into source (say).

Often symbolic links are created from the root of the project distribution directory to the latest version of each release. This allows scripts or users to easily locate the latest release.

Release Documentation

Many project distribute notices for a release (such as RELEASE_NOTES, README, CHANGES and so on) in addition to the main compressed archives. This allows users to read them and decide whether they want to download the archive artifacts.

There are also some compelling arguments for making the documentation for a release available for browsing online. (Issuing documentation as compressed archives is just another type of distribution and the standard rules for artifacts apply.) This makes support of releases easier and helps users understand which release is most appropriate for them.

Some projects do exactly this. For example:

Online documentation should be archived so that it can be easily recreated. Though using www.apache.org/dist for these files is convenient and ensures that they are archived, it should not be used for this purpose. Documentation typically consists of many relatively small files and so is inefficient to sync. Users will view it using http and so there is no advantage in these files being distributed to the mirrors. The cost of syncing to the mirrors should therefore be avoided.

The best approach is to commit the documentation for a release into subversion. Then check out into a directory in the website. Generally, checking generated documentation into source control is often considered bad practise. But it is very suitable for fixing the documentation for a particular release.

It would be possible to use the subversion URL directly. However, checking out into the website space is better from the perspective of server load and website mirroring. So that is recommended.

Archiving Old Releases

Old releases should be archived (by deletion) promptly. For podlings, typically all old releases should be archived when a new one becomes available.

.htaccess files are sometimes used to redirect live urls to archived releases. Direct links to distributions on www.apache.org bypass the mirroring. They should therefore be discouraged. It is therefore best to save the effort and not offer redirects for archived (mirrored) releases.

If a redirect is used then for performance reasons, then a .htaccess in the root incubator directory should be used.

Mirroring Scripts

There are two options:

The generic script can be used immediately and requires the minimum of setup. Creating a custom script allows control over the look, feel and content of the download page but requires more initial effort.

A consequence of mirroring downloads is that the number of downloads cannot be monitored directly by using hits on the Apache site. When using a custom script, it may be possible to use a service like Google analytics to gain a reasonable estimate.

It is important that users verify releases. Apache has no control over the integrity of releases on mirrors.

It is recommended that the download page is used to remind users that they need to verify releases and to give instructions on how they can do this. Links provided to checksums and signatures need to refer to the originals on www.apache.org and not the copies on the mirrors. More information can be found in the signing guide.

Signatures

Understand the guide. See signing best practice.

Each podling should maintain its own KEYS file directly in the podling distribution directory. It is recommended that all committers add their keys to that file.

Dealing With A Defect

Once an artifact has been uploaded, it must never be modified. No matter how serious a defect is discovered, the right action is to roll a new one with a new number.

Very serious defects may necessitate the withdrawl of a release. This should be done by:

  1. Archiving the release (so that it is no accessible from www.apache.org/dist). It may be some hours before an artifact added is archived. Check that archive.apache.org/dist contains the archived release before deleting the original. If necessary, use a temporary .htaccess to prevent access during this period.
  2. Posting a suitable announcement (to the same lists that received the original)
  3. Adding a suitable notice to the download page

Rules

Every incubator release is also an Apache release. So Apache policy must be followed. Incubator policy is additional and builds on these rules.

Release managers for podlings need to read the release and legal documentation on the main Apache site.

A few important topics are discussed below. These are a supplement and not a substitute for the Apache documentation.

Signing

The release manager need to create a signature for every artifact released. It only takes a little time to create a key and the process is reasonably straight forward.

However, it is recommended that enough time is taken to gain a basic understanding of the public key cryptography. Read the Apache Signing Guide and at least some of the background material linked from it.

See:

Naming

Apache releases should contain the full name of the project responsible for the release. This ensures that trademark law can be used against others issuing artifacts with the same name.

For example, one good name for product bar Apache Foo Project would be apache-foo-bar.

Once a podling graduations, it should adopt this naming convention. Whilst in the incubator, practice is a little different. The release name should contain the podling name and may contain apache. Incubator policy insists that it must also contain incubating (though small variations for the sake of readability are usually acceptable).

For example, for podling foo, both apache-foo-incubating and foo-incubating would be acceptable names.

See also:

Incubator policy requires that all incoming code is fully signed off before any release. This simply reinforces the Apache requirements: all code must have appropriate licenses. Potential liability for released code is greater than for unreleased code. So, it is of particular importance that this is true of all released code.

The process of preparing a release should include an audit of the code to ensure that all files have appropriate headers and that all dependencies complies with Apache policy. The release build also needs to be check to ensure that the LICENSE and NOTICE files are included in every released artifact.

See:

Release Guidelines

In addition to policy, the infrastructure, public relations and legal teams also document "guidelines" for projects: documented recommendations which have not yet been blessed as policy. There are usually good reasons why project should follow the guidelines given even if these may be initially obvious.

Guidelines change much more frequently than policy. Release managers should follow the appropriate lists. Subscribe to:

  • legal-discuss for matters related to licensing
  • infrastructure-issues for matters related to

Best Practice

Preparation

Preparation is the key to quality. The release manager will need to organise and coordinate this but need not execute all.

Bugs

The list of open issues needs to be analyzed. The community needs to decide which issues could be resolved for this release and how much energy the community has to deal with these. Time may also become a factor: some issues which cannot be resolved promptly may need to be postponed.

Issue tracking system can be used to help this process especially for complex projects with many open issues. Each open issues should be analyzed and marked appropriately. An accurate view of those bugs which are targeted for the upcoming release helps to concentrate community effort. Consider asking reporters to donate unit tests.

Preparing Documentation

Any documentation that the release contains should be proof-read and spell checked. This is obviously something that needs to happen after the content is finalized. So typically, the release manager needs to coordinate the documentation effort. A release is a good time to concentrate energy on documentation.

Artifact Naming

TODO: should include apache in the title (gives trademark protection against different jars with the same name)

Package Types

TODO: glossary - distribution type: based on the same tagged source built TODO: Common Types of distribution

The source package is canonical. Every release must revolve around a source package. Compiled languages may also wish to create binary packages. These may be platform specific. Some projects may issue builds (ie binary packages) which package the project for particular containers.

All types of packages ship as compressed artifacts. This means multiple packages may be shipped as various compressed artifacts (eg tar.gz and .zip).

Downstream Packagers

TODO: glossary - downstream packager: takes an apache release and packages it for a particular platform. TODO: best practice is to work with downstream packagers and link to their packages rather than roll their own packages. Need to add notes that these are not official releases. TODO: link to notes on working with downstream packagers

TODO: write up gentoo wiki on good upstream. google for other distros.

Source Package

TODO: describe what a source package is; version control for source packages; add content to release documents; export not checkout

Many would argue that for open source projects, the source package is the release: binaries are just for convenience. There are some pragmatic (as well as philosophical) reasons why a source package should be issued:

  • Downstream packages usually prefer to work from an Apache source package. Typically, they will patch the source both to apply fixes and to add elements of their own build system. Being a good upstream encourages wider distribution and use of the project.
  • Source packages encourage developers to modify the code base. Recruiting new developers is vital for the long term health of an Apache project.

It is required that source packages are included in every release. Many packagers (for example, FreeBSD and linux distributions) prefer or demand to work from source releases. Archivists prefer source. Many large organizations prefer to verify and then build their own versions from source. A source package is easy to create but helps to reach these audiences.

Binary Package

A binary package is any package that is not an exported snapshot of the source. Binary packages may include some source code. For some projects, this makes sense. For others, it does not.

Release Documentation

Users expect release documentation to be present in the root directory of the package. If copies of these documents are required elsewhere, it is recommended that the release process creates copies. These documents should be present in all types of packages including source packages. So the master documents should be checked into the base subversion directory.

Collecting all this information may seem a little daunting especially for a starting project. Not at all agile. One approach may be to ask developers to update documentation as they make changes. Another is to use a build tool to collect this information automatically. A third is for the release manager to collect all the information required when the release is made. The disadvantage of this method is that increases the work required to create a release.

Typically, as a project matures, the user base rises and the rate of core development (as opposed to work on modules) slows. The need to inform users of the changes made increases as does the need for quantity release documentation. One approach suitable for new projects is to aim to increase the quality of information supplied with each release as adoption grows.

RELEASE_NOTES (TODO:link). Remember to include a description of the project. CHANGES (link) these may be contained in a separate document or included in the RELEASE NOTES. svn log can be used to collect changes. This is a good opportunity to reinforce the need for good commit messages. CHANGES should also include references to bugs fixed (TODO URLS, searches in JIRA or bugzilla). Include a description of the build process either in a README or BUILDING document. This should include details of the dependencies (both library and to) required to build and run the product but may do so by reference CHANGES should note any incompatibilities introduced since the last release.

Remember that the RELEASE_NOTES will be used as a basis for downstream packagers (TODO add links to this in packagers section) as well as users. Create a short paragraph explain what the product is. The proposal short already include something suitable. Include this description in the RELEASE_NOTES and in ANNOUNCEMENTSs (TODO add links to this in announcements section)

LICENSE and NOTICE

Apache projects may distribute artifacts and documents as part of a release which are not Apache Licensed. All such artifacts must comply with Apache's 3rd party licensing policy.

All the licenses on all the files to be included within a package should be included in the LICENSE document. This LICENSE (courtesy of Apache HTTPD) is a good example. The Apache License is at the top of the LICENSE document. After that, the license for each non-Apache licensed component is included, along with a clear explanation of which files that license applies to.

The NOTICE document is for additional copyright and attribution statements those licenses may require. A typical NOTICE document at a minimum includes a copyright and attribution statement for The Apache Software Foundation. Nothing else belongs in the NOTICE document. See this document for the typical example.

Legal Audit

It is of particular importance that released code is clean. It is good practice to check the provenance of any source documents which do not have license headers. Check that dependencies (and in particular those dependencies that ship in the packages) comply with Apache policy. Legal policy and interpretation changes from time to time so it is worth investing a little time reading again the legal release material.

Compression Formats

Packages of all types should be shipped in a compress format to minimize bandwidth requirements.

Though utilities for all major compression formats are available for all major platforms, not all platforms support all major compression formats by default. Users find it convenient to download the package compressed in a familiar format that is easy to decompress on their platform. It is therefore recommended that each type of package is shipped in a variety of compressed formats. Ship at least one of tar.gz, bz or bz2 for UNIX and linux (but note this). Ship zip for windows.

Some formats are strongly associated with particular platforms. Even when the uncompressed contents have no functional differences, this leads to conventional associations between particular compressed artifacts and platforms. For example, zip with windows and tar.gz with UNIX and linux. Users often expect this association. See notes on line endings for source packages.

Different package types should unpack to directories with different names. This is more convenient for users since:

  • users who download more than one type do not need to take action to ensure that unpacked packages do not overwrite each other
  • it allows easy identification of different package types

For project, Apache Foo (say) with source and binary types, it is conventional for the main binary to unpack to apache-foo and the source to apache-foo-src. Other binary types should unpack to suitably suffixed directories (for example, apache-foo-sdk).

Compressed archives should unpack into a contained directory (rather than straight into the current directory). The directory into which the package unpacks should flow the standard naming convention. apache-foo.tar.gz should unpack to the apache-foo directory.

Source Package Build

This section applies only to compiled languages.

Source packages should contain easy instructions describing how to build the project. The source package should build from instructions contained. TODO: best practices for instructing users about building the project.

Dependencies

TODO: information about dependencies is a FAQ. releases should indicate dependencies and which are optional and which required. if they are not shipped with the package, information should be included about their official home. minimum (and max) supported versions.

dependencies should comply with the current apache policy. TODO: link

TODO: dependencies also include the tools required to build and test the source. to dependencies are often included in BUILDING or README

TODO: particularly important for languages. language should be approached as dependencies and documented. these should be listed in the README or RELEASE NOTEs.

Any changes in dependencies (including dependencies added or removed or changes in supported version) should be noted in the change log for this release. (Where appropriate) check the that the application is built against the correct versions. Note that this includes platform as well as library dependencies.

Distributing Libraries

TODO: ASF policy compliance TODO: project policy - explicit policy should be written down

NOTICE files

Read this. The NOTICE is important and serves several purposes. The contents should be included in all downstream redistributions.

The NOTICE documents copyright notices and other required attributions. This must include:

Every contributor retains the copyright to their contributions and grants Apache only a (generous) license for the work. A release is an act of selection by the PMC. Apache holds the copyright to the collective work that is the release. The Apache copyright in the NOTICE pertains to the collective.

It is strongly recommended that the NOTICE contains only these legally important contents. The incubator disclaimer is best placed in another document.

The exact name may be optionally suffixed. For example, both NOTICE and NOTICE.txt are fine.

Subversion Best Practices

TODO: svn is flexible. branches and tags with svn are not the same as with cvs.

All releases should be identified with a tag. It is occasionally necessary to rebuild releases many years later, and having a tag easily allows this to be done. Tagging is cheap and easy when using subversion. So, every release should be tagged. Releases should be built from a tag, or built from a stable branch (not trunk). If not built from a tag, the approved release candidate should be tagged.

It is useful to adopt a consistent naming convention for tagging releases. This allows release tags to be recognized easily. Typically the tag will be a variation on the name of the release. For example, some projects use ALL CAPS to indicate release tags in which can apache-foo-1.2 would be tagged APACHE_FOO_1_2.

If svn:externals is used, check carefully that a tag is referenced. This ensure that all the source for the release is fixed. If the target of a svn:externals changes then the release will no longer be complete reproducible.

Reproducibility

It is important that any release can be reproduced from the source at any time in the future. Apache releases have long active lives and are permanently archived. It may be necessary (for example, for legal reasons) to provide a new release that is a slight alteration of a previous release. Release managers owe it to those who come afterwards to use build processes that are reproducible.

The build script should fully capture all the processes necessary to create the release. Manual processing (other than creating compressed archives) is a sign that the build is not mature enough for a full Apache release.

The requirements of the build should be fully documented. The versions of tools and platforms used to build the release should be noted.

Release As Advertising

Releases are a primary form of communication with open source users and potential developers. Its useful to think about releases in this way. TODO: what a release says about a projects TODO: link into media relations and announcements (grassroots and mainstream, articles on new features, freshmeat, downstream packagers)

Dependencies

As well as libraries, projects often have more subtle dependencies. Most languages have different versions, and it is important that the versions of a language upon which a project will build and run are clearly documented. The release notes are a typical location for this information.

It is important to review all library dependencies as part of the release process for compliance. Apache policy changes from time-to-time. A list should be compiled of the project's dependencies, including those shipped as binary libraries and those shipped as source together with the licenses for those dependencies. These lists should be checked against the latest policy documents.

This list should also be used to check for compliance with US export regulations. If any dependencies are cryptographic libraries then it may be necessary to fill in some paperwork.

Announcements

TODO: release managers should ensure that releases are announced. TODO: links to release management FAQs TODO: consider grassroots sites eg freshmeat.net TODO: link note on press release

Please ensure that release announcements include a brief description of the product near the start of the text. Although readers on the developer and user lists will presumably know what the product does, it's unlikely that other readers will have much idea. Don't force recipients to read the product website merely to find out whether the software is of interest to them.

Announcements should be signed by the release manager with the key used to sign the release. Note that this may mean creating a plain text signature on the machine used to sign the release and then transferring this.

Announcements should be posted from the release manager's apache.org address.

Incubator Release Vote

All releases by podlings must be approved by the TODO: link Incubator PMC. The conventional process is for the podling to follow the usual Apache process (including TODO: link release vote) and then call for a Incubator PMC VOTE on the TODO: link general incubator list.

The release manager should post the call for the VOTE.

  • Links to the release artifacts
  • Links to the PPMC release vote thread
  • Link to the tag from which the release is cut

Release managers should subscribe to a number of Apache-wide mailing lists. Decisions are taken on these mailing lists and issues resolved. Documentation is updated later (if at all). It's important that release managers ensure that they keep up to date.

  • TODO: link General infrastructure list
  • TODO: link Legal discuss list

Release managers for incubating podlings should also subscribe to the TODO: link general list.

Release Candidates

A release candidate is a set of artifacts upon which a vote is held for a release. The actual nature of the release candidate depends on the release system adopted by a the project.

Those projects adopting a system of blessing a candidate will start by creating a candidate which will then be promoted by a series of votes until it either fails or reaches full release status. In this case, the same candidate may be known as a release candidate, an alpha, a beta and then a full release. Other projects may release alphas and/or betas or developer releases until the project has agreed there is sufficient quality in place to make the code available as a General Availibility (GA) release.

Those projects which have release candidates will vote on a sample release candidate.

Only formally-approved releases may be distributed from the main directories. There are a number of reasons for this:

  • the URL indicates the status of the release
  • the main directory is archived for historical purposes. To conserve space only official release should be archived.
  • syncing too many short lived artifacts use bandwidth

TODO: links to infra dev

It is traditional that release managers use their Apache home space to make available release candidates. TODO:link to dev instructions on using Apache web space. Please remember to delete release candidates after voting concludes.

Signatures

TODO: links to release signing documentation

Ensure that the code signing public key is uploaded to a network in good time. It may take some days for keys to propagate to all servers in the network.

Novice signers should read the documentation and practice. Consider using an isolated installation to store the code signing key and to sign releases.

Notes

License Issues

TODO: content It is important that the right legal contracts are in place for all source code at Apache and that the right process has been followed. For the majority of the time, this is easy: committers create original works which are licensed by Apache under a CLA or CCLA TODO: complete content

The Incubator Disclaimer

TODO: the incubator requires that users are informed that the by including a standard disclaimer. may be include in README, RELEASE_NOTES DISCLAIMER. It is recommended that it is not included in NOTICES

Release Candidates

There are two distinct approaches: release candidate and TODO.

Versioning

There are several good versioning strategies used by projects at Apache. The version strategy adopted is less important than adopting one that is clear and consistent and documenting it. If there are strong opinions amongst developers on the right way to version then document this strategy. If there are not then time can be saved by adopting an existing strategy with a good fit. It is usually better to start by copying the documentation rather than using by reference. Strategies sometimes change as do URLs. Project often (in time) find that they need to expand or fine tune their versioning strategy and this is hard if another project's documentation is used. A project should therefore use their own strategy.

TODO: notes on strategies.

The most common system of versioning is based on major.minor.point where major, minor and point are all integers. Note that these are not decimals and there is no necessity to raise a major version number after the minor version has reached 9. The exact rules vary and it is recommended that a project agrees and documents its own rules.

In addition to an official release with a particular version number, any number of unofficial releases of various types may share the same number. These are usually terms alphas, beta, release candidates, TODO: link to foundation release documents. These release may be designated by adding an appropriate suffix to the release number (for example, 3.11.15RC1). The exact form is not as important as documenting the system adopted by the project and its consistent use. Users should be able to predict the status of the release from the artifact and the documentation.

TODO: (move to a note) Notes on 0.x releases verses alpha and betas. It is better to use 0.x releases than to create numerous alphas and betas for 1.0. Conventionally, 0.x releases are aimed at early adopters only without a strong promise of easy upgrade or backwards compatibility. 0.x is a good designation for a product that is not feature complete but whose code is solid for those features which are implemented.

Version Numbering Between Releases

It is useful to use a system of versioning for development (non-release) version numbers that allows artifacts built from a development code stream to be distinguished by their version from releases. Choose any reasonable and appropriate system but documented it so that users understand how to recognize the difference.

There are two major approaches to this problem. Some use a suffix to indicate a development stream. Others use a property of the version number (conventionally odd or even).

TODO: odd, even version numbers - research httpd, linux

SNAPSHOT and dev are commonly used suffixes. Typically these are set to the next minor version number in sequence. For example, after cutting apache-foo-1.5.6, the version would be 1.6.

Unique Artifact Names

Every artifact distributed should have a name that is unique. This allows each artifact to the recognized by its name and ensures that different artifacts do not overwrite each other. Including apache in the name of the artifact not only gives brand recognition but means influence can be exerted over errant downstream repackages by trademark law. If this practice is adopted then the release manager merely has to ensure that each artifact is uniquely named within the set of Apache artifacts. By including the name of the project, each artifact only has to be unique within the artifacts release by the project. When a project distributes several distinct products then the product name should also be included.

For each product, each type of package should be assigned a consistent and unique name. Conventionally, source packages use src and main binary packages no prefix. Typically each type of package is made available in a number of compression formats. These are conventionally distinguished by the usual file type suffix.

So, the traditional format is apache-project-product-type-version.format.

Release Notes

TODO: rewrite this is section distinguishing between the content and the presentation. content may (and probably should) be replicated in many different places. The content presented in the RELEASE_NOTES needs to be plain text etc

Release notes are a vital to for communication between an open source project and its users. They are often the first documentation a user will read. First impressions matter. Their content will often serve as the basis for TODO: link announcements and other TODO: link grassroots publicity.

The content may be replicated in many places. Use the content as news on the project website and as the basis for the download page. Use the content for posting to TODO: link grassroots media. Use the content for postings to announcement lists.

The release notes should be in a common and easily read format (plain text is best). The release notes should easily located and so should be positioned in the base directory.

The content can often be edit for use in the announcement.

TODO: move to best practice?

TODO: contents include a short description of the project and links to the apache site

Whenever possible, each type of package should ship with release notes. The release notes document should therefore be committed to the source repository so that it ships with the source package.

The first paragraph of the release notes should introduce the project and the release. It is often useful to characterize the release (TODO examples). Spend time thinking about the best organization. Important information should be prominent.

The exact contents of the release notes varies and it is not unusual for this content to be distributed amongst several documents. This is fine: these recommendations should be viewed as pertaining to the mass of notes that accompany the release and do not need to be included in any particular document.

Any changes which may effect a user who wishes to replace the last version with this should be highlighted. In particular, document:

  • Deprecations
  • Binary incompatibilities
  • Semantic incompatibilities (TODO: glossary)

It is recommended that (where possible) automated tests are used to verify compatibility. Note that the discovery of incompatibilities at this stage may require a change in version number or revision of function.

Release notes should be readable with minimal requirement. So, they should be plain text with explicit line breaks at a reasonable number of characters (80, say). Most projects now produce documentation that is formatted as html. The development and supply of release notes in this format is fine but release managers should create and supply a plain text version by stripping out the formatting.

Remember that a user may have received the software indirectly. So, include some brief details about the project and an url.

Release notes are important TODO: glossary guerrilla advertising for a project. They should provide:

  • an introduction for those who don't know the project in detail
  • briefly state the aims of the project
  • briefly indicate the project's roadmap and how this release fits into it
  • list ways to get involved with the project
  • describe how to raise issues

This may be included by reference to the main documentation but (for the benefit of those that don't read document) should be mentioned.

Change Logs

TODO: should this be in best practices TODO: add examples

A change log indicates the changes made to the project since the last release. Some projects include a change log in the release notes documents. Others use a separate document often called ChangeLog.

Change logs can be created in various ways. Some project ask committers to fill in details each time they commit. Other rely on analyzing the commit messages when the release is prepared. (To do this, go through the logs since the revision of the last release. TODO example)

The change log typically includes every feature added and every bug resolved since the last release.

Signing Releases

TODO: links to dev pages batch signing scripts in committers

TODO: consider moving content into foundation docs

OpenPGP compatible signatures must be generated to accompany releases. Public key encryption is a deep and wide subject. However, the knowledge required to safely create signatures for Apache releases can be learnt in a short time. TODO: link to foundation docs

What can be sometimes confusing is that though a signature is required, connection to the web of trust is only strongly encouraged not required.

A signature from a key which is well connected to the Apache web of trust gives greater security than a signature that is not.

However, the essential use of signatures at Apache is to allow release managers themselves to check whether an artifact is identical to the release they created. All that is required in this case is that the release managers knows whether a signature is their own.

On Multiple Products

A project may release a number of distinct products created by the same community. TODO: differences between products and packages.

On Template Sources

TODO: this is probably a best practice

Source files should contain the license header whenever this is reasonable. Templates are source documents and so this principle applies to them as well.

If these templates are used to generate documents which form part of the package then the documents generated should contain the license header.

However, if this template is used by a user to generate output, usually this output should be free of license restrictions. Most templating languages allow comments which are not included in the output. If this is allowed then the license header should be included in the template as a comment. If not then consider adding a NOTE or a README to the directory rather than a license header.

On Export Regulations

TODO preamble TODO link to legal

On Line Endings

It is convenient for users and more consistent if line endings for appropriate files (text, xml, html and so on) reflect the platform usually associated with the compression format (CRLF for the zip and LF for the tar.gz).

Source packages can use svn export --native-es.

On Press Releases

It is rare for Apache projects to issue mainstream press releases and it is very rare for releases to justify this. If there is likely to be mainstream press interest in a release then please talk to the public relations committee.

TODO: link to public relations committee

On License Headers

TODO: links into legal documentation. discuss issues about which files need to apply.

All source capable of copyright should contain license header. Easiest way to comply is to ensure that every human readable file has the header. Note that source includes not just the source code compiled into the final product but also all other resources such as style sheets, test code and resources, build files and documentation source. When in doubt, add a header.

TODO: to

The issue of licenses on generated documentation is a little controversial. Copyright may not subsist in a document which is generated by an transformation from an original. In which case, the license header may be unnecessary. License headers should always be present in the original. Where it is reasonable to do so, the templates should also add the license header to the generated documents.

TODO: check with legal-discuss then move this content into the legal pages and link with commentary

TODO: link into provenance

Code Provenance

License headers may seem trivial and to some extent that is true. The important issue is code provenance. All code contributions must be tracked and the provenance of the code traceable.

Commit comments are an important to. When source which is not originally created for the project is committed, the message should contain details about the provenance of that source. When a patch is applied from an issue tracking system, the issue should be noted in the commit message.

Releases are important. Released code is distributed widely. It is important that the provenance for all released code is known and is appropriate. License headers are a way of documenting provenance. Any source which is created for Apache should have the standard boilerplate text. Other source should have the headers (copyright and license) retained. Before a release, source which has not been originally created for (or donated to) Apache should be checked against the current Apache policy.

Any source which does not have a license header must be considered suspect and its provenance checked. There are several classes of source which do not require license headers. It is useful to adopt a policy of documentation in this case perhaps by including a short header if the file is generated (say) or by creating a README in the directory containing license information. This makes code auditing much easier.

Implementations Of Standards

TODO: importance of accurately reporting to the user the state of an implementation TODO: importance of complying with the reporting requirements set by the standard creator

Understanding Content For NOTICE and LICENSE

The Apache License, Version 2 clause 4d states:

If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the flowing places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

Many other similar licenses contain such a clause or something similar to it. So, if the release redistributes any source or artifacts covered by licenses with these clauses, the contents of each NOTICE must be present in the NOTICE distributed with the release.

The contents of the NOTICE are information. The LICENSE document should contain the actual licenses under which each part of the release is distributed but not any notices which the licenses require.

For example (taken from JDBM) this license. The form of this license is similar to the Apache License, Version 1.0. Clause 5 states:

5. Due credit should be given to the JDBM Project (http://jdbm.sourceforge.net/).

The license itself should be appended to the LICENSE document with a header indicating . A suitable note giving credit to the JDBM project (for example This product includes software developed by The JDBM Project (http://jdbm.sourceforge.net/).) should be appended to the NOTICE document.

The contents of the NOTICE document should be included by all downstream distributors and packagers. So, this is the right place for all required credits and attribution notices required by any of the contents of the package (whether legally or ethically). It is better to include any other types of explanations and notes in the RELEASE-NOTES or in a README.

VOTEs

TODO: add bill's excellent quote (apologies for the pun)

Apache releases are high ceremony. A number of formal VOTEs binding upon Apache are required before an release of any type.

When proposing a formal VOTE, it is useful to adopt a conventional form. VOTE threads are conventionally prefixed by a [VOTE] subject. This helps everyone recognize that this is a formal decision and to prioritize their reply. Many people filter their emails into separate VOTE directories by using this prefix.

It is useful to separate a preamble giving context and background (which should be cut from replies) from the possible VOTEs. This makes it easy and quick for people to post their VOTEs. When adopting this format, it's usual to include the proposer's vote after the preamble. For example.

Stating the process for the VOTE (for example, minimum duration) in the preamble allows anyone who objects to the process to register this immediately.

It is conventional to end a VOTE thread with a RESULT post tallying the votes cast. To preserve the thread, this should be done by replying to the original VOTE post and adding a [RESULT] prefix. The subject may be retained as is or edited to indicate the result. For example.

Be careful to check the voters against the appropriate list. Binding votes are cast by people on the appropriate committee. The list of Incubator PMC members is available.

The result post should list each voter and the vote cast. Non binding votes may be listed. If they are then each binding vote should be indicated. If they are not then the post should state that only binding votes have been included. Each voter can (and should) verify that their vote has been correctly tallied. TODO link to problem thread.

When posting replies to a VOTE thread take particular care to change the subject. When changing the subject it is conventional to reply to the relevant post and then edit the subject adding the new subject flowed by [WAS and the d subject. For example, d subject Re: Whatever may be edited to New Foo [WAS Re:Whatever]. This ensures that the thread linkage between the topics is retained but also makes it clear that the post now relates to a new subject.

TODO: move VOTE netiquette into either ppmc or lists and link from here TODO: this has turned into best practice.

Each release requires a positive vote binding on Apache. This means a . There is usually no need for this vote to be private. (A rare exception might be a -1 due to a nearly discovered security issue, for example.)

In the case of podlings, a 3 binding +1s are required by members of the Incubator PMC. To increase the chances of a prompt approval, mentors need to cast their votes on the general list.

On Managing VOTE Threads

TODO: consider whether this would be better moved elsewhere

VOTEs on low volume mailing lists with small numbers of interested parties usually require no management. VOTE threads on high volume mailing lists or mailing lists with large numbers of active contributors require netiquette and active management to avoid confusion. It is not uncommon for those unfamiliar with these environments to find difficulties.

Long VOTE threads are hard to tally. Each opinion needs to be noted and acknowledged. This allows people to check their votes and easily correct mistakes in the count.

It is vital in a long VOTE to ensure that debate is restricted on the VOTE thread. If it looks likely that there is a risk that a VOTE will turn into a debate, it is wise to allow people to discuss the issues first. It is important that off topic or divergent posts are moved to separate threads with other subjects.

It is usually cleaner to abandon a VOTE thread that goes astray or when the proposal needs to be altered - for example, then issues are found in a release candidate. Start a new VOTE thread with the revised proposal (in the case of the example, a new release candidate).

GNU Tar Known Incompatibilities

Typically, applications used to create tar.gz files are based on GNU tar. Unfortunately, the tar which ships with some versions of Saris and some (der) versions of MacOSX is not always compatible with the output of GNU tar. TODO: check information and expand

On Source-Only Releases

A source-only release contains only the source package. Though users often prefer binary packages, source-only releases can be useful early in the life of a project. They require much less ceremony and encourage developers to get involved by finding and fixing bugs. Occasionally, projects whose primary user base typically obtains the software via a downstream repackage may prefer to release source only.

On Binary-Only Releases

Releases containing only binary packages are not performed at The Apache Software Foundation. The Apache Software Foundation distributes open source software to the public. Open source development is characterized by the accessibility of the source. Binary-only packages discourage developers from interacting with the source. Every successful Apache project needs to recruit new developers to carry the project forward.