RFC: Split Release Process

Status: Implemented in smithy-rs#986 and aws-sdk-rust#351

At the time of writing, the aws-sdk-rust repository is used exclusively for the entire release process of both the Rust runtime crates from smithy-rs as well as the AWS runtime crates and the AWS SDK. This worked well when smithy-rs was only used for the AWS SDK, but now that it's also used for server codegen, there are issues around publishing the server-specific runtime crates since they don't belong to the SDK.

This RFC proposes a new split-release process so that the entire smithy-rs runtime can be published separately before the AWS SDK is published.

Terminology

  • Smithy Runtime Crate: A crate that gets published to crates.io and supports the code generated by smithy-rs. These crates don't provide any SDK-only functionality. These crates can support client and/or server code, and clients or servers may use only a subset of them.
  • AWS Runtime Crate: A crate of SDK-specific code that supports the code generated by the aws/codegen module in smithy-rs. These also get published to crates.io.
  • Publish-ready Bundle: A build artifact that is ready to publish to crates.io without additional steps (such as running the publisher tool's fix-manifests subcommand). Publishing one group of crates before another is not considered an additional step for this definition.
  • Releaser: A developer, automated process, or combination of the two that performs the actual release.

Requirements

At a high level, the requirements are: publish from both smithy-rs and aws-sdk-rust while preserving our current level of confidence in the quality of the release. This can be enumerated as:

  1. All Smithy runtime crates must be published together from smithy-rs
  2. AWS runtime crates and the SDK must be published together from aws-sdk-rust
  3. CI on smithy-rs must give confidence that the Smithy runtime crates, AWS runtime crates, and SDK are all at the right quality bar for publish.
  4. CI on the aws-sdk-rust repository must give confidence that the AWS SDK and its runtime crates are at the right quality bar for publish. To do this successfully, it must run against the exact versions of the Smithy runtime crates the code was generated against both before AND after they have been published to crates.io.

Background: How Publishing Worked Before

The publish process to crates.io relied on copying all the Smithy runtime crates into the final aws-sdk-rust repository. Overall, the process looked as follows:

  1. smithy-rs generates a complete aws-sdk-rust source bundle at CI time
  2. The releaser copies the generated bundle over to aws-sdk-rust
  3. The releaser runs the publisher fix-manifests subcommand to correct the Cargo.toml files generated by smithy-rs
  4. The aws-sdk-rust CI performs one last pass on the code to verify it's sound
  5. The releaser runs the publisher publish subcommand to push all the crates up to crates.io

Proposed Solution

CI in smithy-rs will be revised to generate two separate build artifacts where it generates just an SDK artifact previously. Now, it will have two build targets that get executed from CI to generate these artifacts:

  • rust-runtime:assemble - Generates a publish-ready bundle of Smithy runtime crates.
  • aws:sdk:assemble - Generates a publish-ready bundle of AWS runtime crates, SDK crates, and just the Smithy runtime crates that are used by the SDK.

The aws-sdk-rust repository will have a new next branch that has its own set of CI workflows and branch protection rules. The releaser will take the aws:sdk:assemble artifact and apply it directly to this next branch as would have previously been done against the main branch. The main branch will continue to have the same CI as next.

When it's time to cut a release, the releaser will do the following:

  1. Tag smithy-rs with the desired version number
  2. Wait for CI to build artifacts for the tagged release
  3. Pull-request the SDK artifacts over to aws-sdk-rust/next (this will be automated in the future)
  4. Pull-request merge aws-sdk-rust/next into aws-sdk-rust/main
  5. Wait for successful CI in main
  6. Tag release for main
  7. Publish SDK with publisher tool

The server team can then download the rust-runtime:assemble build artifact for the tagged release in smithy-rs, and publish the aws-smithy-http-server crate from there.

Avoiding mistakes by disallowing creation of publish-ready bundles outside of CI

It should be difficult to accidentally publish a locally built set of crates. To add friction to this, the smithy-rs build process will look for the existence of the GITHUB_ACTIONS=true environment variable. If this environment variable is not set, then it will pass a flag to the Rust codegen plugin that tells it to emit a publish = false under [package] in the generated Cargo.toml.

This could be easily circumvented, but the goal is to reduce the chances of accidentally publishing crates rather than making it impossible.

Alternatives Considered

Publish Smithy runtime crates from smithy-rs build artifacts

This approach is similar to the proposed solution, except that the SDK would not publish the Smithy runtime crates. The aws-sdk-rust/main branch would have a small tweak to its CI so that the SDK is tested against the Smithy runtime crates that are published to crates.io This CI process would look as follows:

  1. Shallow clone aws-sdk-rust with the revision being tested
  2. Run a script to remove the path argument for the Smithy runtime crate dependencies for every crate in aws-sdk-rust. For example,
aws-smithy-types = { version = "0.33.0", path = "../aws-smithy-types" }

Would become:

aws-smithy-types = { version = "0.33.0" }
  1. Run the tests as usual

When it's time to cut a release, the releaser will do the following:

  1. Tag smithy-rs with the desired version number
  2. Wait for CI to build artifacts for the tagged release
  3. Pull-request the SDK artifacts over to aws-sdk-rust/next
  4. Wait for successful CI in aws-sdk-rust/next
  5. Download the Smithy runtime crates build artifact and publish it to crates.io
  6. Pull-request merge aws-sdk-rust/next into aws-sdk-rust/main
  7. Wait for successful CI in main (this time actually running against the crates.io Smithy runtime crates)
  8. Tag release for main
  9. Publish SDK with publisher tool

Keep Smithy runtime crates in smithy-rs

This approach is similar to the previous alternative, except that the aws-sdk-rust repository won't have a snapshot of the Smithy runtime crates, and an additional step needs to be performed during CI for the next branch so that it looks as follows:

  1. Make a shallow clone of aws-sdk-rust/next
  2. Retrieve the smithy-rs commit hash that was used to generate the SDK from a file that was generated alongside the rest of the build artifacts from smithy-rs and copied into aws-sdk-rust.
  3. Make a shallow clone of smithy-rs at the correct commit hash
  4. Use a script to add a [patch] section to all the AWS SDK crates to point to the Smithy runtime crates from the local clone of smithy-rs. For example:
# The dependencies section is left alone, but is here for context
[dependencies]
# Some version of aws-smithy-types that isn't on crates.io yet, referred to as `<unreleased>` below
aws-smithy-types = "<unreleased>"

# This patch section gets added by the script
[patch.crates-io]
aws-smithy-types = { version = "<unreleased>", path = "path/to/local/smithy-rs/rust-runtime/aws-smithy-types"}
  1. Run CI as normal.

Note: smithy-rs would need to do the same patching in CI as aws-sdk-rust/next since the generated SDK would not have path dependencies for the Smithy runtime crates (since they are a publish-ready bundle intended for landing in aws-sdk-rust). The script that does this patching could live in smithy-rs and be reused by aws-sdk-rust.

The disadvantage of this approach is that a customer having an issue with the current release wouldn't be able to get a fix sooner by patching their own project's crate manifest to use the aws-sdk-rust/next branch before a release is cut since their project wouldn't be able to find the unreleased Smithy runtime crates.

Changes Checklist

  • In smithy-rs:
    • Move publisher tool from aws-sdk-rust into smithy-rs
    • Modify aws:sdk:assemble target to run the publisher fix-manifests subcommand
    • Add rust-runtime:assemble target that generates publish-ready Smithy runtime crates
    • Add CI step to create Smithy runtime bundle artifact
    • Add GITHUB_ACTIONS=true env var check for setting the publish flag in generated AND runtime manifests
    • Revise publisher tool to publish from an arbitrary directory
  • In aws-sdk-rust:
    • Implement CI for the aws-sdk-rust/next branch
    • Remove the publisher tool
  • Update release process documentation