RFC: Split Release Process
Status: Implemented in smithy-rs#986 and aws-sdk-rust#351
At the time of writing, the aws-sdk-rust
repository is used exclusively
for the entire release process of both the Rust runtime crates from smithy-rs
as
well as the AWS runtime crates and the AWS SDK. This worked well when smithy-rs
was
only used for the AWS SDK, but now that it's also used for server codegen, there
are issues around publishing the server-specific runtime crates since they don't
belong to the SDK.
This RFC proposes a new split-release process so that the entire smithy-rs
runtime
can be published separately before the AWS SDK is published.
Terminology
- Smithy Runtime Crate: A crate that gets published to crates.io and supports
the code generated by
smithy-rs
. These crates don't provide any SDK-only functionality. These crates can support client and/or server code, and clients or servers may use only a subset of them. - AWS Runtime Crate: A crate of SDK-specific code that supports the code generated
by the
aws/codegen
module insmithy-rs
. These also get published to crates.io. - Publish-ready Bundle: A build artifact that is ready to publish to crates.io without
additional steps (such as running the publisher tool's
fix-manifests
subcommand). Publishing one group of crates before another is not considered an additional step for this definition. - Releaser: A developer, automated process, or combination of the two that performs the actual release.
Requirements
At a high level, the requirements are: publish from both smithy-rs
and aws-sdk-rust
while preserving our current level of confidence in the quality of the release. This
can be enumerated as:
- All Smithy runtime crates must be published together from
smithy-rs
- AWS runtime crates and the SDK must be published together from
aws-sdk-rust
- CI on
smithy-rs
must give confidence that the Smithy runtime crates, AWS runtime crates, and SDK are all at the right quality bar for publish. - CI on the
aws-sdk-rust
repository must give confidence that the AWS SDK and its runtime crates are at the right quality bar for publish. To do this successfully, it must run against the exact versions of the Smithy runtime crates the code was generated against both before AND after they have been published to crates.io.
Background: How Publishing Worked Before
The publish process to crates.io relied on copying all the Smithy runtime crates
into the final aws-sdk-rust
repository. Overall, the process looked as follows:
smithy-rs
generates a completeaws-sdk-rust
source bundle at CI time- The releaser copies the generated bundle over to
aws-sdk-rust
- The releaser runs the
publisher fix-manifests
subcommand to correct theCargo.toml
files generated bysmithy-rs
- The
aws-sdk-rust
CI performs one last pass on the code to verify it's sound - The releaser runs the
publisher publish
subcommand to push all the crates up to crates.io
Proposed Solution
CI in smithy-rs
will be revised to generate two separate build artifacts where it generates
just an SDK artifact previously. Now, it will have two build targets that get executed from CI
to generate these artifacts:
rust-runtime:assemble
- Generates a publish-ready bundle of Smithy runtime crates.aws:sdk:assemble
- Generates a publish-ready bundle of AWS runtime crates, SDK crates, and just the Smithy runtime crates that are used by the SDK.
The aws-sdk-rust
repository will have a new next
branch that has its own set of CI workflows
and branch protection rules. The releaser will take the aws:sdk:assemble
artifact and apply it
directly to this next
branch as would have previously been done against the main
branch.
The main
branch will continue to have the same CI as next
.
When it's time to cut a release, the releaser will do the following:
- Tag
smithy-rs
with the desired version number - Wait for CI to build artifacts for the tagged release
- Pull-request the SDK artifacts over to
aws-sdk-rust/next
(this will be automated in the future) - Pull-request merge
aws-sdk-rust/next
intoaws-sdk-rust/main
- Wait for successful CI in
main
- Tag release for
main
- Publish SDK with publisher tool
The server team can then download the rust-runtime:assemble
build artifact for the tagged release
in smithy-rs
, and publish the aws-smithy-http-server
crate from there.
Avoiding mistakes by disallowing creation of publish-ready bundles outside of CI
It should be difficult to accidentally publish a locally built set of crates. To add friction to this,
the smithy-rs
build process will look for the existence of the GITHUB_ACTIONS=true
environment variable.
If this environment variable is not set, then it will pass a flag to the Rust codegen plugin that tells it to
emit a publish = false
under [package]
in the generated Cargo.toml
.
This could be easily circumvented, but the goal is to reduce the chances of accidentally publishing crates rather than making it impossible.
Alternatives Considered
Publish Smithy runtime crates from smithy-rs
build artifacts
This approach is similar to the proposed solution, except that the SDK would not publish
the Smithy runtime crates. The aws-sdk-rust/main
branch would have a small tweak to its CI
so that the SDK is tested against the Smithy runtime crates that are published to crates.io
This CI process would look as follows:
- Shallow clone
aws-sdk-rust
with the revision being tested - Run a script to remove the
path
argument for the Smithy runtime crate dependencies for every crate inaws-sdk-rust
. For example,
aws-smithy-types = { version = "0.33.0", path = "../aws-smithy-types" }
Would become:
aws-smithy-types = { version = "0.33.0" }
- Run the tests as usual
When it's time to cut a release, the releaser will do the following:
- Tag
smithy-rs
with the desired version number - Wait for CI to build artifacts for the tagged release
- Pull-request the SDK artifacts over to
aws-sdk-rust/next
- Wait for successful CI in
aws-sdk-rust/next
- Download the Smithy runtime crates build artifact and publish it to crates.io
- Pull-request merge
aws-sdk-rust/next
intoaws-sdk-rust/main
- Wait for successful CI in
main
(this time actually running against the crates.io Smithy runtime crates) - Tag release for
main
- Publish SDK with publisher tool
Keep Smithy runtime crates in smithy-rs
This approach is similar to the previous alternative, except that the aws-sdk-rust
repository
won't have a snapshot of the Smithy runtime crates, and an additional step needs to be performed
during CI for the next
branch so that it looks as follows:
- Make a shallow clone of
aws-sdk-rust/next
- Retrieve the
smithy-rs
commit hash that was used to generate the SDK from a file that was generated alongside the rest of the build artifacts fromsmithy-rs
and copied intoaws-sdk-rust
. - Make a shallow clone of
smithy-rs
at the correct commit hash - Use a script to add a
[patch]
section to all the AWS SDK crates to point to the Smithy runtime crates from the local clone ofsmithy-rs
. For example:
# The dependencies section is left alone, but is here for context
[dependencies]
# Some version of aws-smithy-types that isn't on crates.io yet, referred to as `<unreleased>` below
aws-smithy-types = "<unreleased>"
# This patch section gets added by the script
[patch.crates-io]
aws-smithy-types = { version = "<unreleased>", path = "path/to/local/smithy-rs/rust-runtime/aws-smithy-types"}
- Run CI as normal.
Note: smithy-rs
would need to do the same patching in CI as aws-sdk-rust/next
since the generated
SDK would not have path dependencies for the Smithy runtime crates (since they are a publish-ready bundle
intended for landing in aws-sdk-rust
). The script that does this patching could live in smithy-rs
and be
reused by aws-sdk-rust
.
The disadvantage of this approach is that a customer having an issue with the current release wouldn't be able
to get a fix sooner by patching their own project's crate manifest to use the aws-sdk-rust/next
branch before
a release is cut since their project wouldn't be able to find the unreleased Smithy runtime crates.
Changes Checklist
- In
smithy-rs
:-
Move publisher tool from
aws-sdk-rust
intosmithy-rs
-
Modify
aws:sdk:assemble
target to run the publisherfix-manifests
subcommand -
Add
rust-runtime:assemble
target that generates publish-ready Smithy runtime crates - Add CI step to create Smithy runtime bundle artifact
-
Add
GITHUB_ACTIONS=true
env var check for setting thepublish
flag in generated AND runtime manifests - Revise publisher tool to publish from an arbitrary directory
-
Move publisher tool from
- In
aws-sdk-rust
:-
Implement CI for the
aws-sdk-rust/next
branch - Remove the publisher tool
-
Implement CI for the
- Update release process documentation