RFC: Publishing the Alpha SDK to Crates.io
Status: Implemented
The AWS SDK for Rust and its supporting Smithy crates need to be published to crates.io so that customers can include them in their projects and also publish crates of their own that depend on them.
This doc proposes a short-term solution for publishing to crates.io. This approach is intended to be executed manually by a developer using scripts and an SOP no more than once per week, and should require less than a dev week to implement.
Terminology
- AWS SDK Crate: A crate that provides a client for calling a given AWS service, such as
aws-sdk-s3
for calling S3. - AWS Runtime Crate: Any runtime crate that the AWS SDK generated code relies on, such as
aws-types
. - Smithy Runtime Crate: Any runtime crate that the smithy-rs generated code relies on, such as
smithy-types
.
Requirements
Versioning
Cargo uses semver for versioning,
with a major.minor.patch-pre
format:
major
: Incompatible API changesminor
: Added functionality in backwards compatible mannerpatch
: Backwards compatible bug fixespre
: Pre-release version tag (omitted for normal releases)
For now, AWS SDK crates (including aws-config
) will maintain a consistent major
and minor
version number
across all services. The latest version of aws-sdk-s3
will always have the same major.minor
version as the
latest aws-sdk-dynamodb
, for example. The patch
version is allowed to be different between service crates,
but it is unlikely that we will make use of patch
versions throughout alpha and dev preview.
Smithy runtime crates will have different version numbers from the AWS SDK crates, but will also maintain
a consistent major.minor
.
The pre
version tag will be alpha
during the Rust SDK alpha, and will be removed once the SDK is in
dev preview.
During alpha, the major
version will always be 0, and the minor
will be bumped for all published
crates for every release. A later RFC may change the process during dev preview.
Yanking
Mistakes will inevitably be made, and a mechanism is needed to yank packages while keeping the latest version of the SDK successfully consumable from crates.io. To keep this simple, the entire published batch of crates will be yanked if any crate in that batch needs to be yanked. For example, if 260 crates were published in a batch, and it turns out there's a problem that requires yanking one of them, then all 260 will be yanked. Attempting to do partial yanking will require a lot of effort and be difficult to get right. Yanking should be a last resort.
Concrete Scenarios
The following changes will be bundled together as a minor
version bump during weekly releases:
- AWS model updates
- New features
- Bug fixes in runtime crates or codegen
In exceptional circumstances, a patch
version will be issued if the fix doesn't require API breaking changes:
- CVE discovered in a runtime crate
- Buggy update to a runtime crate
In the event of a CVE being discovered in an external dependency, if the external dependency is
internal to a crate, then a patch
revision can be issued for that crate to correct it. Otherwise if the CVE
is in a dependency that is part of the public API, a minor
revision will be issued with an expedited release.
For a CVE in generated code, a minor
revision will be issued with an expedited release.
Proposal
The short-term approach builds off our pre-crates.io weekly release process. That process was the following:
- Run script to update AWS models
- Manually update AWS SDK version in
aws/sdk/gradle.properties
in smithy-rs - Tag smithy-rs
- Wait for GitHub actions to generate AWS SDK using newly released smithy-rs
- Check out aws-sdk-rust, delete existing SDK code, unzip generated SDK in place, and update readme
- Tag aws-sdk-rust
To keep things simple:
- The Smithy runtime crates will have the same smithy-rs version
- All AWS crates will have the same AWS SDK version
patch
revisions are exceptional and will be one-off manually published by a developer
All runtime crate version numbers in smithy-rs will be locked at 0.0.0-smithy-rs-head
. This is a fake
version number that gets replaced when generating the SDK.
The SDK generator script in smithy-rs will be updated to:
- Replace Smithy runtime crate versions with the smithy-rs version from
aws/sdk/gradle.properties
- Replace AWS runtime crate versions with AWS SDK version from
aws/sdk/gradle.properties
- Add correct version numbers to all path dependencies in all the final crates that end up in the build artifacts
This will result in all the crates having the correct version and manifests when imported into aws-sdk-rust. From there, a script needs to be written to determine crate dependency order, and publish crates (preferably with throttling and retry) in the correct order. This script needs to be able to recover from an interruption part way through publishing all the crates, and it also needs to output a list of all crate versions published together. This crate list will be commented on the release issue so that yanking the batch can be done if necessary.
The new release process would be:
- Run script to update AWS models
- Manually update both the AWS SDK version and the smithy-rs version in
aws/sdk/gradle.properties
in smithy-rs - Tag smithy-rs
- Wait for automation to sync changes to
aws-sdk-rust/next
- Cut a PR to merge
aws-sdk-rust/next
intoaws-sdk-rust/main
- Tag aws-sdk-rust
- Run publish script
Short-term Changes Checklist
- Prepare runtime crate manifests for publication to crates.io (https://github.com/smithy-lang/smithy-rs/pull/755)
- Update SDK generator to set correct crate versions (https://github.com/smithy-lang/smithy-rs/pull/755)
- Write bulk publish script
- Write bulk yank script
- Write automation to sync smithy-rs to aws-sdk-rust