Generating Common Service Code

This document introduces the project and how code is being generated. It is written for developers who want to start contributing to smithy-rs.

Folder structure

The project is divided in:

  • /codegen-core: contains common code to be used for both client and server code generation
  • /codegen-client: client code generation. Depends on codegen-core
  • /codegen-server: server code generation. Depends on codegen-core
  • /aws: the AWS Rust SDK, it deals with AWS services specifically. The folder structure reflects the project's, with the rust-runtime and the codegen
  • /rust-runtime: the generated client and server crates may depend on crates in this folder. Crates here are not code generated. The only crate that is not published is inlineable, which contains common functions used by other crates, copied into the source crate

Crates in /rust-runtime (informally referred to as "runtime crates") are added to a crate's dependency only when used. For example, if a model uses event streams, the generated crates will depend on aws-smithy-eventstream.

Generating code

smithy-rs's entry points are Smithy code-generation plugins, and is not a command. One entry point is in RustCodegenPlugin::execute and inherits from SmithyBuildPlugin in smithy-build. Code generation is in Kotlin and shared common, non-Rust specific code with the smithy Java repository. They plug into the Smithy gradle plugin, which is a gradle plugin.

The comment at the beginning of execute describes what a Decorator is and uses the following terms:

  • Context: contains the model being generated, projection and settings for the build
  • Decorator: (also referred to as customizations) customizes how code is being generated. AWS services are required to sign with the SigV4 protocol, and a decorator adds Rust code to sign requests and responses. Decorators are applied in reverse order of being added and have a priority order.
  • Writer: creates files and adds content; it supports templating, using # for substitutions
  • Location: the file where a symbol will be written to

The only task of a RustCodegenPlugin is to construct a CodegenVisitor and call its execute() method.

CodegenVisitor::execute() is given a Context and decorators, and calls a CodegenVisitor.

CodegenVisitor, RustCodegenPlugin, and wherever there are different implementations between client and server, such as in generating error types, have corresponding server versions.

Objects used throughout code generation are:

  • Symbol: a node in a graph, an abstraction that represents the qualified name of a type; symbols reference and depend on other symbols, and have some common properties among languages (such as a namespace or a definition file). For Rust, we add properties to include more metadata about a symbol, such as its type
  • RustType: Option<T>, HashMap, ... along with their namespaces of origin such as std::collections
  • RuntimeType: the information to locate a type, plus the crates it depends on
  • ShapeId: an immutable object that identifies a Shape

Useful conversions are:

SymbolProvider.toSymbol(shape)

where SymbolProvider constructs symbols for shapes. Some symbols require to create other symbols and types; event streams and other streaming shapes are an example. Symbol providers are all applied in order; if a shape uses a reserved keyword in Rust, its name is converted to a new name by a symbol provider, and all other providers will work with this new symbol.

Model.expectShape(shapeId)

Each model has a shapeId to shape map; this method returns the shape associated with this shapeId.

Some objects implement a transform method that only change the input model, so that code generation will work on that new model. This is used to, for example, add a trait to a shape.

CodegenVisitor is a ShapeVisitor. For all services in the input model, shapes are converted into Rust; here is how a service is constructed, here a structure and so on.

Code generation flows from writer to files and entities are (mostly) generated only on a need-by-need basis. The complete result is a Rust crate, in which all dependencies are written into their modules and lib.rs is generated (here). execute() ends by running cargo fmt, to avoid having to format correctly Rust in Writers and to be sure the generated code follows the styling rules.