I wrote awsr, a Rust CLI that takes the command line of the AWS CLI v2 and prints the same bytes as the official CLI. The repository is conao3/rust-awscli, under Apache-2.0. awsr sts get-caller-identity finishes in 6.0 ms. On the same PC, the official aws (2.39.3) takes 659 ms, 110 times as long. Help pages come up 380 to 550 times faster. awsr is not an AWS product.

Usage

Type awsr where you typed aws.

$ awsr sts get-caller-identity
$ awsr dynamodb list-tables --output table
$ awsr ec2 describe-instances \
    --query 'Reservations[].Instances[].InstanceId' --output text

awsr reads the same config files (~/.aws/config and ~/.aws/credentials), SSO token cache, and environment variables as aws. The json, text, and table output and the result of --query match aws byte for byte, so your shell scripts that call aws print the same thing with awsr.

awsr carries the botocore models that AWS CLI 2.39.3 ships, and you can call any operation of any service in them as awsr <service> <operation>. I also ported the high-level commands such as s3 cp and s3 sync, configure, and sso login.

Speed

I measured the mean wall time with hyperfine -N on my PC (Intel Core Ultra 5 225U). The commands that call an API ran against a local stub server through --endpoint-url, with static credentials. For awsr I used the nix flake package (statically linked), and for aws the official Linux installer of 2.39.3.

Command	awsr	aws 2.39.3	Speedup
--version	1.3 ms	497 ms	380×
sts get-caller-identity	6.0 ms	659 ms	110×
dynamodb list-tables	5.3 ms	728 ms	140×
ec2 describe-regions	5.0 ms	936 ms	190×
help	1.9 ms	759 ms	400×
ec2 help	1.9 ms	1052 ms	550×
ec2 describe-instances help	3.5 ms	1346 ms	380×

As a floor, a C hello world takes 1.1 ms on the same PC when linked dynamically and 0.7 ms when linked statically. In the commands that call an API, awsr spends the remaining few milliseconds on starting the process, evaluating the endpoint rules, and the HTTP round trip.

aws runs on Python. Each call starts the interpreter, imports many modules, registers plugins, and loads the JSON model of the service. If a script or a CI job calls aws hundreds of times in a loop, it pays 0.5 to 1 second for each call.

How awsr starts fast

aws prepares on each run what awsr prepares at build time: it parses the models for API calls and formats the pages for help.

Models embedded in parsed form

botocore describes the API of each service in a JSON model (service-2.json and others), and aws loads and parses it on each call. The largest model, ec2, weighs 3.5 MB as JSON.

At build time, awsr converts the models into a tree that it walks by offsets plus a table of deduplicated strings, and embeds the result in the binary uncompressed. Each object carries a sorted index of its keys, and awsr looks keys up by binary search. At run time awsr neither parses nor decompresses anything and reads only the shapes the called operation uses, so an ec2 operation takes about as long as an sts one.

awsr generates no Rust code per service. The official AWS SDK for Rust generates typed code for each service, and aws-sdk-ec2 alone takes 150 to 250 seconds to build (awslabs/aws-sdk-rust#113). awsr interprets the models at run time, so the amount of Rust to compile stays the same however many services AWS adds.

I made a few more choices for a fast start:

• The argument parser and the endpoint rule evaluator match patterns without compiling regular expressions.
• awsr builds the tokio runtime when it sends the first request. --version and help do not build it.
• The distributed binaries link glibc statically, so no dynamic loader runs at start.
• awsr allocates memory with glibc's malloc. A build with mimalloc ran --version, the API calls, and help 1.4 to 2.3 times slower.

Help pages formatted with groff at build time

aws help builds reStructuredText from the model and the docs, converts it to a man page with docutils, formats it with groff, and pipes it to the pager. I ported the document builder and the docutils conversion to Rust, but awsr still formats with the local groff, as aws does. Starting groff on each call makes ec2 help take 107 ms.

To skip that cost, the distributed binaries carry all 20,448 pages, formatted with groff at build time. The formatted pages add up to 732 MB (the largest, quicksight update-template, weighs 36 MB). I trained a zstd dictionary on one page in sixteen and compressed the pages with it at level 19, which brings them down to 35.8 MB. With the embedded pages, the binary of the flake package weighs 150 MiB.

groff output depends on the groff version. awsr compares the version of the groff on PATH (the first line of groff -v) with the version that formatted the pages. If they match and no environment variable starting with GROFF is set, awsr decompresses and prints the embedded page. awsr caches the result of groff -v per groff binary, so it starts groff once per machine (ec2 help takes 3.8 ms that time). The GitHub Releases binaries carry pages formatted by the groff 1.23.0 of Ubuntu, and the flake package carries pages formatted by the groff of nixpkgs. With a different version, awsr formats the page with the local groff as aws does, saves the output in ~/.cache/awsr/help, and prints the saved page from the second time on.

Checking the output against aws

Differential testing

Matching aws means matching Python's value formatting. awsr prints floats in the format of Python's repr (1e-05 and 1.0) and booleans in text output as True and False. It counts character widths in table output with unicodedata.east_asian_width from the Python that aws bundles.

To check the ported rules, I wrote awsr-difftest. It runs awsr and aws on the same command line and compares stdout, stderr, the exit code, the file a streaming operation writes, and the requests the stub server receives. I pinned the reference by importing the official installer (2.39.3) with nix, bundled Python included.

The stub server builds each response from the output shape of the operation with a seeded random generator. The generated values include ones that take different formatting paths: 1.50, 1e2, -0, big integers, NaN, timestamps with various offsets, and full-width characters. awsr-difftest skips headers that change on each run, such as the signature.

Running every operation of every service with 4 seeds and the json, text, and table formats gives 233,880 cases. 233,040 of them match aws, and none differ. The other 840 cases call operations that a stub server cannot compare: 552 operations that the CLI does not expose as commands, 144 that go to the real endpoint, 24 that start session-manager-plugin, 12 event streams, and a few more. difftest/exclusions.txt gives the reason for each. awsr-difftest caches the runs of aws, and with the cache a full run takes 2 to 5 minutes.

The help pages

scripts/compare-help.sh compares the help pages with aws. It lists every help command line of aws (20,482 of them), runs each with awsr and aws, and compares stdout, stderr, and the exit code byte for byte. With an awsr that carries the pages formatted at build time, all 20,482 match aws.

Distribution

GitHub Releases carry binaries for Linux (x86_64, x86_64 musl, and aarch64) and macOS (x86_64 and aarch64). You can also install awsr with nix or Cargo.

$ nix profile install github:conao3/rust-awscli
$ cargo install awsr

cargo install formats no help page at build time, so awsr formats each page with the local groff the first time you open it.

The 10 MB limit of crates.io

crates.io accepts .crate files of up to 10 MB. The models that awsr embeds weigh 11.4 MB even after zstd compression, more than one crate can hold. I asked crates.io to raise the limit, and they declined.

I split the models into three data crates that hold only files and their directory.

Crate	Contents	Size
awsr-models	CLI data, endpoints, examples, help inputs, etc.	2.1 MiB
awsr-models-services-a-l	models of services from a to l	5.2 MiB
awsr-models-services-m-z	models of services from m to z	3.7 MiB
awsr	the CLI	3.1 MiB

awsr takes the three data crates as build dependencies, and build.rs reads the files in their directories and embeds them. The data crates share the version of the reference CLI (2.39.3), and I publish new versions of them only when I move to a new release of the AWS CLI. After the split, build.rs produces the same embedded data byte for byte, and awsr runs as fast as before. I timed seven commands against the binary from before the split: they differed by 0.3 ms or less, and swapping the order of the runs swapped the faster side.

Publishing with trusted publishing

A GitHub Actions workflow publishes the crates through crates.io trusted publishing (OIDC). In the Publish workflow, rust-lang/crates-io-auth-action gets a short-lived token, so the repository keeps no crates.io API token in its secrets. The workflow runs cargo publish for the three data crates and then awsr, skipping versions that crates.io already has.

To release, I start the Release workflow by hand. Release bumps the version in Cargo.toml, pushes the commit and the tag, builds binaries for five targets, attaches them to the GitHub release, and then starts Publish. The page of each version on crates.io links to the GitHub Actions run that published it.

Wrapping up

If your scripts or CI jobs call aws many times, replace aws with awsr.