From 16a985700d80044f139a92d97f00a99f072d5c30 Mon Sep 17 00:00:00 2001 From: James Bornholt Date: Wed, 9 Aug 2023 09:16:23 -0500 Subject: [PATCH] Update README and INSTALL documentation for 1.0.0 release (#439) This adds new documentation for installing Mountpoint, using a Docker container, and revamps the README to match these changes. Signed-off-by: James Bornholt --- .github/ISSUE_TEMPLATE/bug-report.yml | 4 +- README.md | 126 +++++----------- doc/INSTALL.md | 204 ++++++++++++++++++++++++++ docker/Dockerfile | 49 +++---- docker/Dockerfile.source | 36 +++++ docker/README.md | 82 +++++++++++ 6 files changed, 384 insertions(+), 117 deletions(-) create mode 100644 doc/INSTALL.md create mode 100644 docker/Dockerfile.source create mode 100644 docker/README.md diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index 10d6a5da9..1ea7b74d8 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -16,9 +16,9 @@ body: attributes: label: Mountpoint for Amazon S3 version description: | - Which version of the mountpoint are you using? You can find out using `mount-s3 --version`. + Which version of Mountpoint are you using? You can find out using `mount-s3 --version`. If you're building from source or a fork, please state that. - placeholder: mountpoint-s3 x.y.z + placeholder: mount-s3 x.y.z validations: required: true - type: input diff --git a/README.md b/README.md index ca2ac2cfd..45e85ad81 100644 --- a/README.md +++ b/README.md @@ -1,118 +1,66 @@ # Mountpoint for Amazon S3 -[![CI Status](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml/badge.svg?branch=main)](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml) +[![Tests](https://github.com/awslabs/mountpoint-s3/actions/workflows/tests.yml/badge.svg)](https://github.com/awslabs/mountpoint-s3/actions/workflows/tests.yml) +[![Integration tests](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml/badge.svg?branch=main)](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml) Mountpoint for Amazon S3 is a simple, high-throughput file client for [mounting an Amazon S3 bucket as a local file system](https://aws.amazon.com/blogs/storage/the-inside-story-on-mountpoint-for-amazon-s3-a-high-performance-open-source-file-client/). With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like `open` and `read`. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface. -Mountpoint for Amazon S3 is optimized for read-heavy workloads that need high throughput, and does not implement the full POSIX specification for file systems. For example, Mountpoint for Amazon S3 only supports write operations to new files, and writes must be made to the file sequentially. See [SEMANTICS.md](doc/SEMANTICS.md) for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. For workloads that need full POSIX support, we recommend [Amazon FSx for Lustre](https://aws.amazon.com/fsx/lustre/) and its [support for linking S3 buckets](https://docs.aws.amazon.com/fsx/latest/LustreGuide/create-dra-linked-data-repo.html). +Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to: +* read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first +* access only some S3 objects out of a larger data set, but can't predict which objects in advance +* upload their output to S3 directly, or upload files from local storage with tools like `cp` -## Current status - -Mountpoint for Amazon S3 is **currently an alpha release and should not be used in production**. We're tracking its production readiness and future features on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We're especially interested in early feedback on features, performance, and compatibility. Please send feedback by [opening a GitHub issue](https://github.com/awslabs/mountpoint-s3/issues/new/choose). - -This release has some notable restrictions: -* The only way to install the alpha release of Mountpoint for Amazon S3 is by compiling from source (see [Installation](#installation) below). This will change in a future release. -* Manual endpoint configuration might be required for some S3 customers (see [Configuration and usage](#configuration-and-usage) below). -* Objects written with Mountpoint for Amazon S3 are [always stored in the S3 Standard storage class](https://github.com/awslabs/mountpoint-s3/issues/34). - -## Getting started - -The alpha release of Mountpoint for Amazon S3 is only available by building from source, and only supports Linux. - -### Installation +but probably not the right fit for applications that: +* use file operations that S3 doesn't natively support, like directory renaming or symlinks +* make edits or appends to existing files in-place (don't work on your Git repository or SQLite database in Mountpoint 😄) -First, install the necessary dependencies. For RPM-based distributions (Amazon Linux 2 and 2023, Fedora, CentOS, etc): +Mountpoint for Amazon S3 does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. See [Mountpoint file system behavior](doc/SEMANTICS.md) for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. - sudo yum install fuse fuse-devel cmake3 clang git pkg-config - -or for apt-based distributions (Debian, Ubuntu, etc): - - sudo apt install fuse libfuse-dev cmake clang git pkg-config - -Second, install the Rust compiler via [rustup](https://rustup.rs/): - - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - source "$HOME/.cargo/env" - -Now clone this repository and its submodules: - - git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git +## Current status -Finally, compile the client: +Mountpoint for Amazon S3 is generally available! We're tracking future feature development on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We're always interested in feedback on features, performance, and compatibility. Please send feedback by [opening a new GitHub issue](https://github.com/awslabs/mountpoint-s3/issues/new/choose) or adding your input to an existing roadmap issue. - cd mountpoint-s3 - cargo build --release +## Getting started -The final binary will be at `target/release/mount-s3`. -The following instructions assume you have the binary on your path. -Add it to a location on your path now, or prefix `mount-s3` commands with the path to the binary. +Run these two commands to install Mountpoint for Amazon S3 on your Amazon Linux EC2 instance (for Graviton instances, replace `x86_64` with `arm64` in the URL): -### Configuration and usage + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm + sudo yum install -y ./mount-s3.rpm -To use Mountpoint for Amazon S3, your host needs [access to valid AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). For example, you could [create a new IAM user and add its credentials to the `~/.aws/credentials` file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or [associate your EC2 instance with an IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). +On Ubuntu, use these commands instead (for Graviton instances, replace `x86_64` with `arm64` in the URL): -Launch Mountpoint for Amazon S3 by running the `mount-s3` command. -Replace `DOC-EXAMPLE-BUCKET` with the name of your bucket, -and specify the directory in which your S3 bucket should be mounted (here, `~/mnt`): + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb + sudo apt-get install -y ./mount-s3.deb - mkdir ~/mnt - mount-s3 DOC-EXAMPLE-BUCKET ~/mnt +See [Installing Mountpoint for Amazon S3](doc/INSTALL.md) for detailed instructions and other installation options (including Docker or building from source). -The client will run in the background by default, and the `~/mnt` directory now gives access to the objects in your S3 bucket. +Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket. +You'll need valid AWS credentials to access your bucket; +for example, Mountpoint will automatically use credentials from [an IAM role associated with your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html), +or you can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. -To unmount your S3 bucket and exit Mountpoint for Amazon S3, run the `umount` command: +To mount your bucket, run this command, +replacing `DOC-EXAMPLE-BUCKET` with the name of your bucket +and `/path/to/mount` with the directory you want to mount the bucket to: - umount ~/mnt + mount-s3 DOC-EXAMPLE-BUCKET /path/to/mount -### Additional configuration +Now you can work with your bucket contents as if they were a local file system: -Mountpoint for Amazon S3 supports the same [configuration options for AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) as the AWS CLI. For example, to use a different profile from your `~/.aws/config` configuration file, set the `AWS_PROFILE` environment variable or pass the `--profile` argument to `mount-s3`. + ls /path/to/mount + echo "Hello World!" > /path/to/mount/Data.txt + cat /path/to/mount/Data.txt -To use Mountpoint for Amazon S3 with [S3 Access Points](https://aws.amazon.com/s3/features/access-points/), use the [bucket-style access point alias](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-alias.html) of the access point as the bucket name to mount. +When you're finished accessing your bucket, you can unmount it (you might need `sudo`): -For the alpha release, additional configuration is required to use Mountpoint for Amazon S3 in some circumstances: + umount /path/to/mount -* **AWS GovCloud, China (Beijing), and China (Ningxia) regions**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument **and** the region name with the `--region` argument. For example, for the AWS GovCloud (US-West) region, specify `--endpoint-url https://s3.us-gov-west-1.amazonaws.com --region us-gov-west-1`. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These regions will be supported via only the `--region` command-line argument in the future (https://github.com/awslabs/mountpoint-s3/issues/4). -* **FIPS, dualstack, and transfer acceleration endpoints**: manually specify the S3 endpoint with the `--endpoint-url` command-line argument. See [the S3 endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) for more details. These endpoints will be configurable in the future (https://github.com/awslabs/mountpoint-s3/issues/4). +See [Configuring Mountpoint for Amazon S3](doc/CONFIGURATION.md) for more details on how to configure and use Mountpoint, including options for providing AWS credentials. -#### Compatibility with other storage services +### Compatibility with other storage services Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3. -### Alternative installation options - -#### Mountpoint for Amazon S3 using Docker - -A sample [Dockerfile](docker/Dockerfile) is provided, using Amazon Linux 2023 as a base. - -First, build the container image: - - cd mountpoint-s3/docker - docker build -t mountpoint-s3 . - -Before running the container, you'll need to provide AWS credentials as [described above](#additional-configuration). -We recommend using short-term credentials whenever possible. -For example, if you run your containers on an EC2 instance, -you can [associate an IAM role with the instance](https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-roles-for-ec2.html) -and Mountpoint for Amazon S3 will automatically assume that role at startup. -[Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) also supports short-term credentials -by [associating an IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) with your ECS task. -If you need to use long-term credentials, set the [`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html), and pass those variables to Docker with the `--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY` arguments. - -To launch the container, run the following command, -replacing `DOC-EXAMPLE-BUCKET` with the name of your S3 bucket: - - docker run -it --cap-add SYS_ADMIN --device /dev/fuse \ - mountpoint-s3 /bin/bash -c "./mount-s3 DOC-EXAMPLE-BUCKET /mnt" - -Inside the container, your bucket will be mounted at the `/mnt` path. - -#### Arch Linux AUR (unofficial) - -If you're using Arch Linux, you can use the unofficial [AUR](https://aur.archlinux.org/packages/mountpoint-s3-git) package. Using [Yay](https://github.com/Jguer/yay): - - yay -S mountpoint-s3-git - ## Contributing We welcome contributions to Mountpoint for Amazon S3! Please see [CONTRIBUTING.md](doc/CONTRIBUTING.md) for more information on how to report bugs or submit pull requests. We especially welcome contributions to issues tagged as [good first issues to work on](https://github.com/awslabs/mountpoint-s3/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). @@ -132,6 +80,6 @@ This project has adopted the [Amazon Open Source Code of Conduct](https://aws.gi This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily: * The [fuser](https://github.com/cberner/fuser) bindings for FUSE (MIT license) -* The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache-2.0 license) +* The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache 2.0 license) See [deny.toml](deny.toml) for a list of licenses used by our dependencies. diff --git a/doc/INSTALL.md b/doc/INSTALL.md new file mode 100644 index 000000000..612505ed7 --- /dev/null +++ b/doc/INSTALL.md @@ -0,0 +1,204 @@ +# Installing Mountpoint for Amazon S3 + +We recommend installing Mountpoint for Amazon S3 by downloading and installing prebuilt packages using the command line. +Other installation options are also available, but are not officially supported by AWS. + +Mountpoint for Amazon S3 is only available for Linux operating systems. + +## Download and install Mountpoint for Amazon S3 from the command line (recommended) + +The instructions for downloading and installing Mountpoint for Amazon S3 depend on which Linux operating system you are using. + +### RPM-based distributions (Amazon Linux, Fedora, CentOS, RHEL, SUSE) + +To download and install Mountpoint for Amazon S3 on RPM-based distributions, including Amazon Linux, follow these steps: + +1. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: + * x86_64: + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm + ``` + * ARM64 (Graviton): + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.rpm + ``` +2. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: + * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm.asc` + * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.rpm.asc` + + Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. +3. Install the package by entering the following command: + ``` + sudo yum install ./mount-s3.rpm + ``` +4. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: + ``` + mount-s3 --version + ``` + You should see output similar to the following: + ``` + mount-s3 1.0.0 + ``` + +### DEB-based distributions (Debian, Ubuntu) + +To download and install Mountpoint for Amazon S3 on DEB-based distributions, follow these steps: + +1. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: + * x86_64: + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb + ``` + * ARM64 (Graviton): + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.deb + ``` +2. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: + * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb.asc` + * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.deb.asc` + + Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. +3. Install the package by entering the following command: + ``` + sudo apt-get install ./mount-s3.deb + ``` +4. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: + ``` + mount-s3 --version + ``` + You should see output similar to the following: + ``` + mount-s3 1.0.0 + ``` + +### Other Linux distributions + +To download and install Mountpoint for Amazon S3 on other Linux distributions, follow these steps: + +1. Install the necessary dependencies. Consult your operating system documentation to install the FUSE and libfuse (v2) packages. +2. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: + * x86_64: + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.tar.gz + ``` + * ARM64 (Graviton): + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.tar.gz + ``` +3. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: + * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.tar.gz.asc` + * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.tar.gz.asc` + + Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. +4. Install the package by entering the following command: + ``` + sudo mkdir -p /opt/aws/mountpoint-s3 && sudo tar -C /opt/aws/mountpoint-s3 -xzf ./mount-s3.tar.gz + ``` +5. Add the `mount-s3` binary to your PATH environment variable. To do so, add the following line to your `$HOME/.profile` file: + ``` + export PATH=$PATH:/opt/aws/mountpoint-s3/bin + ``` + After adding this line, enter the following command: + ``` + source $HOME/.profile + ``` +6. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: + ``` + mount-s3 --version + ``` + You should see output similar to the following: + ``` + mount-s3 1.0.0 + ``` + +### Optional: Verifying the signature of the Mountpoint for Amazon S3 package + +To verify the authenticity and integrity of a downloaded Mountpoint for Amazon S3 package, follow these steps: + +1. Install GnuPG (the `gpg` command) on your host if it is not already available. GnuPG is installed by default on Amazon Linux AMIs. +2. Download the Mountpoint for Amazon S3 public key by entering the following command: + ``` + wget https://s3.amazonaws.com/mountpoint-s3-release/public_keys/KEYS + ``` +3. Import the Mountpoint for Amazon S3 public key into your keyring by entering the following command: + ``` + gpg --import KEYS + ``` +4. Verify the fingerprint of the Mountpoint for Amazon S3 public key by entering the following command: + ``` + gpg --fingerprint mountpoint-s3@amazon.com + ``` + Confirm that the displayed fingerprint string matches the following: + ``` + 673F E406 1506 BB46 9A0E F857 BE39 7A52 B086 DA5A + ``` + If the fingerprint string does not match, do not finish installing Mountpoint for Amazon S3. +5. Download the package signature file by entering the following command. For `signature-link`, use the appropriate signature link from the sections above. + ``` + wget signature-link + ``` +6. Verify the signature of the downloaded package by entering the following command. Use the appropriate `signature-filename` from the previous step. + ``` + gpg --verify signature-filename + ``` + For example, on RPM-based distributions, including Amazon Linux, enter the following command. + ``` + gpg --verify mount-s3.rpm.asc + ``` + The output should report a `Good signature`. If the output includes the phrase `BAD signature`, re-download the Mountpoint for Amazon S3 package file and repeat these steps. If the issue persists, do not finish installing Mountpoint for Amazon S3. The output may include a warning about a trusted signature. This does not indicate a problem, only that you have not independently verified the Mountpoint for Amazon S3 public key. + +## Building Mountpoint for Amazon S3 from source + +You can build Mountpoint for Amazon S3 from source. However, binaries built in this way are not officially supported by AWS. + +1. Install the necessary dependencies. + * For RPM-based distributions (Amazon Linux, Fedora, CentOS, RHEL, SUSE), enter the following command: + ``` + sudo yum install fuse fuse-devel cmake3 clang git pkgconfig + ``` + * For DEB-based distributions (Debian, Ubuntu), enter the following command: + ``` + sudo apt-get install fuse libfuse-dev cmake clang git pkg-config + ``` +2. Install the Rust compiler using [rustup](https://rustup.rs/) by entering the following commands: + ``` + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + source "$HOME/.cargo/env" + ``` +3. Clone the Mountpoint for Amazon S3 source code from GitHub by entering the following command: + ``` + git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git + ``` +4. Build Mountpoint for Amazon S3 by entering the following commands: + ``` + cd mountpoint-s3 + cargo build --release + ``` +5. The final binary will be at `target/release/mount-s3`. Optionally, you can install this binary by copying it to the `/usr/bin` directory with the following command: + ``` + sudo cp target/release/mount-s3 /usr/bin/ + ``` + +## Running Mountpoint for Amazon S3 in a Docker container + +You can run Mountpoint for Amazon S3 in a Docker container using our provided Dockerfiles. +We don't recommend this option, as running a FUSE file system like Mountpoint inside a container +requires giving the container broad root-level privileges to your host system. + +See [Running Mountpoint for Amazon S3 in a Docker container](../docker/README.md) for instructions on using Mountpoint with Docker. + +## Community-supported installation options + +These installation options are not provided by AWS. + +### Arch Linux + +If you're using Arch Linux, you can use the unofficial [AUR](https://aur.archlinux.org/packages/mountpoint-s3-git) package. Using [Yay](https://github.com/Jguer/yay): + + yay -S mountpoint-s3-git + +## Installing previous Mountpoint for Amazon S3 releases + +We recommend always installing the latest Mountpoint for Amazon S3 release, +but if you need to install a previous version, +you can find links to them on the [GitHub Releases](https://github.com/awslabs/mountpoint-s3/releases) page. \ No newline at end of file diff --git a/docker/Dockerfile b/docker/Dockerfile index f19e9bb62..4155cac40 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,34 +1,31 @@ -FROM amazonlinux:2023 as builder +# Download and verify the RPM in this container +FROM public.ecr.aws/amazonlinux/amazonlinux:2023 as builder -# Install build tools -RUN dnf upgrade -y && \ - dnf install -y \ - fuse \ - fuse-devel \ - cmake3 \ - clang \ - git \ - pkg-config && \ - dnf clean all - -# Install rust -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \ - source "$HOME/.cargo/env" +# We need the full version of GnuPG +RUN dnf install -y --allowerasing wget gnupg2 -# Build mountpoint-s3 -RUN git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git && \ - source "$HOME/.cargo/env" && \ - cd mountpoint-s3 && \ - cargo build --release +RUN MP_ARCH=`uname -p | sed s/aarch64/arm64/` && \ + wget -q "https://s3.amazonaws.com/mountpoint-s3-release/latest/$MP_ARCH/mount-s3.rpm" && \ + wget -q "https://s3.amazonaws.com/mountpoint-s3-release/latest/$MP_ARCH/mount-s3.rpm.asc" && \ + wget -q https://s3.amazonaws.com/mountpoint-s3-release/public_keys/KEYS +# Import the key and validate it has the fingerprint we expect +RUN gpg --import KEYS && \ + (gpg --fingerprint mountpoint-s3@amazon.com | grep "673F E406 1506 BB46 9A0E F857 BE39 7A52 B086 DA5A") +# Verify the downloaded binary +RUN gpg --verify mount-s3.rpm.asc -FROM amazonlinux:2023 as release -COPY --from=builder /mountpoint-s3/target/release/mount-s3 /mount-s3 +# Install the RPM in a fresh container to minimize dependencies +FROM amazonlinux:2023 +COPY --from=builder /mount-s3.rpm /mount-s3.rpm RUN dnf upgrade -y && \ - dnf install -y \ - fuse-devel && \ - dnf clean all + dnf install -y ./mount-s3.rpm && \ + dnf clean all && \ + rm mount-s3.rpm + +RUN echo "user_allow_other" >> /etc/fuse.conf -CMD [ "/mount-s3" ] +# Run in foreground mode so that the container can be detached without exiting Mountpoint +ENTRYPOINT [ "mount-s3", "-f" ] diff --git a/docker/Dockerfile.source b/docker/Dockerfile.source new file mode 100644 index 000000000..2a667c5ed --- /dev/null +++ b/docker/Dockerfile.source @@ -0,0 +1,36 @@ +FROM amazonlinux:2023 as builder + +# Install build tools +RUN dnf upgrade -y && \ + dnf install -y \ + fuse \ + fuse-devel \ + cmake3 \ + clang \ + git \ + pkg-config && \ + dnf clean all + +# Install rust +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \ + source "$HOME/.cargo/env" + +# Build mountpoint-s3 +RUN git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git && \ + source "$HOME/.cargo/env" && \ + cd mountpoint-s3 && \ + cargo build --release + + + +FROM amazonlinux:2023 as release +COPY --from=builder /mountpoint-s3/target/release/mount-s3 /mount-s3 + +RUN dnf upgrade -y && \ + dnf install -y fuse fuse-libs && \ + dnf clean all + +RUN echo "user_allow_other" >> /etc/fuse.conf + +# Run in foreground mode so that the container can be detached without exiting Mountpoint +ENTRYPOINT [ "/mount-s3", "-f" ] diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..55cc841e2 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,82 @@ +# Running Mountpoint for Amazon S3 in a Docker container + +This directory contains two Dockerfiles for building container images to run Mountpoint for Amazon S3. +The primary `Dockerfile` builds an image with the latest official Mountpoint release. +The alternative `Dockerfile.source` builds an image by compiling Mountpoint from the latest source version. +We recommend using the primary Dockerfile. + +## Building the image + +First, download the Dockerfile if you don't have it already: + + wget https://github.com/awslabs/mountpoint-s3/blob/main/docker/Dockerfile + +Run this command to build a Docker image: + + docker build -t mountpoint-s3 . + +To build Mountpoint from source, download `Dockerfile.source` instead, and pass the `-f Dockerfile.source` argument to this command. + +## Running the container + +Before running the container, you'll need a way to pass AWS credentials into the container. +See [AWS credentials](../doc/CONFIGURATION.md#aws-credentials) in the configuration documentation for more details. +We recommend using short-term credentials whenever possible: +* If you run your containers on an EC2 instance, you can [associate an IAM role with the instance](https://docs.aws.amazon.com/sdkref/latest/guide/access-iam-roles-for-ec2.html) and Mountpoint for Amazon S3 will automatically assume that role at startup. You may need to [increase the hop limit for IMDS requests](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/configuring-IMDS-existing-instances.html#modify-PUT-response-hop-limit) on your instance for the container to successfully acquire credentials this way. +* [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) supports short-term credentials by [associating an IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) with your ECS task. + +If you need to use long-term credentials, set the [`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html), and pass those variables to Docker with the `--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY` arguments. + +### Running interactively + +To launch Mountpoint in an interactive container, run this command: + + docker run -ti --cap-add SYS_ADMIN --device /dev/fuse --entrypoint bash mountpoint-s3 + +Within the container you can run this command to mount a bucket to the `/mnt` directory, +replacing `DOC-EXAMPLE-BUCKET` with the name of your S3 bucket: + + mount-s3 DOC-EXAMPLE-BUCKET /mnt + +### Running as a service + +You can also run the Docker container as a service, +and access the mounted directory from your host or other Docker containers. +To do so, first create a directory `/path/to/mount` in your host filesystem, +and a subdirectory `/path/to/mount/bucket` to be the target of the mount. +Then run this command, replacing DOC-EXAMPLE-BUCKET with the name of your S3 bucket, +and `/path/to/mount` with the directory you created: + + docker run -d --cap-add SYS_ADMIN --device /dev/fuse \ + --mount type=bind,source=/path/to/mount,target=/mountpoint,bind-propagation=shared \ + mountpoint-s3 \ + DOC-EXAMPLE-BUCKET /mountpoint/bucket + +Your bucket is now mounted in the `/path/to/mount/bucket` directory on the host. +By default, only the user used by the container (likely `root`) will have access to the mount. +You can make the mount accessible to other users by adding the `--allow-other` flag to end of the above command. +See [File and directory permissions](../doc/CONFIGURATION.md#file-and-directory-permissions) +in the configuration documentation for more details on managing this permissions issue. + +### Docker permissions and AWS container services + +Mountpoint uses the Linux FUSE subsystem to mount its file system. +Running a FUSE file system inside a Docker container +requires giving the container elevated root-level privileges to the host instance. +The commands above do this with the `--cap-add SYS_ADMIN` argument to `docker run`. + +These elevated privileges aren't available in AWS-managed container environments like [AWS Fargate](https://aws.amazon.com/fargate/). +They are available in [Amazon Elastic Container Service (ECS)](https://aws.amazon.com/ecs/), +but not enabled by default. +To enable them, add these fields to the `containerDefinitions` section of an ECS task definition: +``` +"privileged": true, +"linuxParameters": { + "devices": [ + { + "hostPath": "/dev/fuse", + "containerPath": "/dev/fuse" + } + ] +} +``` \ No newline at end of file