S3

Simple Storage Service

S3 - Simple Storage Service - https://aws.amazon.com/s3/

  • S3 is object storage

    • Can store any files

    • Size from 0 bytes to 5 TB

  • S3 is universal namespace

    • Each bucket name must be unique across entire S3

  • Objects

    • Key - name of the object

    • Value - data

    • Version ID

    • Metadata

    • Subresources

      • Access Control Lists (ACLs)

      • Torrents

  • Consistency

    • Read after write consistency for PUTs of new objects

    • Eventual consistency for overwrite PUTs and DELETEs (can take some time to propagate)

  • Guarantees

    • 99.99% availability

    • 99.999999999% durability (11x9s - "eleven nines")

  • Features

    • Tiered storage

    • Lifecycle management

    • Versioning

    • Encryption

    • MFA delete

    • Security with ACLs and Bucket Policies

  • Storage classes

    • S3 Standard - 99.99% availability and 11x9s durability, stored redundantly across multiple devices in multiple facilities, and designed to sustain the loss of 2 facilities concurrently.

    • S3 IA (Infrequent Access) - for data that is accessed less frequently but requires rapid access when needed.

    • S3 One Zone IA - cheaper than IA. For data that does not require multiple Availability Zone data resilience.

    • S3 Intelligent Tiering - optimizes cost by automatically moving objects into appropriate and most cost effective tier.

    • S3 Glacier - low cost storage for data archiving. Retrieval times configurable.

    • S3 Glacier Deep Archive - lowest cost with 12 hours of retrieval times.

  • S3 charges

    • Storage

    • Requests

    • Storage Management Pricing

    • Data Transfer Pricing

    • Transfer Acceleration

    • Cross Region Replication Pricing

  • MFA Delete and Versioning

    • S3 versioning enables you to revert to older versions of S3 objects.

    • Multiple versions of an object are stored in the same bucket.

    • Versioning also protects you from accidental / malicious deletes.

    • With versioning enabled, a DELETE action doesn't delete the object version, but applies a delete marker instead.

    • To permanently delete, provide the object Version ID in the delete request.

    • MFA Delete provides an additional layer of protection to S3 Versioning.

    • Once enabled, MFA Delete will enforce 2 things:

      • You'll need a valid code from your MFA device in order to permanently delete an object version.

      • MFA also needed to suspend / reactivate versioning on an S3 bucket.

  • Encryption

    • in-transit - SSL / TLS

    • at rest - Server-side achieved by:

      • SSE-S3 - S3 managed key

      • SSE-KMS - Using Key Management Service

      • SSE-C - Server side encryption with customer provided key

    • client-side encryption

  • Cross-region replication

    • Versioning must be enabled on both the source and the destination buckets

    • Regions must be unique

    • Existing files in the existing bucket are not replicated automatically

    • All subsequent updates will be replicated automatically

    • Delete markers are not replicated

    • Deleting individual versions or delete markers are not replicated

  • Lifecycle policies

    • Automates moving your objects between different storage tiers

    • Can be used in conjunction with versioning

    • Can be applied to current versions and previous versions

  • Transfer Acceleration

    • Files are uploaded to edge locations first, and then from edge locations propagated to S3

  • Largest file that can be uploaded to S3 using PUT is 5Gb

  • Key prefixing is no longer needed to improve performance on S3. S3 uses key-value to determine partition for objects.

Enforcing encryption on S3 buckets

  • If the file is to be encrypted at upload time, the x-amz-server-side-encryption parameter will be included in the request header

  • Two options are currently available:

    • x-amz-server-side-encryption: AES256

    • x-amz-server-side-encryption: kms

  • When this parameter is included in the the header of the PUT request, it tells S3 to encrypt the object at the time of upload, using the specified encryption method.

  • You can enforce the use of Server-Side Encryption by using a Bucket Policy which denies any S3 PUT request which does not include the x-amz-server-side-encryption parameter in the request header.

Example of the bucket policy that denies PutObject operations if server-side encryption header is not specified in the request:

{
  "Id": "Policy1585498423331",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1585498419704",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Deny",
      "Resource": "arn:aws:s3:::mybucketnamehere",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      },
      "Principal": "*"
    }
  ]
}

S3 Batch operations

Batch operations is a new feature that makes it simple to manage billions of objects stored in S3. Customers can make changes to object properties and metadata, and perform other storage management tasks - such as copying objects between buckets, replacing tag sets, modifying access controls, and restoring archived objects from Glacier - for any number of S3 objects in minutes.

Sharing S3 bucket across accounts

  • Using bucket policies and IAM (applies across the entire bucket). Programmatic access only.

  • Using bucket ACLs and IAM (individual objects). Programmatic access only.

  • Cross-account IAM roles. Programmatic and Console access.

PreviousStorage & Data ManagementNextStorage Gateway

Last updated