S3

Simple Storage Service

S3 - Simple Storage Service - https://aws.amazon.com/s3/

• S3 is object storage

  • Can store any files

  • Size from 0 bytes to 5 TB

• S3 is universal namespace

  • Each bucket name must be unique across entire S3

• Objects

  • Key - name of the object

  • Value - data

  • Version ID

  • Metadata

  • Subresources

    • Access Control Lists (ACLs)

    • Torrents

• Consistency

  • Read after write consistency for PUTs of new objects

  • Eventual consistency for overwrite PUTs and DELETEs (can take some time to propagate)

• Guarantees

  • 99.99% availability

  • 99.999999999% durability (11x9s - "eleven nines")

• Features

  • Tiered storage

  • Lifecycle management

  • Versioning

  • Encryption

  • MFA delete

  • Security with ACLs and Bucket Policies

• Storage classes

  • S3 Standard - 99.99% availability and 11x9s durability, stored redundantly across multiple devices in multiple facilities, and designed to sustain the loss of 2 facilities concurrently.

  • S3 IA (Infrequent Access) - for data that is accessed less frequently but requires rapid access when needed.

  • S3 One Zone IA - cheaper than IA. For data that does not require multiple Availability Zone data resilience.

  • S3 Intelligent Tiering - optimizes cost by automatically moving objects into appropriate and most cost effective tier.

  • S3 Glacier - low cost storage for data archiving. Retrieval times configurable.

  • S3 Glacier Deep Archive - lowest cost with 12 hours of retrieval times.

• S3 charges

  • Storage

  • Requests

  • Storage Management Pricing

  • Data Transfer Pricing

  • Transfer Acceleration

  • Cross Region Replication Pricing

• MFA Delete and Versioning

  • S3 versioning enables you to revert to older versions of S3 objects.

  • Multiple versions of an object are stored in the same bucket.

  • Versioning also protects you from accidental / malicious deletes.

  • With versioning enabled, a DELETE action doesn't delete the object version, but applies a delete marker instead.

  • To permanently delete, provide the object Version ID in the delete request.

  • MFA Delete provides an additional layer of protection to S3 Versioning.

  • Once enabled, MFA Delete will enforce 2 things:

    • You'll need a valid code from your MFA device in order to permanently delete an object version.

    • MFA also needed to suspend / reactivate versioning on an S3 bucket.

• Encryption

  • in-transit - SSL / TLS

  • at rest - Server-side achieved by:

    • SSE-S3 - S3 managed key

    • SSE-KMS - Using Key Management Service

    • SSE-C - Server side encryption with customer provided key

  • client-side encryption

• Cross-region replication

  • Versioning must be enabled on both the source and the destination buckets

  • Regions must be unique

  • Existing files in the existing bucket are not replicated automatically

  • All subsequent updates will be replicated automatically

  • Delete markers are not replicated

  • Deleting individual versions or delete markers are not replicated

• Lifecycle policies

  • Automates moving your objects between different storage tiers

  • Can be used in conjunction with versioning

  • Can be applied to current versions and previous versions

• Transfer Acceleration

  • Files are uploaded to edge locations first, and then from edge locations propagated to S3

• Largest file that can be uploaded to S3 using PUT is 5Gb

• Key prefixing is no longer needed to improve performance on S3. S3 uses key-value to determine partition for objects.

Enforcing encryption on S3 buckets

• If the file is to be encrypted at upload time, the x-amz-server-side-encryption parameter will be included in the request header

• Two options are currently available:

  • x-amz-server-side-encryption: AES256

  • x-amz-server-side-encryption: kms

• When this parameter is included in the the header of the PUT request, it tells S3 to encrypt the object at the time of upload, using the specified encryption method.

• You can enforce the use of Server-Side Encryption by using a Bucket Policy which denies any S3 PUT request which does not include the x-amz-server-side-encryption parameter in the request header.

Example of the bucket policy that denies PutObject operations if server-side encryption header is not specified in the request:

{
  "Id": "Policy1585498423331",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1585498419704",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Deny",
      "Resource": "arn:aws:s3:::mybucketnamehere",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      },
      "Principal": "*"
    }
  ]
}

S3 Batch operations

Batch operations is a new feature that makes it simple to manage billions of objects stored in S3. Customers can make changes to object properties and metadata, and perform other storage management tasks - such as copying objects between buckets, replacing tag sets, modifying access controls, and restoring archived objects from Glacier - for any number of S3 objects in minutes.

Sharing S3 bucket across accounts

• Using bucket policies and IAM (applies across the entire bucket). Programmatic access only.

• Using bucket ACLs and IAM (individual objects). Programmatic access only.

• Cross-account IAM roles. Programmatic and Console access.