S3 Overview

Enabling S3 User Access

Any client sending remote procedure calls (RPCs) via S3 requires an S3 access key pair in order to be authenticated. S3 access key pairs can be generated for any user via the VAST Web UI or VAST CLI.

Creating and deleting buckets require special permissions, which can also be assigned to each user via the VAST Web UI or VAST CLI.

To manage S3 access key pairs and permissions, see Managing User S3 Permissions.

S3 Concepts

The basic units of S3 storage are objects. Every object is stored in a bucket. Buckets organize the S3 namespace at the highest level.

Objects consist of object data and metadata. The metadata is a set of name-value pairs that describe the object. These include some default metadata, such as the date last modified, and standard HTTP metadata, such as Content-Type. You can also specify custom metadata at the time the object is stored. An object is uniquely identified within a bucket by a key.

Although the S3 structure is flat, consisting of buckets with no hierarchy of sub-buckets and sub-folders, the '/' character acts as a delimiter from which a hierarchy can be inferred from object keys. Object listings can be filtered by any prefix that appears before the delimiter. For example, an object with key "2019-07-02/AboutVASTS3.rtf" would be returned by an object listing filtered by the prefix "2019-07-02/".

Objects can be up to one TB in size and up to two KB in metadata.

Rules for Object Naming

The following rules apply to object names:

Rules for Bucket Naming

The following rules apply to bucket names:

Interfaces

Amazon S3 is a REST service. VAST supports a subset of Amazon S3's REST APIs. Using REST, you can use HTTP requests to perform operations on the S3 service, such as create, fetch, and delete buckets and objects.

You can use any toolkit that supports HTTP to use the REST API. Read our VAST Data-specific guidelines for using the S3cmd command line tool or the Boto 3 SDK for Python.

S3 Service Endpoint

The endpoint for the S3 service can be any of the cluster's Virtual IPs. VAST Cluster can listen to S3 service requests on HTTP or on HTTPS.

Security Credentials and Permissions

Users require an access key and secret key in order to make authenticated requests. Users can create buckets and delete buckets only if they are assigned special permissions for these actions. You can grant users keys and specific permissions to create and delete buckets via the VAST Web UI or via the VAST CLI.

Authenticating Requests

Every interaction with the S3 service is either authenticated or anonymous. Authentication is a process of verifying the identity of the requester trying to access the service. Authenticated requests must include a signature value that authenticates the request sender. VAST Cluster supports AWS signature version 2 and AWS signature version 4.

If you are using Boto 3 or S3cmd, the libraries compute the signature from the keys you provide. However, if you make direct REST API calls in your application, you must write the code to compute the signature and add it to the request.

Managing Access to Data

You can manage access to buckets and objects using Access Control Lists (ACLs). See S3 Access Control Lists (ACLs) for more information.

It is possible to grant 'S3 superuser' permission to a given user. This gives the user full access to buckets and objects, overriding ACLs. For information about how to do this, see Managing User S3 Permissions.

S3 Versioning

You can set a versioning state on a bucket and get information on the state of the bucket's versioning. You can also retrieve metadata about the versions of all the objects in a bucket. See Supported S3 API Operations for more information.