VAST Cluster enables you to store and access data in the Element Store using Amazon's Simple Storage Service (S3) as an access protocol. S3 is supported as part of VAST Cluster's multiprotocol access feature which enables you to provide shared access to data sets via different storage and file system protocols.
Amazon S3 is a REST service. VAST supports a subset of Amazon S3's REST APIs. Using REST, client users can use HTTP requests to perform operations on the S3 service, such as create, fetch, and delete buckets and objects.
In S3 nomenclature, the basic unit of storage is an object. Every object is stored in a bucket. Buckets organize the S3 namespace.
Objects consist of object data and metadata. The metadata is a set of name-value pairs that describe the object. These include some default metadata, such as the date last modified, and standard HTTP metadata, such as Content-Type. You can also specify custom metadata at the time the object is stored. An object is uniquely identified within a bucket by a key.
Although the S3 structure is flat, consisting of buckets with no hierarchy of sub-buckets and sub-folders, the '/' character acts as a delimiter from which a hierarchy can be inferred from object keys. Object listings can be filtered by any prefix that appears before the delimiter. For example, an object with key "2019-07-02/AboutVASTS3.rtf" would be returned by an object listing filtered by the prefix "2019-07-02/".
Objects can be up to one TB in size and up to two KB in metadata.
All S3 buckets are manageable via VMS as part of the view feature, which enables multiprotocol access.
S3 buckets can be created by creating views via VMS and enabling an S3 bucket on each view. It is also possible for users to create buckets via RPC. A view is created for the bucket, and the view is manageable via VMS.
For more information, see:
Previous to VAST Cluster 4.2, S3 buckets were not part of the view feature. When upgrading to 4.2, a view is created for every S3 bucket that already exists. This view is given the default S3 view policy with S3 Native security flavor. It is possible to change the policy and thereby change the flavor to NFS.
In order to make any type of authenticated remote procedure calls (RPCs), each client user requires an access key and a secret key. S3 access key pairs are generated via VMS per user. There is no other way to generate valid access key pairs for the VAST Cluster S3 service.
Permission to create buckets and permission to delete buckets via S3 RPC are granted via VMS per user.
It is also possible to grant 'S3 superuser' permission via VMS to a given user. This is a special permission that gives the user full access to buckets and objects, overriding ACLs. In multiprotocol terms, it is considered equivalent to the NFS super user "root".
To manage S3 access key pairs and permissions, see Managing User S3 Permissions.
Every interaction with the S3 service is either authenticated or anonymous.
Authentication is a process of verifying the identity of the requester trying to access the service. Authenticated requests must include a signature value that authenticates the request sender. VAST Cluster supports AWS signature version 2 and AWS signature version 4.
The signature must be computed from an access key pair. In order to make authenticated requests, a user needs an access key pair, which must be generated via VMS. When using S3 clients such as Boto 3 and S3cmd, the user provides the access key pair and the client's libraries compute the signature from the provided access key pair. Users making direct REST API calls in their own application must write the code to compute the signature and add it to the request.
It is possible to request access without providing an access key or signature. This is known as anonymous access. Anonymous access is blocked by default and can be allowed per bucket through a setting in the view configuration called Anonymous Access.
When anonymous access is allowed, anonymous requests to access an object are granted depending on permissions granted to all users:
If the security flavor set in the view policy is S3 Native, and the individual object's ACL explicitly grants access to the "AllUsers" group, then anonymous requests are granted.
If the security flavor is NFS, and the permission mode bits on the directory and on the file allow access to "others", then anonymous requests are granted.
Connection over HTTPS is encrypted using the VAST Data FIPS Object Module for OpenSSL, which is certified compliant with the requirements of FIPS 140-2. The NIST validation for the module can be found at https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4107.
Connection over HTTP and over HTTPS are both enabled by default. Each option can be disabled in the cluster settings (see S3 Connection Settings).
The access check algorithm and the type of ACLs that are honored depends on the security flavor defined in the view policy.
The following security flavors support S3 bucket views:
With S3 Native flavor, S3 ACLs on the bucket and on objects in the bucket are set by default and can be modified via S3 RPC. For information about S3 ACLs, see S3 Access Control Rules (ACLs) .
With NFS flavor, ACLs cannot be changed by S3 clients.
For information about these security flavors and how permissions work with each flavor, see Controlling File and Directory Permissions Across Protocols.
The following rules apply to bucket names:
Bucket names must be unique across all existing bucket names on the cluster.
Bucket names must comply with DNS naming conventions.
Bucket names must be at least 3 and no more than 63 characters long.
Bucket names must not contain uppercase characters or underscores.
Bucket names must start with a lowercase letter or number.
Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.).
Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.
The following rules apply to object names:
Object names (keys) must be up to 1024 characters long.
Object names can contain up to 200 '/' characters.
Object names can contain slashes. This is useful for 'S3 directory' type usage.
Limitations apply to accessibility via other protocols. See S3 Directories for details.
When nested files and directories are created by other protocols under a view, the files may be accessed as S3 objects using the file path within the view, including delineating slashes, as the object name. See Accessing Subdirectories of Multi-Protocol Views.