VAST Cluster enables you to store and access data in the Element Store using Amazon's Simple Storage Service (S3) as an access protocol. S3 is supported as part of VAST Cluster's multiprotocol access feature which enables you to provide shared access to data sets via different storage and file system protocols.
Amazon S3 is a REST service. VAST supports a subset of Amazon S3's REST APIs. Using REST, client users can use HTTP requests to perform operations on the S3 service, such as create, fetch, and delete buckets and objects.
In S3 nomenclature, the basic unit of storage is an object. Every object is stored in a bucket. Buckets organize the S3 namespace.
Objects consist of object data and metadata. The metadata is a set of name-value pairs that describe the object. These include some default metadata, such as the date last modified, and standard HTTP metadata, such as Content-Type. You can also specify custom metadata at the time the object is stored. An object is uniquely identified within a bucket by a key.
Although the S3 structure is flat, consisting of buckets with no hierarchy of sub-buckets and sub-folders, the '/' character acts as a delimiter from which a hierarchy can be inferred from object keys. Object listings can be filtered by any prefix that appears before the delimiter. For example, an object with key "2019-07-02/AboutVASTS3.rtf" would be returned by an object listing filtered by the prefix "2019-07-02/".
Objects can be up to one TB in size and up to two KB in metadata.
In order to make any type of authenticated remote procedure calls (RPCs), each client user requires an access key and a secret key. S3 access key pairs are generated via VMS per user. There is no other way to generate valid access key pairs for the VAST Cluster S3 service.
To manage S3 access key pairs, see Managing User S3 Permissions.
Every interaction with the S3 service is either authenticated or anonymous.
Authentication is a process of verifying the identity of the requester trying to access the service. Authenticated requests must include a signature value that authenticates the request sender. VAST Cluster supports AWS signature version 2 and AWS signature version 4.
The signature must be computed from an access key pair. In order to make authenticated requests, a user needs an access key pair, which must be generated via VMS. When using S3 clients such as Boto 3 and S3cmd, the user provides the access key pair and the client's libraries compute the signature from the provided access key pair. Users making direct REST API calls in their own application must write the code to compute the signature and add it to the request.
It is possible to request access without providing an access key or signature. This is known as anonymous access. Anonymous access is blocked by default and can be allowed per bucket through a setting in the view configuration called Anonymous Access.
When anonymous access is allowed, anonymous requests to access an object are granted depending on permissions granted to all users:
If the security flavor set in the view policy is S3 Native, and the individual object's ACL explicitly grants access to the "AllUsers" group, then anonymous requests are granted.
If the security flavor is NFS, and the permission mode bits on the directory and on the file allow access to "others", then anonymous requests are granted.
Connection over HTTPS is encrypted using the VAST Data FIPS Object Module for OpenSSL, which is certified compliant with the requirements of FIPS 140-2. The NIST validation for the module can be found at https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4107.
Connection over HTTP and over HTTPS are both enabled by default. Each option can be disabled in the cluster settings (see S3 Connection Settings).
Permissions are controlled through the following mechanisms:
S3 User policies. This is the recommended and newer method of managing access control. All supported S3 actions can be permitted through S3 user policies. S3 user policies define permissions through statements that allow and deny specified scopes of actions on specified scopes of resources. They are manageable via VMS, where they can be created, attached to users and attached to groups of users. For full details about using S3 user policies, see Managing S3 User Policies.
S3 ACLs. These are legacy access management entities that are attached to buckets and objects and define grantees and wide scopes of permissions. Newer features require permissions that cannot be granted via ACLs. A basic default ACL is set automatically on each bucket and object. The ACLs can be modified by S3 API requests. For information about S3 ACLs, see Managing S3 Access Control Rules (ACLs).
User settings in VMS. VMS enables the following S3 permission settings per user. They are overruled by conflicting permissions set by user policies:
Permission to create buckets.
Permission to delete buckets.
'superuser' permission, a VAST permission that gives the user the equivalent of ACL 'full control' permission on all buckets and objects, overriding ACLs. In multiprotocol terms, it is considered equivalent to the NFS super user "root".
For information about managing these permissions, see Managing User S3 Permissions.
The S3 Native and NFS security flavors are compatible with S3 access. Access checking follows different algorithms depending on the security flavor configured in the view policy used by the view in which the requested resource resides.
When the security flavor is S3 Native, access checking is performed as follows:
If an S3 user policy that is attached to the user or to one of the user's groups denies the requested action, the request is denied. For NFS requests, this applies only to request types that map to S3 user permissions.
Otherwise, if an S3 user policy attached to the user or to one of the user's groups allows the requested action, the request is allowed. For NFS requests, this applies only to request types that map to S3 user permissions.
Otherwise, permissions are checked according to S3 ACLs and the VMS-granted user permissions.
When the security flavor is NFS, S3 requests are checked first against S3 user policies, while NFS requests are not. S3 requests that are neither denied nor allowed by S3 user policies, are checked against NFS permission mode bits. A default mode bits configuration defined in the view policy is applied to objects that are created via S3. NFS requests are checked only against NFS permission mode bits.
All S3 buckets are manageable via VMS as part of the view feature, which enables multiprotocol access.
S3 buckets can be created by creating views via VMS and enabling an S3 bucket on each view. It is also possible for users to create buckets via RPC. A view is created for the bucket, and the view is manageable via VMS.
For more information, see:
The following rules apply to bucket names:
Bucket names must be unique across all existing bucket names on the cluster.
Bucket names must comply with DNS naming conventions.
Bucket names must be at least 3 and no more than 63 characters long.
Bucket names must not contain uppercase characters or underscores.
Bucket names must start with a lowercase letter or number.
Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.).
Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.
The following rules apply to object names:
Object names (keys) must be up to 1024 characters long.
Object names can contain up to 200 '/' characters.
Object names can contain slashes. This is useful for 'S3 directory' type usage.
Limitations apply to accessibility via other protocols. See S3 Directories for details.
When nested files and directories are created by other protocols under a view, the files may be accessed as S3 objects using the file path within the view, including delineating slashes, as the object name. See Accessing Subdirectories of Multi-Protocol Views.
The VAST Cluster S3 service supports a subset of the actions supported by the AWS S3 API. For a list of supported actions, see Supported S3 API Actions.
The following features are supported: