To connect to the low-level client interface, use Boto3’s client() method. You must pass your VAST Cluster S3 credentials and other configurations as parameters with hardcoded values. This is the only way to specify a VAST Cluster VIP as the S3 endpoint.
The following example imports the boto module and instantiates a client with the minimum configuration needed for connecting the client to your VAST Cluster S3 account over an HTTP connection:
import boto3 s3_client = boto3.client( 's3', use_ssl=False, endpoint_url=<ENDPOINT-URL> aws_access_key_id=<ACCESS-KEY>, aws_secret_access_key=<SECRET-KEY> region_name=<REGION> config=boto3.session.Config( signature_version='s3v4' s3={'addressing_style': 'path'} ) )
in which:
-
<ENDPOINT-URL>
can be any of the cluster's Virtual IPs, prefixed by http://. For example,http://198.51.100.255
, in which 198.51.100.255 is one of the cluster's VIPs.Note
To retrieve the cluster's virtual lPs:
-
In the VAST Web UI, open the menu (click
), select Configuration and then select the Virtual IPs tab. The Virtual IPs list shows you which virtual IPs are configured on each CNode.
-
In the VAST CLI, run the
vip list
command.
-
-
<ACCESS-KEY>
and<SECRET-KEY>
are your S3 key pair. -
<REGION>
can be any string. It is required ifsignature_version=S3v4
.
For HTTPS Connection
For an HTTPS connection, pass parameters as follows in the client() call:
-
Enable HTTPS by setting
use_ssl=True
instead ofuse_ssl=False
. -
If the default certificate trust store does not recognize the signer of the installed certificate, you can use the
verify
parameter to specify a non default path to the certificate trust store. If you're using a self signed certificate, you can point this to the certificate itself. For example:verify="path/to/client/cert.pem"
-
Alternatively, you can use the
verify
parameter to disable verification:verify=False
Once you have an instance of the S3 service client, you can call the create_bucket() method on the client instance to create a bucket.
Note
The LocationConstraint(string) parameter can be provided within the CreateBucketConfiguration (dict) parameter to specify a string representing a region where the bucket is created. If you don't specify a region, the region is set to 'vast-1'.
In this example, we create a bucket called mybucket.
response = s3_client.create_bucket( Bucket='mybucket' )
The list_buckets() method returns a list of all buckets owned by the authenticated sender of the request.
The list_objects_v2() method returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.
The list_objects() method is a prior version of the same method, supported for backward compatibility.
This example retrieves the list of objects in the bucket "mybucket".
response = s3_client.list_objects_v2( Bucket='mybucket', )
The head_bucket() method is used to determine if a bucket exists and if the user has permission to access it.
The delete_bucket() method deletes a bucket. All objects in the bucket must be deleted before the bucket can be deleted.
Before setting ACL permissions, we recommend you read Managing S3 Access Control Rules (ACLs).
The put_bucket_acl () method sets the permissions on a bucket using access control lists (ACL).
To grant permission to a user, specify the grantee with the following parameters:
-
For users on external providers only (AD, LDAP etc) pass:
-
The EmailAddress parameter and provide the user's principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS).
-
The Type parameter and provide AmazonCustomerByEmail as its value.
-
-
For any users (including users on the local provider), pass:
-
The ID parameter and provide the user's VID as its value.
Tip
A VID is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the
user query
VAST CLI command and specifying udb as the context of the query. The output includes the user's VID. -
The Type parameter and provide CanonicalUser as its value.
-
To grant permission to a predefined group, specify Group as the 'Type' and pass the group's URI as the 'URI':
-
For the All Users group: 'http://acs.amazonaws.com/groups/global/AllUsers'
-
For the Authenticated Users group: 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
In this example, a user with VID 3 is granted full control permission to the bucket my_bucket owned by JDoe whose VID is 2.
response = s3_client.put_bucket_acl( AccessControlPolicy={ 'Grants': [ { 'Grantee': { 'ID': '54', 'Type': 'CanonicalUser', }, 'Permission': 'FULL_CONTROL' }, ], 'Owner': { 'DisplayName': 'BSmith', 'ID': '4' } }, Bucket='BobsBucket', )
In the following example, the Authenticated_Users group is granted READ permission on the bucket BobsBucket.
response = s3_client.put_bucket_acl( AccessControlPolicy={ 'Grants': [ { 'Grantee': { 'Type': 'Group', 'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers' }, 'Permission': 'READ' }, ], 'Owner': { 'DisplayName': 'BSmith', 'ID': '4' } }, Bucket='BobsBucket', )
The get_bucket_acl() method retrieves the ACL of a bucket.
To learn about VAST Cluster's support for S3 ACLs, read Managing S3 Access Control Rules (ACLs).
The get_bucket_location() method is used to return the region in which the bucket resides.
If a region was specified in the CreateBucket() request, it is returned by this method. If no region was specified in the CreateBucket() request, the bucket region was set to 'vast-1', which is returned by this method.
The copy_object() method creates a copy of an object already stored on the server.
In this example, we copy the object MyObject from the bucket MyBucket to the bucket MyOtherBucket and name the copy MyObjectCopy.
response = S3_client.copy_object( Bucket='MyOtherBucket', CopySource='MyBucket/MyObject' Key='MyObjectCopy', )
The get_object() method retrieves an object.
To download a specified range of bytes of an object, use the Range parameter. For more information about the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.
In this example, we download only bytes 32-64 of the object "MyObject" from the bucket "MyBucket".
response = S3_client.get_object( Bucket='MyBucket', Key='MyObject', Range='bytes=32-64', )
s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Bucket in which object is stored |
OBJECT |
Object for which to retrieve metadata |
Retrieve metadata of file vast.s3cfg.
$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg s3://bucket1/vast.s3cfg (object): File size: 176 Last mod: Mon, 24 Jun 2019 19:40:27 GMT MIME type: text/plain Storage: STANDARD MD5 sum: 9de54b7b7a89d9526b32305a331dea6a SSE: none Policy: none CORS: none ACL: vastuser: FULL_CONTROL x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u name:vastdata
The delete_objects() method deletes multiple objects in a bucket.
response = s3_client.delete_objects( Bucket='mybucket', Delete={ 'Objects': [ {'Key': 'file1'}, {'Key': 'file2'}, {'Key': 'file3'}, ], }, )
Set Access Control List (ACL) Permissions on an Object
Before setting ACL permissions, we recommend you read Managing S3 Access Control Rules (ACLs).
The put_object_acl() method sets the permissions on an object using access control lists (ACL).
To grant permission to a user, specify the grantee with the following parameters:
-
For users on external providers only (AD, LDAP etc) pass:
-
The EmailAddress parameter and provide the user's principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS).
-
The Type parameter and provide AmazonCustomerByEmail as its value.
-
-
For any users (including users on the local provider), pass:
-
The ID parameter and provide the user's VID as its value.
Tip
A VID is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the
user query
VAST CLI command and specifying udb as the context of the query. The output includes the user's VID. -
The Type parameter and provide CanonicalUser as its value.
-
To grant permission to a predefined group, specify Group as the 'Type' and pass the group's URI as the 'URI':
-
For the All Users group: 'http://acs.amazonaws.com/groups/global/AllUsers'
-
For the Authenticated Users group: 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
In this example, a user with VID 3 is granted full control permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2.
response = client.put_object_acl( AccessControlPolicy={ 'Grants': [ { 'Grantee': { 'ID': '3', 'Type': 'CanonicalUser', }, 'Permission': 'FULL_CONTROL' }, ], 'Owner': { 'DisplayName': 'JDoe', 'ID': '2' } }, Bucket='my_bucket', Key='my_object', )
In this example, the predefined AUTHENTICATED_USERS group is granted WRITE permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2..
response = client.put_object_acl( AccessControlPolicy={ 'Grants': [ { 'Grantee': { 'Type': 'Group', 'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers' }, 'Permission': 'WRITE' }, ], 'Owner': { 'DisplayName': 'JDoe', 'ID': '2' } }, Bucket='my_bucket', Key='my_object', )
The get_object_acl() method returns an object's ACL.
To learn about VAST Cluster's support for S3 ACLs, read Managing S3 Access Control Rules (ACLs).
The create_multipart_upload() method initiates a multipart upload and returns an upload ID.
After initiating the multipart upload, you then need to upload all parts and then complete the upload.
Files that accumulate in the audit directory do not expire. {but users only have read access, so therefore...}
The abort_multipart_upload() method aborts a multipart upload after it was initiated.
After a multipart upload is aborted, no additional parts can be uploaded using the upload ID of that multipart upload. The storage consumed by any previously uploaded parts will be freed. However, if any part uploads are currently in progress, those part uploads might or might not succeed. As a result, it might be necessary to abort a given multipart upload multiple times in order to completely free all storage consumed by all parts.
The complete_multipart_upload() method completes a multipart upload by assembling previously uploaded parts.
The upload_part() method uploads a part in a multipart upload that was already initiated.
After uploading all parts, the upload needs to be completed.
The upload_part_copy() method uploads a part of a multipart upload by copying data from an existing object as data source.
The list_parts() method lists the parts that have been uploaded for a specific multipart upload.
Comments
0 comments
Article is closed for comments.