Articles in this section

Using the Boto 3 Client

Sara Levy
  • Updated
Follow

Connecting to the Boto3 Client Interface

To connect to the low-level client interface, use Boto3’s client() method. You must pass your VAST Cluster S3 credentials and other configurations as parameters with hardcoded values. This is the only way to specify a VAST Cluster VIP as the S3 endpoint.

The following example imports the boto module and instantiates a client with the minimum configuration needed for connecting the client to your VAST Cluster S3 account over an HTTP connection:

import boto3
s3_client = boto3.client(
    's3',
     use_ssl=False,
     endpoint_url=<ENDPOINT-URL>    
     aws_access_key_id=<ACCESS-KEY>,
     aws_secret_access_key=<SECRET-KEY>
     region_name=<REGION>
     config=boto3.session.Config(
          signature_version='s3v4'
          s3={'addressing_style': 'path'}
    
     )
)

in which:

  • <ENDPOINT-URL> can be any of the cluster's Virtual IPs, prefixed by http://. For example, http://198.51.100.255, in which 198.51.100.255 is one of the cluster's VIPs.

    Note

    To retrieve the cluster's virtual lPs:

    • In the VAST Web UI, open the menu (click menu_button.png), select Configuration and then select the Virtual IPs tab. The Virtual IPs list shows you which virtual IPs are configured on each CNode.

    • In the VAST CLI, run the vip list command.

  • <ACCESS-KEY> and <SECRET-KEY> are your S3 key pair.

  • <REGION> can be any string. It is required if signature_version=S3v4.

For HTTPS Connection

For an HTTPS connection, pass parameters as follows in the client() call:

  • Enable HTTPS by setting use_ssl=True instead of use_ssl=False .

  • If the default certificate trust store does not recognize the signer of the installed certificate, you can use the verify parameter to specify a non default path to the certificate trust store. If you're using a self signed certificate, you can point this to the certificate itself. For example:

    verify="path/to/client/cert.pem"

  • Alternatively, you can use the verify parameter to disable verification:

    verify=False
 

Create a Bucket

Once you have an instance of the S3 service client, you can call the create_bucket() method on the client instance to create a bucket.

Note

The LocationConstraint(string) parameter can be provided within the CreateBucketConfiguration (dict) parameter to specify a string representing a region where the bucket is created. If you don't specify a region, the region is set to 'vast-1'.

Example

In this example, we create a bucket called mybucket.

response = s3_client.create_bucket(
    Bucket='mybucket'
)
 
 

List All Buckets on the Server

The list_buckets() method returns a list of all buckets owned by the authenticated sender of the request.

Example

response = s3_client.list_buckets()
 
 

List Objects in a Bucket

The list_objects_v2() method returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.

The list_objects() method is a prior version of the same method, supported for backward compatibility.

Example

This example retrieves the list of objects in the bucket "mybucket".

response = s3_client.list_objects_v2(
    Bucket='mybucket',
)
 
 

Determine if bucket exists

The head_bucket() method is used to determine if a bucket exists and if the user has permission to access it.

 

Deleting a Bucket

The delete_bucket() method deletes a bucket. All objects in the bucket must be deleted before the bucket can be deleted.

Example

response = s3_client.delete_bucket(
    Bucket='mybucket'
)
 
 

Set ACL Permissions on a Bucket

Before setting ACL permissions, we recommend you read Managing S3 Access Control Rules (ACLs).

The put_bucket_acl () method sets the permissions on a bucket using access control lists (ACL).

Syntax Notes

To grant permission to a user, specify the grantee with the following parameters:

  • For users on external providers only (AD, LDAP etc) pass:

    • The EmailAddress parameter and provide the user's principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS).

    • The Type parameter and provide AmazonCustomerByEmail as its value.

  • For any users (including users on the local provider), pass:

    • The ID parameter and provide the user's VID as its value.

      Tip

      A VID is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the user query VAST CLI command and specifying udb as the context of the query. The output includes the user's VID.

    • The Type parameter and provide CanonicalUser as its value.

To grant permission to a predefined group, specify Group as the 'Type' and pass the group's URI as the 'URI':

  • For the All Users group: 'http://acs.amazonaws.com/groups/global/AllUsers'

  • For the Authenticated Users group: 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'

 

Examples

In this example, a user with VID 3 is granted full control permission to the bucket my_bucket owned by JDoe whose VID is 2.

response = s3_client.put_bucket_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'ID': '54',
                    'Type': 'CanonicalUser',
                },
                'Permission': 'FULL_CONTROL'
            },
        ],
        'Owner': {
            'DisplayName': 'BSmith',
            'ID': '4'
        }
    },
    Bucket='BobsBucket',
)

In the following example, the Authenticated_Users group is granted READ permission on the bucket BobsBucket.

response = s3_client.put_bucket_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {

                    'Type': 'Group',
                    'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
                },
                'Permission': 'READ'
            },
        ],
        'Owner': {
            'DisplayName': 'BSmith',
            'ID': '4'
        }
    },
    Bucket='BobsBucket',
)
 
 

Return the ACL of a bucket

The get_bucket_acl() method retrieves the ACL of a bucket.

To learn about VAST Cluster's support for S3 ACLs, read Managing S3 Access Control Rules (ACLs).

 

Return the location of a bucket

The get_bucket_location() method is used to return the region in which the bucket resides.

If a region was specified in the CreateBucket() request, it is returned by this method. If no region was specified in the CreateBucket() request, the bucket region was set to 'vast-1', which is returned by this method.

Example

response = s3_client.get_bucket_location(
    Bucket='mybucket',
)
 
 

Creating an Object

The put_object() method adds an object to a bucket.

 

Creating a copy of an object

The copy_object() method creates a copy of an object already stored on the server.

Example

In this example, we copy the object MyObject from the bucket MyBucket to the bucket MyOtherBucket and name the copy MyObjectCopy.

response = S3_client.copy_object(
    Bucket='MyOtherBucket',
    CopySource='MyBucket/MyObject'
    Key='MyObjectCopy',
)
 
 

Retrieving an Object

The get_object() method retrieves an object.

To download a specified range of bytes of an object, use the Range parameter. For more information about the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.

Example

In this example, we download only bytes 32-64 of the object "MyObject" from the bucket "MyBucket".

response = S3_client.get_object(
    Bucket='MyBucket',
    Key='MyObject',
    Range='bytes=32-64',
)
 
 

Retrieve Object Metadata

  s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]

CONFIGFILE

Configuration file. Defaults to $HOME/.s3cfg

BUCKET

Bucket in which object is stored

OBJECT

Object for which to retrieve metadata

Example

Retrieve metadata of file vast.s3cfg.

$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg
s3://bucket1/vast.s3cfg (object):
   File size: 176
   Last mod:  Mon, 24 Jun 2019 19:40:27 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   9de54b7b7a89d9526b32305a331dea6a
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       vastuser: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast                                               data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u                                               name:vastdata
 
 

Deleting an Object

The delete_object() method deletes an object.

 

Deleting Multiple Objects

The delete_objects() method deletes multiple objects in a bucket.

Example

response = s3_client.delete_objects(
    Bucket='mybucket',
    Delete={
        'Objects': [
            {'Key': 'file1'},
            {'Key': 'file2'},
            {'Key': 'file3'},
        ],
    },
)
 
 
 

Set Access Control List (ACL) Permissions on an Object

Before setting ACL permissions, we recommend you read Managing S3 Access Control Rules (ACLs).

The put_object_acl() method sets the permissions on an object using access control lists (ACL).

Syntax Notes

To grant permission to a user, specify the grantee with the following parameters:

  • For users on external providers only (AD, LDAP etc) pass:

    • The EmailAddress parameter and provide the user's principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS).

    • The Type parameter and provide AmazonCustomerByEmail as its value.

  • For any users (including users on the local provider), pass:

    • The ID parameter and provide the user's VID as its value.

      Tip

      A VID is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the user query VAST CLI command and specifying udb as the context of the query. The output includes the user's VID.

    • The Type parameter and provide CanonicalUser as its value.

To grant permission to a predefined group, specify Group as the 'Type' and pass the group's URI as the 'URI':

  • For the All Users group: 'http://acs.amazonaws.com/groups/global/AllUsers'

  • For the Authenticated Users group: 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'

 

Examples

In this example, a user with VID 3 is granted full control permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2.

response = client.put_object_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'ID': '3',
                    'Type': 'CanonicalUser',
                },
                'Permission': 'FULL_CONTROL'
            },
        ],
        'Owner': {
            'DisplayName': 'JDoe',
            'ID': '2'
        }
    },
    Bucket='my_bucket',
    Key='my_object',
)

In this example, the predefined AUTHENTICATED_USERS group is granted WRITE permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2..

response = client.put_object_acl(
    AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'Type': 'Group',
                    'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
                },
                'Permission': 'WRITE'
            },
        ],
        'Owner': {
            'DisplayName': 'JDoe',
            'ID': '2'
        }
    },
    Bucket='my_bucket',
    Key='my_object',
)
 
 

Return the ACL of an object

The get_object_acl() method returns an object's ACL.

To learn about VAST Cluster's support for S3 ACLs, read Managing S3 Access Control Rules (ACLs).

 

Initiate a Multipart Upload

The create_multipart_upload() method initiates a multipart upload and returns an upload ID.

After initiating the multipart upload, you then need to upload all parts and then complete the upload.

 

Managing the Protocol Audit Directory

Files that accumulate in the audit directory do not expire. {but users only have read access, so therefore...}

 

Abort a Multipart Upload

The abort_multipart_upload() method aborts a multipart upload after it was initiated.

After a multipart upload is aborted, no additional parts can be uploaded using the upload ID of that multipart upload. The storage consumed by any previously uploaded parts will be freed. However, if any part uploads are currently in progress, those part uploads might or might not succeed. As a result, it might be necessary to abort a given multipart upload multiple times in order to completely free all storage consumed by all parts.

 

Complete a Multipart Upload

The complete_multipart_upload() method completes a multipart upload by assembling previously uploaded parts.

 

Upload a part in a Multipart Upload

The upload_part() method uploads a part in a multipart upload that was already initiated.

After uploading all parts, the upload needs to be completed.

 

Upload a part by copying data from an existing object as data source

The upload_part_copy() method uploads a part of a multipart upload by copying data from an existing object as data source.

 

List Uploaded Parts for a Specific Multipart Upload

The list_parts() method lists the parts that have been uploaded for a specific multipart upload.

 

List Multipart Uploads

The list_multipart_uploads() method lists all multipart uploads in progress.

 
 

Related articles

Comments

0 comments

Article is closed for comments.