Using Boto 3 with VAST Cluster S3

In this example, we create a bucket called mybucket.

response = s3_client.create_bucket(
    Bucket='mybucket'
)

response = s3_client.list_buckets()

This example retrieves the list of objects in the bucket "mybucket".

response = s3_client.list_objects(
    Bucket='mybucket',
)

response = s3_client.delete_bucket(
    Bucket='mybucket'
)

To grant permission to a user, specify CanonicalUser as the 'Type' and pass the user's VID as the 'ID'.

To grant permission to a predefined group, specify Group as the 'Type' and pass the group's URI as the 'URI':

In this example, a user with VID 3 is granted full control permission to the bucket my_bucket owned by JDoe whose VID is 2.

response = s3_client.put_bucket_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'ID': '54',
                    'Type': 'CanonicalUser',
                },
                'Permission': 'FULL_CONTROL'
            },
        ],
        'Owner': {
            'DisplayName': 'BSmith',
            'ID': '4'
        }
    },
    Bucket='BobsBucket',
)

In the following example, the Authenticated_Users group is granted READ permission on the bucket BobsBucket.

response = s3_client.put_bucket_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {

                    'Type': 'Group',
                    'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
                },
                'Permission': 'READ'
            },
        ],
        'Owner': {
            'DisplayName': 'BSmith',
            'ID': '4'
        }
    },
    Bucket='BobsBucket',
)

In this example, we download only bytes 32-64 of the object "MyObject" from the bucket "MyBucket".

response = S3_client.get_object(
    Bucket='MyBucket',
    Key='MyObject',
    Range='bytes=32-64',
)

response = s3_client.delete_objects(
    Bucket='mybucket',
    Delete={
        'Objects': [
            {'Key': 'file1'},
            {'Key': 'file2'},
            {'Key': 'file3'},
        ],
    },
)

In this example, a user with VID 3 is granted full control permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2.

response = client.put_object_acl(
      AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'ID': '3',
                    'Type': 'CanonicalUser',
                },
                'Permission': 'FULL_CONTROL'
            },
        ],
        'Owner': {
            'DisplayName': 'JDoe',
            'ID': '2'
        }
    },
    Bucket='my_bucket',
    Key='my_object',
)

In this example, the predefined AUTHENTICATED_USERS group is granted WRITE permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2..

response = client.put_object_acl(
    AccessControlPolicy={
        'Grants': [
            {
                'Grantee': {
                    'Type': 'Group',
                    'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
                },
                'Permission': 'WRITE'
            },
        ],
        'Owner': {
            'DisplayName': 'JDoe',
            'ID': '2'
        }
    },
    Bucket='my_bucket',
    Key='my_object',
)

The following parameters must be passed in the resource() call since the endpoint URL and access key pair must be hardcoded:

If you are passing signature_version=S3v4, then you must pass region_name (string). You can use any string value for the region name.

For an HTTPS connection, pass parameters as follows in the resource() call:

Set use_ssl=False.

In this example, some optional advanced session configurations are passed in. The endpoint URL, S3 user access key and secret key are hard coded, as needed. SSL is disabled, so the connection will be over HTTP.

# Import the boto3 module
import boto3
# Set advanced configuration parameters 
boto_params = {'s3': {'addressing_style': 'path'}, 'signature_version': 's3v4'}
# Create a resource to the Amazon S3 service, passing configurations and credentials as parameters
s3_resource = boto3.resource('s3', use_ssl=False,
                                   endpoint_url=<ENDPOINT-URL>,
                                   aws_access_key_id=<ACCESS-KEY>,
                                   aws_secret_access_key=<SECRET-KEY>,                    
                                   region_name=<REGION>,
                                   config=boto3.session.Config(**self.boto_params),
                                           )

In this example, we iterate through all the buckets on the server. We print out the name of each bucket using the identifier name, which must be set for each bucket.

for bucket in s3_resource.buckets.all():
	print (bucket.name)

In this example, we create a bucket called mybucket as a child resource of the S3 service resource s3_resource.

s3_resource.create_bucket(Bucket="mybucket")

In this example, we instantiate an S3 bucket by its name "the bucket", and iterate through all of the objects in the bucket. We print out the name of each object, using the key identifier, and the last date each object was modified, using the last_modified attribute.

s3_resource.Bucket("the_bucket")
for s3_object in bucket.objects.all():
    print(obj.key, obj.last_modified)

s3_resource.Bucket("the_bucket").delete()

In this example, a user with VID 3 is granted full control permission to the bucket my_bucket owned by JDoe whose VID is 2.

s3_resource.BucketAcl ('my_bucket').put( AccessControlPolicy={
  'Grants': [
        {
           'Grantee': {
                'ID': '3' ,
                'Type': 'CanonicalUser'
      },
          'Permission': 
               'FULL_CONTROL'
         },
   ],
      'Owner': {
        'DisplayName': 'JDoe',
        'ID': '2'
      }
   },
)

In this example, we iterate through the ACL grants and print out the user name and permission of each grantee who has permission to access the bucket.

acl = bucket.Acl()
for grant in acl.grants:
    print(grant['Grantee']['DisplayName'], grant['Permission'])

In this example, we add a file called "object_name" with a body "object data" to the "the_bucket" bucket, specifying a checksum value to store with the object. {is it correct to put that "b" in by "object data"? syntax is Body=b'bytes'|file, - check what that means}

s3_resource.Bucket("the_bucket").put_object(Key="object_name", Body=b"object data", Metadata={ 'checksum': '1234567890'})

In this example, we create a new object called file2 in bucket bucket2 by copying an object called file1 that is stored in bucket bucket1. CopySource is the only required parameter for copy_from(). In this case, we are specifying the source in string format. See copy_from() for full syntax details.

s3_resource.Object('bucket2','file2').copy_from(CopySource='bucket1/file1')

In this example, we retrieve the object "my_object" from the bucket "my_bucket" by calling the get() action on the bucket. The response is a python dictionary, inside which is the 'Body', which is the body data in the format Streamingbody(). We read the content of the object by calling the read() function on the 'Body'.

s3_resource.Object("my_bucket", "my_object").get()['Body'].read()

In this example, we download only the byte range 32-64 of the object "my_object" from the bucket "my_bucket".

s3_resource.Object("my_bucket", "my_object").get(Range='bytes=32-64')['Body'].read()

s3_resource.Object("the_bucket", "object_name").metadata

s3_resource.Object("the_bucket", "object_name").delete()

s3_resource.Bucket("the_bucket").delete_objects(
    Delete={
        'Objects': [
            {'Key': 'file1'},
            {'Key': 'file2'},
            {'Key': 'file3'},
        ],
    },
)

In this example, a user with VID 3 is granted full control permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2.

s3_resource.ObjectAcl('my_bucket', 'my_object').put(AccessControlPolicy = {
      'Grants': [
        {
           'Grantee': {
                'ID': '3' ,
                'Type': 'CanonicalUser'
      },
          'Permission': 
               'FULL_CONTROL'
         },
   ],
      'Owner': {
        'DisplayName': 'JDoe',
        'ID': '2'
      }
   },
)

In this example, the predefined AUTHENTICATED_USERS group is granted WRITE permission to the object my_object in the bucket my_bucket owned by JDoe whose VID is 2..

#attempting group permission example - guessing to specify group name as "displayname" and that the name I used is sufficient and we don't need the long version

s3_resource.ObjectAcl('my_bucket', 'my_object').put(AccessControlPolicy = {
      'Grants': [
        {
       'Grantee':{
       'Type ':'Group'
       'URI': 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
       }, 
      'Permission':'WRITE'
         },
   ],
       'Owner': {
        'DisplayName': 'JDoe',
        'ID': '2'
        }
   },
)

In this example, we iterate through the ACL grants and print out the user name and permission of each grantee who has permission to access the object.

acl = object.Acl() 
for grant in acl.grants: 
    print(grant['Grantee']['DisplayName'], grant['Permission'])