Note
Documentation for S3cmd can be found on s3cmd's website. Please use the S3cmd documentation and inline command help as needed in conjunction with our specific guidelines for using S3cmd with VAST Cluster.
S3cmd is a command line client for uploading, retrieving and managing data with the S3 service.
The sections below describe how to install and configure S3cmd for use with VAST Cluster and how to use it to perform supported tasks.
Browse to https://s3tools.org/s3cmd for system requirements and download.
Make sure the s3cmd is installed on your client machine. For example, on CentOS/Redhat Linux systems:
sudo yum install s3cmd -y
Once s3cmd is installed, you need to configure it to connect to your VAST Cluster S3 account.
Configurations, including user account credentials, are specified in a configuration file. The default configuration file is located under your home directory ($HOME/.s3cfg). This file contains all of the S3cmd settings. All s3cmd commands default to the default configuration file.
You can create an additional configuration file for your VAST-specific settings, or add them to the default configuration file. When using an additional configuration file, specify the file in your S3cmd commands.
Your configuration file should minimally include the following content. Save it to your home directory as <filename>.s3cfg, where <filename> is a name of your choice. In our examples. we will use vast.s3cfg.
[default] access_key = <ACCESS-KEY> secret_key = <SECRET-KEY> host_base = <CLUSTER VIRTUAL IP> host_bucket = <CLUSTER VIRTUAL IP> use_https = false|true
In which:
-
<ACCESS-KEY>
and<SECRET-KEY>
are your S3 key pair. -
<CLUSTER VIRTUAL IP>
can be any of the cluster's Virtual IPs.Note
To retrieve the cluster's virtual lPs:
-
In the VAST Web UI, open the menu (click
), select Configuration and then select the Virtual IPs tab. The Virtual IPs list shows you which virtual IPs are configured on each CNode.
-
In the VAST CLI, run the
vip list
command.
Caution
Do not specify a port number. The port is not configurable. Specifying the default port can cause an error with signature version 4.
-
-
To connect over HTTPS, set
use_https= true
. When this is set, the service will listen on port 443, the secure port. HTTPS connection will only work if an SSL certificate for S3 is installed on the VMS. See Installing an SSL Certificate. -
To connect over HTTP, set
use_https= false
.
For more configuration options, see s3cmd documentation. Configuration options can be passed in individual commands or in the configuration file.
For HTTPS ConnectionsIf you are configuring S3cmd to connect to S3 over HTTPS, you need to make sure that the connection doesn't fail due to failed certificate verification. An SSL certificate must be installed on the VMS for the S3 service. The S3 service provides the installed certificate to S3cmd as part of the SSL handshake when establishing an HTTPS connection to the service. By default, S3cmd verifies the certificate using the default local certificate trust store. The local path to the default certificate store is OS dependent. If the default trust store doesn't recognize the CA that signed the certificate, such as with a self signed certificate, the connection will fail with the default configuration.
The following are commands you might include in the configuration file to handle SSL certificate verification:
-
To disable certificate verification:
no-check-certificate
-
Alternatively, to set a non-default path to the certificate trust store and point it to your certificate:
ca-certs = <CA_CERTS_FILE>
-
Often there is a mismatch between the hostname embedded in the certificate and the service endpoint URL, which causes failure when the hostname is verified. To disable checking the hostname:
no-check-hostname
VAST Cluster supports both signature version 4 (default) and signature version 2. To enforce signature version 2 specify the following in the configuration file:
signature_v2=true
s3cmd [-c CONFIGFILE] mb s3://BUCKET
Create a bucket named bucket1 specifying our configuration file as the non default vast.s3cfg.
$ s3cmd -c vast.s3cfg mb s3://bucket1 Bucket 's3://bucket1/' created
s3cmd [-c CONFIGFILE] ls s3://
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
List all buckets on the server, specifying the root path of all buckets (s3://) and specifying our configuration file as the non default vast.s3cfg.
$ s3cmd -c vast.s3cfg ls s3:// 2019-06-24 09:55 s3://bucket1
s3cmd [-c CONFIGFILE] ls [s3://BUCKET[/PREFIX]]}
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Name of bucket |
PREFIX |
Filter objects by prefix. Although the S3 structure is flat consisting of buckets containing objects with no hierarchy of sub-buckets and sub-folders, object keys can include prefixes separated by '/' as a delimiter. Hierarchy can be inferred from the prefixes. |
List objects in the bucket bucket1.
$ s3cmd -c vast.s3cfg ls s3://bucket1 2019-06-24 19:51 205 s3://bucket1/blue.txt 2019-06-24 19:52 93 s3://bucket1/brown.txt 2019-06-24 19:40 176 s3://bucket1/vast.s3cfg
$ s3cmd [-c CONFIGFILE] info s3://BUCKET
CONFIGFILE |
The name of your VAST configuration file |
BUCKET |
The name of the bucket |
Retrieve the metadata of the bucket bucket1.
$ s3cmd -c vast.s3cfg info s3://bucket1 s3://bucket1/ (bucket): Location: vast-1 Payer: none Expiration Rule: none Policy: none CORS: none ACL: vastuser: FULL_CONTROL
s3cmd [-c CONFIGFILE] rb s3://BUCKET
Note
This command can be used with the --recursive
flag which removes the bucket with all of its contents. This form of the command calls the DeleteObjects AWS S3 API which is supported from VAST Cluster 3.2.0.
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Name of bucket |
Before setting ACL permissions, we recommend you read S3 Access Control Lists (ACLs).
s3cmd [-c CONFIGFILE] setacl s3://BUCKET[/OBJECT] [--acl-grant=PERMISSION:{USER_VAST_ID|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}]
CONFIGFILE |
Configuration file. Defaults to $HOME/s3cf |
BUCKET |
Name of bucket |
--acl-grant |
Grant stated permission for stated user or group. |
--acl-revoke |
Revoke stated permission for stated user or group |
PERMISSION |
Type of permission to grant:
|
USER_VAST_ID |
User's VAST ID. This is VAST Cluster's unique ID per user. The VAST ID is displayed as 'ID' in the output of the VAST CLI command |
GROUP |
Two pre-defined groups are supported: 'http://acs.amazonaws.com/groups/global/AllUsers' 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers' |
Grant full control to the predefined group AuthenticatedUsers to access the bucket mybucket.
$ s3cmd setacl s3://mybucket --acl-grant='full_control:http://acs.amazonaws.com/groups/global/AuthenticatedUsers'
Grant read permission to the predefined AllUsers group to access the bucket mybucket.
$ s3cmd setacl s3://mybucket --acl-grant='read:http://acs.amazonaws.com/groups/global/AllUsers'
Grant user with VAST ID 14 write permission to the bucket mybucket:
$ s3cmd setacl s3://mybucket --acl-grant='write:14
s3cmd [-c CONFIGFILE] info s3://BUCKET
Retrieve all metadata of the bucket bucket1. The output includes the bucket's ACL.
$ s3cmd -c vast.s3cfg info s3://bucket1 s3://bucket1/ (bucket): Location: vast-1 Payer: none Expiration Rule: none Policy: none CORS: none ACL: vastuser: FULL_CONTROL
s3cmd [-c CONFIGFILE] [--disable-multipart] put FILE [FILE...] s3://BUCKET[/PREFIX]
CONFIGFILE |
|
--disable-multipart |
Disables multipart upload for files over 15 MB (which by default are loaded as multipart uploads) |
FILE |
Name of local file to upload |
BUCKET |
|
PREFIX |
Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by '/'. Object listing can be filtered by prefixes and delimiters. See List Objects in a Bucket. For example, when uploading a file called file.asp, if you specify a prefix /dev/misc the complete S3 object key becomes dev/misc/file.asp. A file system hierarchy can be inferred consisting of a folder called misc nested under a folder called dev in which the file file.asp is contained. |
Add file blue.txt to bucket bucket1.
$ s3cmd -c vast.s3cfg put blue.txt s3://bucket1 upload: 'blue.txt' -> 's3://bucket1/blue.txt' [1 of 1] 205 of 205 100% in 0s 4.33 kB/s done
s3cmd [-c CONFIGFILE] cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET1 |
Bucket from which to copy object |
OBJECT1 |
Source object to copy |
BUCKET2 |
Bucket in which to create new object |
OBJECT2 |
Name of destination object, if different from source object |
Create copy of file.zip, which is stored in bucket mybucket, store the copy in the same bucket and call it copyoffile.zip:
$ s3cmd -c vast.s3cfg cp s3://mybucket/file.zip s3://mybucket/copyoffile.zip
s3cmd [-c CONFIGFILE] get s3://BUCKET/OBJECT LOCAL_FILE
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Bucket from which to retrieve stored object |
OBJECT |
Object to download |
LOCAL_FILE |
The local file path to download to |
Download the file file.zip from the bucket bucket.zip.
$ s3cmd -c vast.s3cfg get s3://mybucket/file.zip file.zip
s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Bucket in which object is stored |
OBJECT |
Object for which to retrieve metadata |
Retrieve metadata of file vast.s3cfg.
$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg s3://bucket1/vast.s3cfg (object): File size: 176 Last mod: Mon, 24 Jun 2019 19:40:27 GMT MIME type: text/plain Storage: STANDARD MD5 sum: 9de54b7b7a89d9526b32305a331dea6a SSE: none Policy: none CORS: none ACL: vastuser: FULL_CONTROL x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u name:vastdata
s3cmd [-c CONFIGFILE] del s3://BUCKET/OBJECT
or (alias for del)
s3cmd [-c CONFIGFILE] rm s3://BUCKET/OBJECT
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Name of bucket |
OBJECT |
Name of object to delete |
Delete the file brown.txt from the bucket bucket1.
$ s3cmd -c vast.s3cfg rm s3://bucket1/brown.txt delete: 's3://bucket1/brown.txt'
Before setting ACL permissions, we recommend you read S3 Access Control Lists (ACLs).
s3cmd [-c CONFIGFILE] setacl s3://BUCKET/OBJECT [--acl-grant=PERMISSION:{USER_VAST_ID|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}]
CONFIGFILE |
|
BUCKET |
|
OBJECT |
Name of object |
--acl-grant |
Grant stated permission for stated user or group. |
--acl-revoke |
Revoke stated permission for stated user or group |
PERMISSION |
Type of permission to grant:
|
USER_VAST_ID |
User's VAST ID. This is VAST Cluster's unique ID per user. The VAST ID is displayed as 'ID' in the output of the VAST CLI command |
GROUP |
Two pre-defined groups are supported: 'http://acs.amazonaws.com/groups/global/AllUsers' 'http://acs.amazonaws.com/groups/global/AuthenticatedUsers' |
Grant write permission to object myfile.asp in bucket mybucket to the user whose VID is 6.
$ s3cmd -c vast.s3cfg setacl s3://mybucket/myfile.asp --acl-grant=write:6
s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Name of bucket in which object is stored |
OBJECT |
Name of object for which to return ACL |
Get ACL of file vast.s3cfg as part of full metadata output.
$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg s3://bucket1/vast.s3cfg (object): File size: 176 Last mod: Mon, 24 Jun 2019 19:40:27 GMT MIME type: text/plain Storage: STANDARD MD5 sum: 9de54b7b7a89d9526b32305a331dea6a SSE: none Policy: none CORS: none ACL: vastuser: FULL_CONTROL x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u name:vastdata
Multipart uploads are automatically used when a file to upload is larger than 15MB. The file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is uploaded separately. The file is reconstructed at the destination when the transfer is completed. If an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded.
s3cmd [-c CONFIGFILE] put [--multipart-chunk-size-mb=SIZE] FILE s3://BUCKET[/PREFIX]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
SIZE |
Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded-multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB. |
FILE |
A file over 15 MB in size. (If smaller, the file will be uploaded without being split into parts.) |
BUCKET |
The bucket to which to upload the file. |
PREFIX |
Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by '/'. Object listing can be filtered by prefixes and delimiters. See List Objects in a Bucket. For example, when uploading a file called file.asp, if you specify a prefix /dev/misc the complete S3 object key becomes dev/misc/file.asp. A file system hierarchy can be inferred consisting of a folder called misc nested under a folder called dev in which the file file.asp is contained. |
Start uploading the large file Big.zip in parts of 500 MB:
$ s3cmd -c vast.s3cfg put --multipart-chunk-size-mb=500 Big.zip s3://mybucket/ upload: 'Big.zip' > 's3://mybucket/Big.zip' [part 1 of 16, 500MB] [1 of 1]
After a multipart upload is aborted, no additional parts can be uploaded using the upload ID of the multipart upload. The storage consumed by any previously uploaded parts will be freed. To verify that all parts have been removed, see List Uploaded Parts for a Specific Multipart Upload.
s3cmd [-c CONFIGFILE] abortmp s3://BUCKET/OBJECT ID
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
Name of bucket to which object is being uploaded |
OBJECT |
Name of object which started uploading as multipart upload |
ID |
ID of the multipart upload |
If we kill the upload process mid-upload (ctrl-C), the upload ID is provided:
upload: 'Big.zip' -> 's3://mybucket/Big.zip' [part 5 of 16, 500MB] [1 of 1] 13041664 of 524288000 2% in 109s 116.48 kB/s^CERROR: Upload of 'Big.zip' part 5 failed. Use s3cmd abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM to abort the upload, or s3cmd --upload-id Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put ... to continue the upload.
Now we can use this upload ID to abort the upload:
$ s3cmd -c vast.s3cfg abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y78LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM
s3cmd [-c CONFIGFILE] listmp s3://BUCKET/OBJECT ID
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
The bucket the object was uploaded to |
OBJECT |
The name of the object being uploaded |
ID |
The ID of the multipart upload |
List uploaded parts for specific initiated upload of Big.zip to mybucket.
$ s3cmd -c vast.s3cfg listmp s3://mybucket/Big.zip Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM
s3cmd [-c CONFIGFILE] multipart s3://BUCKET [ID]
CONFIGFILE |
Configuration file. Defaults to $HOME/.s3cfg |
BUCKET |
The name of a bucket (multipart uploads to the specified bucket are listed) |
ID |
The ID of a specific multipart upload (for filtering list) |
To list all current multipart uploads to bucket mybucket in progress
$ s3cmd -c vast.s3cfg multipart s3://mybucket
Comments
0 comments
Article is closed for comments.