This page explains how to access SQUID's file system EXAScaler or ONION-object with S3 commands and S3API commands using SQUID or AWS CLI(AWS Command Line Interface) installed on your own terminal.
Advance preparation
Obtaining Access Keys
Obtaining Access Keys (SQUID)
Obtaining Access Keys (ONION-object)
Installing AWS CLI
Installing the AWS CLI
AWS CLI initial setting
aws configure --profile=[profie name]
→The message shown as follows will come up:
AWS Access Key ID [None]: (Enter your access key obtained previously.)
AWS Secret Access Key [None]: (Enter your secret key obtained previously.)
Default region name [None]: (For ONION-object, enter "osakau". In the case of SQUID, no setting is required.)
Default output format [None]:No setting required. Please leave blank and press Enter.
The default profile can be set by specifying the following environment variables. If this environment variable is not specified, --profile must be specified for each S3/S3 API command.
[For Windows]
set AWS_DEFAULT_PROFILE=[profile name]
[For Mac Linux]
export AWS_DEFAULT_PROFILE=[profile name]
If you want to check the default profile, please go through the following:
aws configure list
Using S3 Commands
The command is easier than the S3API command. Please use it for simple operations such as creating/deleting buckets and moving/deleting objects.
First of all
For access to SQUID- | --endpoint-url=https://squidgw.hpc.cmc.osaka-u.ac.jp |
---|---|
For access to ONION-object- | --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp |
Create buckets
aws --endpoint-url=[endpoint URL] s3 mb s3://[bucket name to create]
[Example command to create a bucket called "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 mb s3://onion-object-test
※It is possible to create buckets using S3API even on SQUID, but the contents directly manipulated by the file on the front-end server can not be synchronized. Therefore, we recommend you to log in to the frontend and then use the s3dsbucket command to create the bucket.
Copying data
aws --endpoint-url=[endpoint URL] s3 cp s3://(bucket name)/(key)
[Example command to copy a file called "sample" on your local environment to bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp sample s3://onion-object-test
[Example command to copy a file called "sample" on your local environment under the directory "dir" in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp sample s3://onion-object-test/dir/
[Example command to copy a file called "sample" on your local environment under the directory "dir" in bucket "onion-object-test" on ONION-object]
[Example command to copy all files in the current directory under directory "dir" in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp ./ s3://onion-object-test/dir/ --recursive
※By using the *--recursive option, the command is executed for all files in the specified directory.
Listing buckets and objects
aws --endpoint-url=[endpoint URL] s3 ls s3://(bucket name)
[Display bucket list on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 ls
[Display the list of objects in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 ls s3://onion-object-test
About Other S3 Commands
Using High-Level (S3) Commands in AWS CLI
Using S3API Commands
SQUID does not support all S3APIs.
In addition to buckets and object PUT/GET, only some functions such as ACLs, bucket policies, and versioning are supported. ONION-object supports most of S3APIs. Please refer to the following for the supported APIs.
SQUID compatible API ONION-object compatible API
First of all
Please be careful that if you do not, it will not work properly.
For access to SQUID- | --endpoint-url=https://squidgw.hpc.cmc.osaka-u.ac.jp |
---|---|
For access to ONION-object- | --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp |
Upload objects
aws --endpoint-url=[endpoint URL] s3api put-object --bucket (bucket name) --key (key name) --body (file name to upload)
[Upload "sample" to the directory "dir" in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp/ s3api put-object --bucket onion-object-test --key dir/sample --body sample
Obtaining Bucket ACL
aws --endpoint-url=[endpoint URL] s3api get-bucket-acl --bucket (bucket name)
[Get ACL of bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp/ s3api get-bucket-acl --bucket onion-object-test
About Other S3API Commands
Using API-Level (s3api) Commands in the AWS CLI