This page explains how to access SQUID's file system EXAScaler or ONION-object with S3 commands and S3API commands using SQUID or AWS CLI(AWS Command Line Interface) installed on your own terminal.

 

Advance preparation

Obtaining Access Keys

Obtain the access key and secret key from the S3-compatible storage (SQUID's file system or ONION-object) that you want to access with the AWS CLI.
Obtaining Access Keys (SQUID)
Obtaining Access Keys (ONION-object)

 

Installing AWS CLI

Please install AWS CLI based on the following page (AWS official documentation page). It is already installed on SQUID's front-end servers.
Installing the AWS CLI

 

AWS CLI initial setting

Use the following command for initially setting for accessing to SQUID and Onion-object: When using both SQUID and ONION-object, specify a separate [profile name] for each and go through the initial setting.

aws configure --profile=[profie name]
 

→The message shown as follows will come up:
 AWS Access Key ID [None]: (Enter your access key obtained previously.)
 AWS Secret Access Key [None]: (Enter your secret key obtained previously.)
 Default region name [None]: (For ONION-object, enter "osakau". In the case of SQUID, no setting is required.)
 Default output format [None]:No setting required. Please leave blank and press Enter.

 

The default profile can be set by specifying the following environment variables. If this environment variable is not specified, --profile must be specified for each S3/S3 API command.

[For Windows]
set AWS_DEFAULT_PROFILE=[profile name]
[For Mac Linux]
export AWS_DEFAULT_PROFILE=[profile name]

 

If you want to check the default profile, please go through the following:

aws configure list

 

Using S3 Commands

Use the AWS CLI's S3 commands to manipulate buckets and objects.
The command is easier than the S3API command. Please use it for simple operations such as creating/deleting buckets and moving/deleting objects.
 

First of all

Because the AWS CLI is designed and targeted for accessing Amazon's cloud resources, you must specify the "--endpoint-url" option each time you want to access SQUID or ONION-object. If you have not specified AWS_DEFAULT_PROFILE, please specify"--profile" too. Please be careful that if you do not, it will not work properly.

For access to SQUID- --endpoint-url=https://squidgw.hpc.cmc.osaka-u.ac.jp
For access to ONION-object- --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp

 

Create buckets

Use the following command to create a new bucket:

aws --endpoint-url=[endpoint URL] s3 mb s3://[bucket name to create]
 

[Example command to create a bucket called "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 mb s3://onion-object-test

※It is possible to create buckets using S3API even on SQUID, but the contents directly manipulated by the file on the front-end server can not be synchronized. Therefore, we recommend you to log in to the frontend and then use the s3dsbucket command to create the bucket.

 

Copying data

Copy the file from your local environment to SQUID/ONION-object with the following command.

aws --endpoint-url=[endpoint URL] s3 cp s3://(bucket name)/(key)
 

[Example command to copy a file called "sample" on your local environment to bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp sample s3://onion-object-test
[Example command to copy a file called "sample" on your local environment under the directory "dir" in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp sample s3://onion-object-test/dir/
[Example command to copy a file called "sample" on your local environment under the directory "dir" in bucket "onion-object-test" on ONION-object]
[Example command to copy all files in the current directory under directory "dir" in bucket "onion-object-test" on ONION-object]

aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 cp ./ s3://onion-object-test/dir/ --recursive
※By using the *--recursive option, the command is executed for all files in the specified directory.

 

Listing buckets and objects

The following command displays a list of buckets and objects on SQUID/ONION-object.

aws --endpoint-url=[endpoint URL] s3 ls s3://(bucket name)
 

[Display bucket list on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 ls
[Display the list of objects in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp s3 ls s3://onion-object-test

 

About Other S3 Commands

Please refer to the following AWS official web page.
Using High-Level (S3) Commands in AWS CLI

 

Using S3API Commands

Use the AWS CLI's S3API commands to manipulate buckets and objects.
SQUID does not support all S3APIs.
In addition to buckets and object PUT/GET, only some functions such as ACLs, bucket policies, and versioning are supported. ONION-object supports most of S3APIs. Please refer to the following for the supported APIs.
SQUID compatible API  ONION-object compatible API
 

First of all

Because the AWS CLI is a command for accessing Amazon's cloud resources, you must specify the "--endpoint-url" option each time you want to access SQUID or ONION-object. If you have not set AWS_DEFAULT_PROFILE, please specify "--profile" too.
Please be careful that if you do not, it will not work properly.

For access to SQUID- --endpoint-url=https://squidgw.hpc.cmc.osaka-u.ac.jp
For access to ONION-object- --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp

 

Upload objects

Upload data to any bucket with S3API PUT-Object.

aws --endpoint-url=[endpoint URL] s3api put-object --bucket (bucket name) --key (key name) --body (file name to upload)
 

[Upload "sample" to the directory "dir" in bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp/ s3api put-object --bucket onion-object-test --key dir/sample --body sample

 

Obtaining Bucket ACL

Gets the Access Control List (ACL) of the bucket with the S3API Get-Bucket-ACL.

aws --endpoint-url=[endpoint URL] s3api get-bucket-acl --bucket (bucket name)
 

[Get ACL of bucket "onion-object-test" on ONION-object]
aws --endpoint-url=https://s3-osakau.oniongw.hpc.cmc.osaka-u.ac.jp/ s3api get-bucket-acl --bucket onion-object-test

 

About Other S3API Commands

Please refer to the following AWS official web page.
Using API-Level (s3api) Commands in the AWS CLI