SQUID offers "S3 Data Service" as a data-transfer way using S3 (Simple Storage Sercice) API. You can use this service on SQUID from the running application through the use of S3 API. The data stored via S3 Data Service is on /sqfs/s3/(UUID)/(Bucket name)/(Object name) and so you can access it on the front-end and compute nodes of SQUID.
For the authentication, a pair of access key and secret key is used your id and password. Taking advantage of this mechnism, you can share your data with your collaborators by distributing the access key.
How to use
Your setup and configuration required.
To use S3 API, you have to issue a pair of access key and secret key on the front-end server of SQUID in advance. You can use the s3dskey command on the front end node to manipulate those keys.
Issuance of access key
-
You can make a pair of access key and secret key as follows.
$ s3dskey create --group=(group name)
You can learn what group you are using the groups command.
|
% groups ocean G12345 (In the case of general users, the name of your group is displayed.) % groups hpci hp12345 (In the case or HPCI and JHPCN name, your group name is displayed after hpci.) % groups |
You can see the following output if your creation of accesskey and secret key is successfuly done. You should remember access key and secret key for data transfer .
|
1 2 3 4 5 6 7 8 9 |
+-----------+------------------------------------------+ | accesskey | AKIA7X1KXXXYYYZZZ000 | | enabled | True | | fspaths | None | | fsuid | 60101:10 | | secretkey | **************************************** | | tag | (Serial number):(user name):(group name) | | uuid | ea60f00a60hufaweofapo12813nfawe9506e1216 | +-----------+------------------------------------------+ |
| accesskey | you need this key for the access using S3 API |
|---|---|
| secretkey | a kind of password information (credential information) linked to access key. |
| tag | the inforation on user and group related to the access key. |
| uuid | key ID. This is required for access on file system. |
Manipulation of access key
-
The manipulation of access key is done by feeding a subcommand to the s3dskey command. Please check the command help if you do not know what type of subcommands are available.
| s3dskey list | display the list of access key issued. |
|---|---|
| s3dskey create | create accesskey and secret key pair. |
| s3dskey disable | disable accesskey |
| s3dskey enable | enable accesskey |
| s3dskey reset | reset secretkey |
| s3dskey --help | display command help |
Create Bucket
-
In order to upload / download objects (files) through the S3 API, you need to create an object management unit called a bucket. Use the s3dsbucket command to create a bucket.
$ s3dsbucket create --key=(access key) --bucket=(Bucket name)
--key : Specify the pre-generated access key
--bucket : Specify the bucket name to create
* Specify the bucket name according to the naming convention below.
- more than 3 characters and less than 255 characters
- Only uppercase letters (A-Z), lowercase letters (a-z), numbers (0-9), periods (.), Hyphens (-), and underscores (_) can be specified.
- The bucket name is unique and cannot be duplicated with a bucket name created by yourself or another user.
* Bucket creation is also possible with the PUT BUCKET method of the S3 API. However, it is recommended to create it with the s3dsbucket command. Because the contents operated directly on the front-end cannot be synchronized with the S3 API.
Bucket Operation
-
Bucket operations are performed with subcommands of the s3dsbucket command. The list of subcommands is as follows.
| s3dsbucket list | List of created buckets |
|---|---|
| s3dsbucket create | Create a new bucket |
| s3dsbucket sync | Synchronize files in the file system with objects in the S3 API |
| s3dsbucket delete | Delete bucket * Possible only when there are no objects in the bucket |
| s3dsbucket --help | display command help |
How to do from the source of data transfer.
-
As of today, we have confirmed the follwoing client tools to transfer data from your local PCs an Servers to SQUID using S3 API.
s3curl
S3 Browser
In the following, how to transfer data using s3curl are briefly overviewed.
s3curl example
-
1. You can retrieve s3curl using github as follows.
git clone https://github.com/rtdp/s3curl.git
2. you can make configuration file for s3curl.
vi ~/.s3curl
Please add the following description to your .s3curl file. For username, you can use as you like.
|
1 2 3 4 5 6 7 8 |
%awsSecretAccessKeys = ( username => { id => 'Access Key issued by s3dskey', key => 'Secret Key issued by s3dskey', }, ); push(@endpoints, 'squidgw.hpc.cmc.osaka-u.ac.jp') |
3. you can make a bucket on SQUID as follows.
In the example below, "username-test-bucket" is created. The name of bucket should be identical on the system and so too simple bucket name may be used by other users.
LANG=C
s3curl.pl --id username --createBucket -- -s https://squidgw.hpc.cmc.osaka-u.ac.jp/username-test-bucket
4. upload the file(object) on the bucket just created on SQUID using s3 curl.
s3curl.pl --id username --put test -- -s -v https://squidgw.hpc.cmc.osaka-u.ac.jp/username-test-bucket/test
5. The uploaded (tranfered) file (object) is going to the following place on SQUID>
/sqfs/s3/[uuid issued by s3dskey]/[Bucket name]/
