SQUID offers "S3 Data Service" as a data-transfer way using S3 (Simple Storage Sercice) API. You can use this service on SQUID from the running application through the use of S3 API. The data stored via S3 Data Service is on /sqfs/s3/(UUID)/(Bucket name)/(Object name) and so you can access it on the front-end and compute nodes of SQUID.

For the authentication, a pair of access key and secret key is used your id and password. Taking advantage of this mechnism, you can share your data with your collaborators by distributing the access key.

How to use

    Your setup and configuration required.

      To use S3 API, you have to issue a pair of access key and secret key on the front-end server of SQUID in advance. You can use the s3dskey command on the front end node to manipulate those keys.

      Issuance of access key

        You can make a pair of access key and secret key as follows.

        $ s3dskey create --group=(group name)

        You can learn what group you are using the groups command.

        Example of the groups command

        % groups
        ocean G12345
        (In the case of general users, the name of your group is displayed.)
        % groups
        hpci hp12345
        (In the case or HPCI and JHPCN name, your group name is displayed after hpci.)

        % groups
        hpci hp12345 jh67890 G34567
        (In the case that you belongs to multiple groups and research projects, all of groups are displayed.)

        You can see the following output if your creation of accesskey and secret key is successfuly done. You should remember access key and secret key for data transfer .

        accesskey you need this key for the access using S3 API
        secretkey a kind of password information (credential information) linked to access key.
        tag the inforation on user and group related to the access key.
        uuid key ID. This is required for access on file system.


      Manipulation of access key

        The manipulation of access key is done by feeding a subcommand to the s3dskey command. Please check the command help if you do not know what type of subcommands are available.

        s3dskey list display the list of access key issued.
        s3dskey create create accesskey and secret key pair.
        s3dskey disable disable accesskey
        s3dskey enable enable accesskey
        s3dskey reset reset secretkey
        s3dskey --help display command help


      Create Bucket

        In order to upload / download objects (files) through the S3 API, you need to create an object management unit called a bucket. Use the s3dsbucket command to create a bucket.

        $ s3dsbucket create --key=(access key) --bucket=(Bucket name)

        --key : Specify the pre-generated access key
        --bucket : Specify the bucket name to create

        * Specify the bucket name according to the naming convention below.
        - more than 3 characters and less than 255 characters
        - Only uppercase letters (A-Z), lowercase letters (a-z), numbers (0-9), periods (.), Hyphens (-), and underscores (_) can be specified.
        - The bucket name is unique and cannot be duplicated with a bucket name created by yourself or another user.

        * Bucket creation is also possible with the PUT BUCKET method of the S3 API. However, it is recommended to create it with the s3dsbucket command. Because the contents operated directly on the front-end cannot be synchronized with the S3 API.


      Bucket Operation

        Bucket operations are performed with subcommands of the s3dsbucket command. The list of subcommands is as follows.

        s3dsbucket list List of created buckets
        s3dsbucket create Create a new bucket
        s3dsbucket sync Synchronize files in the file system with objects in the S3 API
        s3dsbucket delete Delete bucket * Possible only when there are no objects in the bucket
        s3dsbucket --help display command help

    How to do from the source of data transfer.

      As of today, we have confirmed the follwoing client tools to transfer data from your local PCs an Servers to SQUID using S3 API.
      S3 Browser
      In the following, how to transfer data using s3curl are briefly overviewed.


      s3curl example

        1. You can retrieve s3curl using github as follows.

        git clone https://github.com/rtdp/s3curl.git

        2. you can make configuration file for s3curl.

        vi ~/.s3curl

        Please add the following description to your .s3curl file. For username, you can use as you like.

        3. you can make a bucket on SQUID as follows.
        In the example below, "username-test-bucket" is created. The name of bucket should be identical on the system and so too simple bucket name may be used by other users.

        s3curl.pl --id username --createBucket -- -s https://squidgw.hpc.cmc.osaka-u.ac.jp/username-test-bucket

        4. upload the file(object) on the bucket just created on SQUID using s3 curl.

        s3curl.pl --id username --put test -- -s -v https://squidgw.hpc.cmc.osaka-u.ac.jp/username-test-bucket/test

        5. The uploaded (tranfered) file (object) is going to the following place on SQUID>

        /sqfs/s3/[uuid issued by s3dskey]/[Bucket name]/