Adam.birds: Created page with "=Amazon S3= ==Setup== Paths are specified as '''remote:bucket''' (or '''remote:''' for the '''lsd''' command.) You may put subdirectories in too, eg '''remote:bucket/path/to..."

2016-04-30T15:01:50Z

Created page with "=Amazon S3= ==Setup== Paths are specified as '''remote:bucket''' (or '''remote:''' for the '''lsd''' command.) You may put subdirectories in too, eg '''remote:bucket/path/to..."

New page

=Amazon S3=

==Setup==

Paths are specified as '''remote:bucket''' (or '''remote:''' for the '''lsd''' command.) You may put subdirectories in too, eg '''remote:bucket/path/to/dir'''.

Here is an example of making an s3 configuration. First run:

<pre>
rclone config
</pre>

This will guide you through an interactive setup process:

<pre>
No remotes found - make a new one
n) New remote
s) Set configuration password
n/s> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Amazon Cloud Drive
\ "amazon cloud drive"
2 / Amazon S3 (also Dreamhost, Ceph)
\ "s3"
3 / Backblaze B2
\ "b2"
4 / Dropbox
\ "dropbox"
5 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
6 / Google Drive
\ "drive"
7 / Hubic
\ "hubic"
8 / Local Disk
\ "local"
9 / Microsoft OneDrive
\ "onedrive"
10 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
11 / Yandex Disk
\ "yandex"
Storage> 2
Get AWS credentials from runtime (environment variables or EC2 meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
1 / Enter AWS credentials in the next step
\ "false"
2 / Get AWS credentials from the environment (env vars or IAM)
\ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> access_key
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> secret_key
Region to connect to.
Choose a number from below, or type in your own value
/ The default endpoint - a good choice if you are unsure.
1 | US Region, Northern Virginia or Pacific Northwest.
| Leave location constraint empty.
\ "us-east-1"
/ US West (Oregon) Region
2 | Needs location constraint us-west-2.
\ "us-west-2"
/ US West (Northern California) Region
3 | Needs location constraint us-west-1.
\ "us-west-1"
/ EU (Ireland) Region Region
4 | Needs location constraint EU or eu-west-1.
\ "eu-west-1"
/ EU (Frankfurt) Region
5 | Needs location constraint eu-central-1.
\ "eu-central-1"
/ Asia Pacific (Singapore) Region
6 | Needs location constraint ap-southeast-1.
\ "ap-southeast-1"
/ Asia Pacific (Sydney) Region
7 | Needs location constraint ap-southeast-2.
\ "ap-southeast-2"
/ Asia Pacific (Tokyo) Region
8 | Needs location constraint ap-northeast-1.
\ "ap-northeast-1"
/ South America (Sao Paulo) Region
9 | Needs location constraint sa-east-1.
\ "sa-east-1"
/ If using an S3 clone that only understands v2 signatures
10 | eg Ceph/Dreamhost
| set this and make sure you set the endpoint.
\ "other-v2-signature"
/ If using an S3 clone that understands v4 signatures set this
11 | and make sure you set the endpoint.
\ "other-v4-signature"
region> 1
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint>
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
1 / Empty for US Region, Northern Virginia or Pacific Northwest.
\ ""
2 / US West (Oregon) Region.
\ "us-west-2"
3 / US West (Northern California) Region.
\ "us-west-1"
4 / EU (Ireland) Region.
\ "eu-west-1"
5 / EU Region.
\ "EU"
6 / Asia Pacific (Singapore) Region.
\ "ap-southeast-1"
7 / Asia Pacific (Sydney) Region.
\ "ap-southeast-2"
8 / Asia Pacific (Tokyo) Region.
\ "ap-northeast-1"
9 / South America (Sao Paulo) Region.
\ "sa-east-1"
location_constraint> 1
Remote config
--------------------
[remote]
env_auth = false
access_key_id = access_key
secret_access_key = secret_key
region = us-east-1
endpoint =
location_constraint =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
</pre>

==Usage==

This remote is called '''remote''' and can now be used like this:

See all buckets

<pre>
rclone lsd remote:
</pre>

Make a new bucket

<pre>
rclone mkdir remote:bucket
</pre>

List the contents of a bucket

<pre>
rclone ls remote:bucket
</pre>

Sync '''/home/local/directory''' to the remote bucket, deleting any excess files in the bucket.

<pre>
rclone sync /home/local/directory remote:bucket
</pre>

==Modified Time==

The modified time is stored as metadata on the object as '''X-Amz-Meta-Mtime''' as floating point since the epoch accurate to 1 ns.

==Multipart uploads==

rclone supports multipart uploads with S3 which means that it can upload files bigger than 5GB. Note that files uploaded with multipart upload don’t have an MD5SUM.

==Buckets and Regions==

With Amazon S3 you can list buckets ('''rclone lsd''') using any region, but you can only access the content of a bucket from the region it was created in. If you attempt to access a bucket from the wrong region, you will get an error, '''incorrect region, the bucket is not in 'XXX' region'''.

==Authentication==

There are two ways to supply '''rclone''' with a set of AWS credentials. In order of precedence:

*Directly in the rclone configuration file (as configured by '''rclone config''')
**set '''access_key_id''' and '''secret_access_key'''
*Runtime configuration:
**set '''env_auth''' to '''true''' in the config file
**Exporting the following environment variables before running '''rclone'''
***Access Key ID: '''AWS_ACCESS_KEY_ID''' or '''AWS_ACCESS_KEY'''
***Secret Access Key: '''AWS_SECRET_ACCESS_KEY''' or '''AWS_SECRET_KEY'''
**Running '''rclone''' on an EC2 instance with an IAM role

If none of these option actually end up providing '''rclone''' with AWS credentials then S3 interaction will be non-authenticated (see below).

==Anonymous access to public buckets==

If you want to use rclone to access a public bucket, configure with a '''blank access_key_id''' and '''secret_access_key'''. Eg

<pre>
No remotes found - make a new one
n) New remote
q) Quit config
n/q> n
name> anons3
What type of source is it?
Choose a number from below
1) amazon cloud drive
2) b2
3) drive
4) dropbox
5) google cloud storage
6) swift
7) hubic
8) local
9) onedrive
10) s3
11) yandex
type> 10
Get AWS credentials from runtime (environment variables or EC2 meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
* Enter AWS credentials in the next step
1) false
* Get AWS credentials from the environment (env vars or IAM)
2) true
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id>
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key>
...
</pre>

Then use it as normal with the name of the public bucket, eg

<pre>
rclone lsd anons3:1000genomes
</pre>

You will be able to list and copy data but not upload it.

==Ceph==

Ceph is an object storage system which presents an Amazon S3 interface.

To use rclone with ceph, you need to set the following parameters in the config.

<pre>
access_key_id = Whatever
secret_access_key = Whatever
endpoint = https://ceph.endpoint.goes.here/
region = other-v2-signature
</pre>

Note also that Ceph sometimes puts '''/''' in the passwords it gives users. If you read the secret access key using the command line tools you will get a JSON blob with the '''/''' escaped as '''\/'''. Make sure you only write '''/''' in the secret access key.

Eg the dump from Ceph looks something like this (irrelevant keys removed).

<pre>
{
"user_id": "xxx",
"display_name": "xxxx",
"keys": [
{
"user": "xxx",
"access_key": "xxxxxx",
"secret_key": "xxxxxx\/xxxx"
}
],
}
</pre>

Because this is a json dump, it is encoding the '''/''' as '''\/''', so if you use the secret key as '''xxxxxx/xxxx''' it will work fine.

[[Category:RClone]]
[[Category:Linux]]
[[Category:Contents]]

RClone - Amazon S3 - Revision history

Adam.birds: Created page with "=Amazon S3= ==Setup== Paths are specified as '''remote:bucket''' (or '''remote:''' for the '''lsd''' command.) You may put subdirectories in too, eg '''remote:bucket/path/to..."