overdue
This commit is contained in:
bel
311
.rclone_repo/docs/content/swift.md
Executable file
311
.rclone_repo/docs/content/swift.md
Executable file
@@ -0,0 +1,311 @@
|
||||
---
|
||||
title: "Swift"
|
||||
description: "Swift"
|
||||
date: "2014-04-26"
|
||||
---
|
||||
|
||||
<i class="fa fa-space-shuttle"></i>Swift
|
||||
----------------------------------------
|
||||
|
||||
Swift refers to [Openstack Object Storage](https://docs.openstack.org/swift/latest/).
|
||||
Commercial implementations of that being:
|
||||
|
||||
* [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/)
|
||||
* [Memset Memstore](https://www.memset.com/cloud/storage/)
|
||||
* [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/)
|
||||
* [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc)
|
||||
* [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html)
|
||||
|
||||
Paths are specified as `remote:container` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, eg `remote:container/path/to/dir`.
|
||||
|
||||
Here is an example of making a swift configuration. First run
|
||||
|
||||
rclone config
|
||||
|
||||
This will guide you through an interactive setup process.
|
||||
|
||||
```
|
||||
No remotes found - make a new one
|
||||
n) New remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
n/s/q> n
|
||||
name> remote
|
||||
Type of storage to configure.
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Amazon Drive
|
||||
\ "amazon cloud drive"
|
||||
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
|
||||
\ "s3"
|
||||
3 / Backblaze B2
|
||||
\ "b2"
|
||||
4 / Box
|
||||
\ "box"
|
||||
5 / Cache a remote
|
||||
\ "cache"
|
||||
6 / Dropbox
|
||||
\ "dropbox"
|
||||
7 / Encrypt/Decrypt a remote
|
||||
\ "crypt"
|
||||
8 / FTP Connection
|
||||
\ "ftp"
|
||||
9 / Google Cloud Storage (this is not Google Drive)
|
||||
\ "google cloud storage"
|
||||
10 / Google Drive
|
||||
\ "drive"
|
||||
11 / Hubic
|
||||
\ "hubic"
|
||||
12 / Local Disk
|
||||
\ "local"
|
||||
13 / Microsoft Azure Blob Storage
|
||||
\ "azureblob"
|
||||
14 / Microsoft OneDrive
|
||||
\ "onedrive"
|
||||
15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
||||
\ "swift"
|
||||
16 / Pcloud
|
||||
\ "pcloud"
|
||||
17 / QingCloud Object Storage
|
||||
\ "qingstor"
|
||||
18 / SSH/SFTP Connection
|
||||
\ "sftp"
|
||||
19 / Webdav
|
||||
\ "webdav"
|
||||
20 / Yandex Disk
|
||||
\ "yandex"
|
||||
21 / http Connection
|
||||
\ "http"
|
||||
Storage> swift
|
||||
Get swift credentials from environment variables in standard OpenStack form.
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Enter swift credentials in the next step
|
||||
\ "false"
|
||||
2 / Get swift credentials from environment vars. Leave other fields blank if using this.
|
||||
\ "true"
|
||||
env_auth> true
|
||||
User name to log in (OS_USERNAME).
|
||||
user>
|
||||
API key or password (OS_PASSWORD).
|
||||
key>
|
||||
Authentication URL for server (OS_AUTH_URL).
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Rackspace US
|
||||
\ "https://auth.api.rackspacecloud.com/v1.0"
|
||||
2 / Rackspace UK
|
||||
\ "https://lon.auth.api.rackspacecloud.com/v1.0"
|
||||
3 / Rackspace v2
|
||||
\ "https://identity.api.rackspacecloud.com/v2.0"
|
||||
4 / Memset Memstore UK
|
||||
\ "https://auth.storage.memset.com/v1.0"
|
||||
5 / Memset Memstore UK v2
|
||||
\ "https://auth.storage.memset.com/v2.0"
|
||||
6 / OVH
|
||||
\ "https://auth.cloud.ovh.net/v2.0"
|
||||
auth>
|
||||
User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).
|
||||
user_id>
|
||||
User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
|
||||
domain>
|
||||
Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
|
||||
tenant>
|
||||
Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
|
||||
tenant_id>
|
||||
Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
|
||||
tenant_domain>
|
||||
Region name - optional (OS_REGION_NAME)
|
||||
region>
|
||||
Storage URL - optional (OS_STORAGE_URL)
|
||||
storage_url>
|
||||
Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
|
||||
auth_token>
|
||||
AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
|
||||
auth_version>
|
||||
Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE)
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Public (default, choose this if not sure)
|
||||
\ "public"
|
||||
2 / Internal (use internal service net)
|
||||
\ "internal"
|
||||
3 / Admin
|
||||
\ "admin"
|
||||
endpoint_type>
|
||||
Remote config
|
||||
--------------------
|
||||
[test]
|
||||
env_auth = true
|
||||
user =
|
||||
key =
|
||||
auth =
|
||||
user_id =
|
||||
domain =
|
||||
tenant =
|
||||
tenant_id =
|
||||
tenant_domain =
|
||||
region =
|
||||
storage_url =
|
||||
auth_token =
|
||||
auth_version =
|
||||
endpoint_type =
|
||||
--------------------
|
||||
y) Yes this is OK
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
```
|
||||
|
||||
This remote is called `remote` and can now be used like this
|
||||
|
||||
See all containers
|
||||
|
||||
rclone lsd remote:
|
||||
|
||||
Make a new container
|
||||
|
||||
rclone mkdir remote:container
|
||||
|
||||
List the contents of a container
|
||||
|
||||
rclone ls remote:container
|
||||
|
||||
Sync `/home/local/directory` to the remote container, deleting any
|
||||
excess files in the container.
|
||||
|
||||
rclone sync /home/local/directory remote:container
|
||||
|
||||
### Configuration from an Openstack credentials file ###
|
||||
|
||||
An Opentstack credentials file typically looks something something
|
||||
like this (without the comments)
|
||||
|
||||
```
|
||||
export OS_AUTH_URL=https://a.provider.net/v2.0
|
||||
export OS_TENANT_ID=ffffffffffffffffffffffffffffffff
|
||||
export OS_TENANT_NAME="1234567890123456"
|
||||
export OS_USERNAME="123abc567xy"
|
||||
echo "Please enter your OpenStack Password: "
|
||||
read -sr OS_PASSWORD_INPUT
|
||||
export OS_PASSWORD=$OS_PASSWORD_INPUT
|
||||
export OS_REGION_NAME="SBG1"
|
||||
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
|
||||
```
|
||||
|
||||
The config file needs to look something like this where `$OS_USERNAME`
|
||||
represents the value of the `OS_USERNAME` variable - `123abc567xy` in
|
||||
the example above.
|
||||
|
||||
```
|
||||
[remote]
|
||||
type = swift
|
||||
user = $OS_USERNAME
|
||||
key = $OS_PASSWORD
|
||||
auth = $OS_AUTH_URL
|
||||
tenant = $OS_TENANT_NAME
|
||||
```
|
||||
|
||||
Note that you may (or may not) need to set `region` too - try without first.
|
||||
|
||||
### Configuration from the environment ###
|
||||
|
||||
If you prefer you can configure rclone to use swift using a standard
|
||||
set of OpenStack environment variables.
|
||||
|
||||
When you run through the config, make sure you choose `true` for
|
||||
`env_auth` and leave everything else blank.
|
||||
|
||||
rclone will then set any empty config parameters from the environment
|
||||
using standard OpenStack environment variables. There is [a list of
|
||||
the
|
||||
variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment)
|
||||
in the docs for the swift library.
|
||||
|
||||
### Using an alternate authentication method ###
|
||||
|
||||
If your OpenStack installation uses a non-standard authentication method
|
||||
that might not be yet supported by rclone or the underlying swift library,
|
||||
you can authenticate externally (e.g. calling manually the `openstack`
|
||||
commands to get a token). Then, you just need to pass the two
|
||||
configuration variables ``auth_token`` and ``storage_url``.
|
||||
If they are both provided, the other variables are ignored. rclone will
|
||||
not try to authenticate but instead assume it is already authenticated
|
||||
and use these two variables to access the OpenStack installation.
|
||||
|
||||
#### Using rclone without a config file ####
|
||||
|
||||
You can use rclone with swift without a config file, if desired, like
|
||||
this:
|
||||
|
||||
```
|
||||
source openstack-credentials-file
|
||||
export RCLONE_CONFIG_MYREMOTE_TYPE=swift
|
||||
export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
|
||||
rclone lsd myremote:
|
||||
```
|
||||
|
||||
### --fast-list ###
|
||||
|
||||
This remote supports `--fast-list` which allows you to use fewer
|
||||
transactions in exchange for more memory. See the [rclone
|
||||
docs](/docs/#fast-list) for more details.
|
||||
|
||||
### --update and --use-server-modtime ###
|
||||
|
||||
As noted below, the modified time is stored on metadata on the object. It is
|
||||
used by default for all operations that require checking the time a file was
|
||||
last updated. It allows rclone to treat the remote more like a true filesystem,
|
||||
but it is inefficient because it requires an extra API call to retrieve the
|
||||
metadata.
|
||||
|
||||
For many operations, the time the object was last uploaded to the remote is
|
||||
sufficient to determine if it is "dirty". By using `--update` along with
|
||||
`--use-server-modtime`, you can avoid the extra API call and simply upload
|
||||
files whose local modtime is newer than the time it was last uploaded.
|
||||
|
||||
### Specific options ###
|
||||
|
||||
Here are the command line options specific to this cloud storage
|
||||
system.
|
||||
|
||||
#### --swift-storage-policy=STRING ####
|
||||
Apply the specified storage policy when creating a new container. The policy
|
||||
cannot be changed afterwards. The allowed configuration values and their
|
||||
meaning depend on your Swift storage provider.
|
||||
|
||||
#### --swift-chunk-size=SIZE ####
|
||||
|
||||
Above this size files will be chunked into a _segments container. The
|
||||
default for this is 5GB which is its maximum value.
|
||||
|
||||
### Modified time ###
|
||||
|
||||
The modified time is stored as metadata on the object as
|
||||
`X-Object-Meta-Mtime` as floating point since the epoch accurate to 1
|
||||
ns.
|
||||
|
||||
This is a defacto standard (used in the official python-swiftclient
|
||||
amongst others) for storing the modification time for an object.
|
||||
|
||||
### Limitations ###
|
||||
|
||||
The Swift API doesn't return a correct MD5SUM for segmented files
|
||||
(Dynamic or Static Large Objects) so rclone won't check or use the
|
||||
MD5SUM for these.
|
||||
|
||||
### Troubleshooting ###
|
||||
|
||||
#### Rclone gives Failed to create file system for "remote:": Bad Request ####
|
||||
|
||||
Due to an oddity of the underlying swift library, it gives a "Bad
|
||||
Request" error rather than a more sensible error when the
|
||||
authentication fails for Swift.
|
||||
|
||||
So this most likely means your username / password is wrong. You can
|
||||
investigate further with the `--dump-bodies` flag.
|
||||
|
||||
This may also be caused by specifying the region when you shouldn't
|
||||
have (eg OVH).
|
||||
|
||||
#### Rclone gives Failed to create file system: Response didn't have storage storage url and auth token ####
|
||||
|
||||
This is most likely caused by forgetting to specify your tenant when
|
||||
setting up a swift remote.
|
||||
Reference in New Issue
Block a user