rclone is a command line tool, similar to rsync, with the difference that it can sync, move, copy, and in general do other file operations on cloud services. You can use rclone to create backups of your servers or personal computers or to store your files in the cloud, optionally adding encryption to keep them safe from prying eyes, which is what this post is about.

rclone supports the most popular cloud storage services including but not limited to Dropbox, Amazon Cloud Drive, Google Drive, and OneDrive. You can get unlimited Amazon Cloud Storage for $60 a year (update – this is no longer true for US-based accounts and it now costs $us 60 per TB per year), and unlimited Google Drive Storage if you pay for a G Suite account (around $10 for a domain per year) for $10 per month. G Suite lists unlimited storage starting when you have 5 users or more, but the guys at reddit.com/r/datahoarder state that, while the limit is there, Google doesn’t check for your usage so you can have unlimited storage as well (you do get “official” unlimited storage once you have 5 people in your G Suite account). I cannot verify the validity of this claim currently (Update – It’s real, guys).

Getting Started With rclone

Using the command line can be daunting, but it doesn’t have to be. rclone’s commands are nicely structured and are very easy to do. The initial configuration is straight forward and it doesn’t take too long.

Installation

You can skip this section if you have experience with rclone.

The Installation Page has instructions to install on your favorite Operating System. rclone runs in Linux, macOS, and Windows.

If you use macOS, you can install it with a single command using Homebrew:

brew install rclone

If you don’t use Homebrew, refer to the link above to install rclone without it.

Configuring and Using a Normal rclone Remote

You can skip this section if you already have experience with rclone, if you know how to configure a remote, and if you know how to use the basic rclone commands (copy, move, etc).

Before you can use an encrypted remote, you need to understand how to use normal ones.

What is a remote, anyway?

I have been saying this word a lot, so it might not be obvious what it means. rclone’s base concept is the remote, and it’s simply a cloud service that you have configured to use with rclone. For example, if you configure your Dropbox account, then it becomes a remote for rclone. The same with your Google account – Configure your Google account, and it will be a remote for rclone.

rclone config

rclone config is the command you use for configuration purposes, including creating remotes. In this example, I will configure a standard Gmail account. Go ahead and type this in your terminal:

rclone config

If you already have some remotes, they will be listed at the top. The output of the command will be something like this:

Name                 Type
====                 ====
AmazonCloudDrive  amazon cloud drive
EncryptedACD      crypt

e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q>

Naturally, you won’t see any remotes at the top if you don’t have any yet. Go ahead and press n to create a new remote, followed by enter. The first thing it will do is to prompt you for a name. The name of the remote can be anything you recognize. Should not contain spaces. I will name mine aibanezDrive.

Following the name, you will be prompted for a Storage. I am configuring Google Drive, so I will choose option 7.

Many providers will prompt you for a client_id and a client_secret. You don’t need these most of the time, and you should only use them if you created a custom app for your remotes. In my experience, I haven’t had the need to do that. You can press enter without writing anything at both prompts.

Later you will be asked if you want to use auto config. You will say yes to this if you are not running a headless system (no GUI). By choosing this option, rclone will launch a browser window prompting you to log in to your account (in this case, Gmail). You will see the standard login flow for your provider. After you are done with the credentials and allowing rclone to use your storage account, you will see a screen saying “Success!” and it will ask you to look back at your rclone terminal window.

Success

You will see something like this:

[aibanezDrive]
client_id = 
client_secret = 
token = {...}

Press y to save the remote. Then press q and enter to exit the configuration prompt.

To configure that your remote was saved successfully, you can run rclone listremotes. This will show you all your remotes, with the name you gave them. On my system, this prints:

AmazonCloudDrive
EncryptedACD
aibanezDrive

Trying out our rclone remote

Now that we have created a remote, we need to try it out! In this example we will create a list of files and directories and upload them to our Google Drive account using rclone. I need to remind you that we are currently playing with a non encrypted remote, so don’t choose any files you wouldn’t store unencrypted in the cloud. We will create the encrypted remote at the end of this tutorial.

For this part, I will go to ~Desktop and create a directory called Cards there. I will create a plain file and two directories which contain more plain files there. Essentially, I will be creating this:

Cards
 - Sakura
   - The Mirror
   - The Fly
   - The Sword
   - The Arrow
 - Shaoran
   - The Time
   - The Storm
   - The Return
   - The Freeze
 - None.txt

Following the convention of using Card Captor Sakura examples, I will create two folders for both main characters along with a few cards they caught in the show, and a None.txt file that would contain a list of cards none of them caught.

If you want to do the same, just run these commands below to create the same folder structure:

mkdir ~/Desktop/Cards && cd ~/Desktop/Cards
mkdir Sakura Shaoran
touch Sakura/The\ Mirror
touch Sakura/The\ Fly
touch Sakura/The\ Sword
touch Sakura/The\ Return
touch Shaoran/The\ Time
touch Shaoran/The\ Storm
touch Shaoran/The\ Return
touch Shaoran/The\ Freeze
touch None.txt

Copying the local directory to Google Drive

The command to copy files and directories is very straight forward.

Copy to local directory
rclone copy LOCAL_DIRECTORY_OR_FILE REMOTE_DIRECTORY

Note that you can copy both entire directories or just files within them.

The remote path starts with REMOTE_NAME:. Since in this example we are copying to the aibanezDrive remote, you would write aibanezDrive:, followed by the full path to copy to (starting with a /).

Copy from the remote directory

Just like you can copy files to your remote, you can also copy from the remote to your local computer. Otherwise it would be quite useless!

rclone copy REMOTE_DIRECTORY_OR_FILE LOCAL_DIRECTORY

rclone Copying Example

If you are using the same folder as me, you can copy and paste this command to see how it works. If not, you will have to modify REMOTE_DIRECTORY_OR_FILE and LOCAL_DIRECTORY as needed.

So to copy our Cards directory to /Cards, you would write this:

rclone copy ~/Desktop/Cards aibanezDrive:/Cards

After the operation finishes, you will see a small report on the operation:

Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            9
Elapsed time:       28.9s

(Please disregard the Elapsed Time for a few files. It is known the internet in my country sucks and there’s no really no way to get a better upload than 1mbps).

You can now go to your Google Drive account using your web browser. You will see the Cards directory at the root of your account.

Cards

Going inside that directory you will see the other two folders and the None.txt file.

Inside Cards

Finally going inside either the Sakura or Shaoran directories you can see their respective files as well.

Sakura

Now, if you want to copy something From your Google Drive account to your local computer, the steps are just as easy, with the only change being that the order of the local and remote folders are switched in the command.

Go ahead and upload any file to your Google Drive account. I uploaded a file called “IMG_1434.JPG” and I uploaded it to my root. If I wanted to copy this file from my Drive account to my Local computer, inside the ~/Desktop/Cards directory using rclone, I’d use this command:

rclone copy aibanezDrive:/IMG_1434.JPG ~/Desktop/Cards

And that’s it! Remember you can use the copy command for both directories and files.

Other Commands

rclone Supports a whole lot of commands as listed in their Documentation. Some useful ones are sync (it keeps a remote directory in sync with a local one), move, delete (remove the contents of a path), purge (remove a path and all its contents), ls (List all the objects in the specified path), lsd (list all the directories under the path), mkdir, and rmdir.

There is also a whole lot of interesting flags you can use with your commands, like --bwlimit to limit the amount of bandwidth rclone will use and --transfers to limit the amount of files that get transferred in batch, and others.

Other Notes

It’s important to note that some commands may not work or may behave differently based on what kind of remote you are using. The Storage Systems Overview page has a helpful table and a few notes for how commands may behave. There’s also a page for every supported remote that lists their quirks, features, and possible different behaviors for common commands.

Configuring and Using an Encrypted Remote.

We are finally at the main point of this post. Hooray!

Due to the nature of cloud storage, you may not want to store your files in a plain format, because that implies that they are storied unsafely in some other guy’s computer and they might be visible to someone else. If someone breaks into your cloud storage account, they can see your files. That wouldn’t be pretty.

At the same time, there’s some files that may be fine to store unencrypted. Maybe a list of groceries you need to buy, or other kind of files that there is no issue if other people see them or you just feel save storing them as is.

rclone supports the use of encryption, and it can be used in such a way that a remote can hold both encrypted and non-encrypted files.

The crypt Remote.

To use encryption, you create a crypt remote. This is an special kind of remote. If you configure any other kind of remote, you are creating a direct connection between your computer and another remote, and that’s what makes crypt different. Unlike the other remotes, crypt is placed on top of an existing remote in order to do its job, and is not a direct connection like the others.

The remote we created in this tutorial, aibanezDrive was a direct remote between your computer and Google Drive. We will now create a crypt remote that uses this remote as an underlying requirement for the encrypted one. So the crypt remote will encrypt the files, pass them over to your standard remote, and this one will end the encrypted files to the cloud.

Creating a crypt Remote.

Before we create an account, delete the /Cards folder from your Google Drive (if you created it), as we will be using this same directory to show how crypt works.

With that out of the way, we need to run rclone config again. Press n to create a new remote.

The one thing I don’t like about rclone is that, when listing your remotes, you can’t see what underlying remote a crypt one is using, so when prompted for the name, I recommend giving it a name that helps you identify both the crypt and the underlying remote it’s using. I will be naming it aibanezDrive_Crypt.

After the name, enter 5 to create a crypt remote.

This is where the configuration takes a weird turn, unlike the others. When prompted for a remote, you specify an existing remote, along with its path. The path you will choose will be the root of the remote.

Suppose you choose the root of your remote to be /Archive/Encrypted. This will cause the crypt remote to store all the encrypted files there, and also, when you refer to the root of the remote – say, aibanezDrive_Crypt:/ -, you will be referring to the whole /Archive/Encrypted directory in your storage of choice. In other words, your remote won’t be able to see anything outside this path you specified. If you are a UNIX user, you can think that your remote is chrooted to /Archive/Encrypted.

In this example, I want to archive encrypted files in aibanezDrive:/Archive/Encrypted. Note that we are not referring to the remote we are creating itself, but rather the underlying one.

Next you will be prompted if you want to encrypt the file names. The way I see it, yes is the only right choice here but you might not mind it if the file names are visible. Choosing yes has some complications all listed in the crypt section of the documentation. There’s issues with long file names and paths. But, in general, if your file names are below 156 characters, you should be fine on all providers, granted that some provides may not have this issue. Please refer to the documentation to see if you can find any info on filename length.

Choose 2.

Next you will be asked to create a password or to have one generated for you. Both options are strong and it depends on how much you trust the RNG of your system.

I will be writing my own.

Next you will be asked for the salt. In cryptography and security, a salt modifies a string so it’s hash is entirely different. The salt is what makes your password cat to be stored as entirely different strings in databases that store passwords.

You have three options this time: To provide one, have one generated for you, or to not use a hash at all (not recommended). Keep in mind that whether you provide one or have one generated for you, you will have to store in a safe place, as rclone will ask you for both when interacting with a crypt remote. Not providing the salt when required will result in you not having access to these files later.

I will be providing my own hash.

After providing the salt you will see the newly created remote:

[aibanezDrive_Crypt]
remote = aibanezDrive:/Archive/Encrypted
filename_encryption = on
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***

Press y at the prompt and then q.

You now have your newly created remote and you can use it exactly the same you would use any other remote. For this example, I will be running exactly the same example as the one I used to demonstrate how copying works with a normal Google Drive remote, except I will just change the remote names and paths.

rclone copy ~/Desktop/Cards aibanezDrive_Crypt:/

After the script is done, you can verify if they exist in your Drive account, but you won’t be able to see their contents (or filenames) at all! Remember that /Archive/Encrypted is the root of the remote, so you your Google Drive will have this same path where you will see the encrypted files.

Crypt Results

Note that, unlike a normal remote, you cannot just add files and restore them. They will not get magically encrypted and you will just add normal files without encryption if you uploaded them directly. So if you want to encrypt the file names, you can only do it with your crypt remote with rclone

And since they are encrypted, the only way to download them is with rclone itself as well. So normally, you would want to list the files that are available in the controller in order to do that. You can do that with the ls and lsd commands:

rclone ls aibanezDrive_Crypt:/
Andys-iMac:Cards andyibanez$ rclone ls aibanezDrive_Crypt:/
   358768 IMG_1434.JPG
        0 None.txt
        0 Shaoran/The Freeze
        0 Shaoran/The Return
        0 Shaoran/The Storm
        0 Shaoran/The Time
        0 Sakura/The Sword
        0 Sakura/The Mirror
        0 Sakura/The Return
        0 Sakura/The Fly

Note that rclone really tries to list everything when just using the ls command. You an list the directories only using lsd (this would not print IMG_1434.JPG and None.txt in this case), or if you just want to view the files (not directories) at the top level, you can run the ls command with the --max-depth=1 flag:

Andys-iMac:Cards andyibanez$ rclone ls --max-depth=1 aibanezDrive_Crypt:/
   358768 IMG_1434.JPG
        0 None.txt

Now suppose I want to download the Sakura folder to my computer:

rclone copy aibanezDrive_Crypt:/Sakura ~/Desktop/restored

This will download the contents from Sakura into ~/Desktop/restored.

Conclusion

rclone is a fantastic tool for creating backups and for using cloud storage. If you want to use rsync but would rather store your files someplace else, rclone is the perfect tool to do it. It supports many cloud providers, it’s open source, it’s under active development, and it supports cryptography out of the box.

Positive SSL