Introduction
In my last post, I talked about setting up an ODROID-HC2 as a NAS using OpenMediaVault. I have that up and running, and I’ve also written a few scripts to backup my data to a few of the SMB shares.
Now, I need to get that data shipped offsite to an external location to cover my 3-2-1 backup strategy:
- 3 backups
- 2 different types of media
- 1 backup offsite
Software
My cloud storage provider of choice is B2. I’ve written about them in the past and have generally had good luck with them, so I’d like to keep using them.
I was looking for software to transfer my backups to an offsite location, but it had it fit a few requirements:
- work with B2 by default - B2 is object storage, so I can’t just push files to it via SSH or SCP, I need a client that can speak in B2’s HTTP API language
- support encryption locally - because this data is going to be stored on devices not controlled by me, I wanted it to be encrypted locally before being sent to storage
- open source - since this program is going to be encrypting my data, I want it to be auditable and trustworthy
While looking for backup programs, I compared the following:
I ended up choosing rclone for this task, instead of Duplicity. Duplicity is great, but it requires a good bit of memory to run, and it writes temporary files to local storage while it encrypts and uploads them. Because the ODROID-HC2 has limited hardware, I didn’t want this to become a problem. As far as I can tell, rclone doesn’t have these problems or limitations. In addition, this backup is really a backup of a backup, so I’m just interested in pushing large amounts of data offsite as quickly as possible, which rclone seems to be suited for.
Setup rclone
Install rclone
Rclone is available in the default Debian/Ubuntu repositories. However, the version is pretty out of date. On Ubuntu, you can add a PPA to get a newer version, but on Debian you can’t. Because of this, I recommend downloading the .deb directly from the rclone website.
sudo curl -sLO https://downloads.rclone.org/rclone-current-linux-arm.deb && sudo dpkg -i rclone-current-linux-arm.deb
Then, verify rclone is installed.
rclone -V
Configure rclone remote
Rclone uses a concept called remotes. Remotes are just remote storage locations, and you can nest remotes inside of other remotes.
First, configure rclone. It’s important to do this as the user that will be running rclone, so keep that in mind if you want to use a service account (more about this later).
rclone config
Press n to create a new remote.
n
Name the remote (I’m using backup01).
backup01
Press 3 to select B2 from the menu.
3
Enter your B2 account ID (you get this from B2’s control panel).
xxxxxxxxxxxx
Enter your B2 application ID (you get this from B2’s control panel).
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Leave the endpoint blank.
Press y to to save your configuration.
y
Configure rclone encrypted remote
Now, we’re going to created an encrypted remote inside of our first remote. Anything that goes into this encrypted remote will be encrypted automatically.
Press n to create a new remote.
n
Name the remote (I’m using backup01_crypt).
backup01_crypt
Press 5 to select crypt from the menu.
5
Enter the name of your rclone remote (e.g., backup01), followed by your B2 bucket name.
backup01:bucket01
Press 2 to encrypt the file names.
2
Choose to create or generate a passphrase (I’m going to generate one).
Choose to create or generate a salt (I’m going to generate one).
Press y to to save your configuration.
y
Run backup
When you want to backup a file or directory, use the command below.
rclone sync /home/local/directory remote:bucket
In my case, the remote is backup01_crypt, and the bucket name is bucket01.
rclone sync /home/local/directory backup01_crypt:bucket01
Once the files are synced, you can list the files on the remote using the command below.
rclone lsf remote:bucket
In my case, the remote is backup01_crypt, and the bucket name is bucket01.
rclone lsf backup01_crypt:bucket01
ODROID performance
Overall, it look a little under 24 hours to do the initial transfer of around 580GB. During that time, my ODROID was hovering about 65-70°C.
CPU temp: 68°C
Also during this time, the ODROID-HC2 was using about 1GB RAM out of the available 2GB.
total used free shared buff/cache available
Mem: 1993 965 209 51 818 921
Swap: 996 91 905
CPU usage hovered around 60% total capacity.
Load average: 0.59 0.56 0.71
A note about bandwidth
By default, rclone will transfer four files at the same time. If you want to change this number, you can use the --transfers flag. Obviously, if you have the hardware to support it, you can increase this number, which will decrease the amount of time it takes to complete your transfers, at the cost of CPU, memory, and bandwidth.
rclone --transfers=6 sync /home/local/directory backup01_crypt:bucket01
My internet connection is Verizon FiOS 100/100Mbps. I found that when using the default of four transfers at once, my bandwidth usage was about 60-80Mbps. However, when I kicked the transfers up to six, my bandwidth was 100% utilized at 100Mbps. This does decrease transfer time, but the internet in my house was almost unusable because the connection was saturated. Just something to keep in mind.
Backup your config file!
By default, rclone stores all of your B2 account information, your password, salt, and settings in the ~/.rclone.conf file of the account you used to configure rclone. If you lose this file, you lose access to all of your backups. Obviously, backup this file, but don’t save the backup on your encrypted storage.
Helpful commands
I’ve tried to gather some of the most useful commands from rclone’s website.
| Command | Description |
|---|---|
rclone listremotes |
List all remotes |
rclone config show remote |
Show config for a remote |
rclone size remote:bucket |
Show total size and number of objects on remote |
rclone ls remote:bucket |
List objects on remote |
rclone lsd remote:bucket |
List directories on remote |
rclone ncdu remote:bucket |
Like NCDU, but for your remote. Useful for seeing what is taking up disk space |
rclone mount --read-only remote: /path/to/local/directory |
Mount your remote in a read-only state on a local directory |
rclone tree remote:bucket |
Like tree Caveat - if filenames are encrypted, it will show them as encrypted |
rclone serve http remote:bucket --addr :8080 |
Start a HTTP server to browse the remote, listen on all IPs on port 8080. Caveat - if filenames are encrypted, it will show them as encrypted |
rclone sync /home/local/directory remote:bucket |
Copy source to destination, but do delete files in the destination if they were deleted from source (like the --delete flag in rsync) |
rclone copy /home/local/directory remote:bucket |
Copy source to destination, but do not delete files in the destination if they were deleted from source |
rclone cleanup remote:bucket |
Delete old versions of files stored on the remote |
Hope this helps!
-Logan
