Image of the glider from the Game of Life by John Conway
Skip to content

ZFS Administration, Part XIII- Sending and Receiving Filesystems

Table of Contents

Zpool Administration ZFS Administration Appendices
0. Install ZFS on Debian GNU/Linux 9. Copy-on-write A. Visualizing The ZFS Intent Log (ZIL)
1. VDEVs 10. Creating Filesystems B. Using USB Drives
2. RAIDZ 11. Compression and Deduplication C. Why You Should Use ECC RAM
3. The ZFS Intent Log (ZIL) 12. Snapshots and Clones D. The True Cost Of Deduplication
4. The Adjustable Replacement Cache (ARC) 13. Sending and Receiving Filesystems
5. Exporting and Importing Storage Pools 14. ZVOLs
6. Scrub and Resilver 15. iSCSI, NFS and Samba
7. Getting and Setting Properties 16. Getting and Setting Properties
8. Best Practices and Caveats 17. Best Practices and Caveats

Now that you're a pro at snapshots, we can move to one of the crown jewels of ZFS- the ability to send and receive full filesystems from one host to another. This is epic, and I am not aware of any other filesystem that can do this without the help of 3rd party tools, such as "dd" and "nc".

ZFS Send

Sending a ZFS filesystem means taking a snapshot of a dataset, and sending the snapshot. This ensures that while sending the data, it will always remain consistent, which is crux for all things ZFS. By default, we send the data to a file. We then can move that single file to an offsite backup, another storage server, or whatever. The advantage a ZFS send has over "dd", is the fact that you do not need to take the filesystem offilne to get at the data. This is a Big Win IMO.

To send a filesystem to a file, you first must make a snapshot of the dataset. After the snapshot has been made, you send the snapshot. This produces an output stream, that must be redirected. As such, you would issue something like the following:

# zfs snapshot tank/test@tuesday
# zfs send tank/test@tuesday > /backup/test-tuesday.img

Now, your brain should be thinking. You have at your disposal a whole suite of Unix utilities to manipulate data. So, rather than storing the raw data, how about we compress it with the "xz" utility?

# zfs send tank/test@tuesday | xz > /backup/test-tuesday.img.xz

Want to encrypt the backup? You could use OpenSSL or GnuPG:

# zfs send tank/test@tuesday | xz | openssl enc -aes-256-cbc -a -salt > /backup/test-tuesday.img.xz.asc

ZFS Receive

Receiving ZFS filesystems is the other side of the coin. Where you have a data stream, you can import that data into a full writable filesystem. It wouldn't make much sense to send the filesystem to an image file, if you can't really do anything with the data in the file.

Just as "zfs send" operates on streams, "zfs receive" does the same. So, suppose we want to receive the "/backup/test-tuesday.img" filesystem. We can receive it into any storage pool, and it will create the necessary dataset.

# zfs receive tank/test2 < /backup/test-tuesday.img

Of course, in our sending example, I compressed and encrypted a sent filesystem. So, to reverse that process, I do the commands in the reverse order:

# openssl enc -d -aes-256-cbc -a -in /storage/temp/testzone.gz.ssl | unxz | zfs receive tank/test2

The "zfs recv" command can be used as a shortcut.

Combining Send and Receive

Both "zfs send" and "zfs receive" operate on streams of input and output. So, it would make sense that we can send a filesystem into another. Of course we can do this locally:

# zfs send tank/test@tuesday | zfs receive pool/test

This is perfectly acceptable, but it doesn't make a lot of sense to keep multiple copies of the filesystem on the same storage server. Instead, it would make better sense to send the filesystem to a remote box. You can do this trivially with OpenSSH:

# zfs send tank/test@tuesday | ssh "zfs receive pool/test"

Check out the simplicity of that command. You're taking live, running and consistent data from a snapshot, and sending that data to another box. This is epic for offsite storage backups. On your ZFS storage servers, you would run frequent snapshots of the datasets. Then, as a nightly cron job, you would "zfs send" the latest snapshot to an offsite storage server using "zfs receive". And because you are running a secure, tight ship, you encrypt the data with OpenSSL and XZ. Win.


Again, I can't stress the simplicity of sending and receiving ZFS filesystems. This is one of the biggest features in my book that makes ZFS a serious contender in the storage market. Put it in your nightly cron, and make offsite backups of your data with ZFS sending and receiving. You can send filesystems without unmounting them. You can change dataset properties on the receiving end. All your data remains consistent. You can combine it with other Unix utilities.

It's just pure win.

{ 8 } Comments

  1. Niek Bergboer using Google Chrome 24.0.1312.57 on Mac OS | February 12, 2013 at 12:26 pm | Permalink

    Snapshots, and the ability to send and receive them, makes for automatic incremental backups: uses this, and you can trivially do hourly (or even more often) filesystem replication, keeping all snapshots.

  2. max using Opera 9.80 on Linux Mint 64 bits | April 18, 2013 at 8:32 am | Permalink

    Thanks for zfs guide. Its very helpfull.
    But I have a trouble with send-receive.
    I have two pools in my system. I tried to send snapshot from first pool to second. Firest dataset had compression property.
    I tried to execute command such as:
    # zfs send pool750/data@01 | zfs receive -o compression=gzip pool250

    But system said:

    invalid option 'o'
    receive [-vnFu]
    receive [-vnFu] [-d | -e]

    For the property list, run: zfs set|get

    For the delegated permission list, run: zfs allow|unallow

    I have last version of ubuntu-zfs package.
    What is wrong?

    P.S. Sorry for my english

  3. Aaron Toponce using Google Chrome 26.0.1410.43 on GNU/Linux 64 bits | April 18, 2013 at 9:31 am | Permalink

    According to the zfs(8) manpage, "zfs receive -o" is not a valid option. "zfs receive | recv [-vnFu] filesystem|volume|snapshot" or "zfs receive | recv [-vnFu] [-d|-e] filesystem".

    By default, ZFS saves dataset settings when sending them to another location. If you want to change a setting, then you will need to run a separate "zfs set" command after the sending of the dataset has finished.

  4. max using Opera 9.80 on Linux Mint 64 bits | April 18, 2013 at 11:25 pm | Permalink

    >>According to the zfs(8) manpage, “zfs receive -o” is not a valid option.

    Yes, but in your post was written about this feature.

    >> By default, ZFS saves dataset settings when sending them to another location.

    Hm... I tried to send about 280GB from gziped dataset to another pool yesterday. The pool was about 320GB free space. This was epic fail for me because compression wasn't automatically set on destination dataset.

  5. Aaron Toponce using Google Chrome 25.0.1364.160 on GNU/Linux 64 bits | April 19, 2013 at 2:27 am | Permalink

    "zfs create" has the -o switch for setting dataset parameters. "zfs send" and "zfs receive" do not have this switch, and no where in this post do I state that "-o" is valid for "zfs send" or "zfs receive". If you run "zfs get compress pool/dataset" on your dataset, and "gzip" is returned, then when using "zfs send" you need to use the " -p" switch:


    Include the dataset's properties in the stream. This flag is implicit when -R is specified. The receiving system must also support this feature.

  6. max using Opera 9.80 on Linux Mint 64 bits | April 19, 2013 at 3:03 am | Permalink

    Thanks a lot.

  7. Warren Downs using Google Chrome 20.0.1132.47 on Ubuntu | July 1, 2013 at 5:40 pm | Permalink

    Regarding 'no where in this post do I state that “-o” is valid for “zfs send” or “zfs receive”', you may wish to change the above post to make that statement true (quoting original post):

    Further, you can change dataset properties on the receiving end. If the sending filesystem was not compressed, but you wish to compress the data on the receiving side, you can enable that as follows:

    # zfs send tank/test@tuesday | ssh "zfs receive -o compression=lzjb pool/test"

  8. syed using Firefox 24.0 on Windows XP | October 28, 2013 at 12:57 pm | Permalink

    Thanks, really appreciate it. Was very helpful.

{ 6 } Trackbacks

  1. [...] Sending and Receiving Filesystems [...]

  2. [...] Sending and Receiving Filesystems [...]

  3. [...] Sending and Receiving Filesystems [...]

  4. [...] Sending and Receiving Filesystems [...]

  5. [...] Sending and Receiving Filesystems [...]

  6. [...] Sending and Receiving Filesystems [...]

Post a Comment

Your email is never published nor shared.

Switch to our mobile site