Image of the glider from the Game of Life by John Conway
Skip to content

ZFS Administration, Part XIII- Sending and Receiving Filesystems

Table of Contents

Zpool Administration ZFS Administration Appendices
0. Install ZFS on Debian GNU/Linux 9. Copy-on-write A. Visualizing The ZFS Intent Log (ZIL)
1. VDEVs 10. Creating Filesystems B. Using USB Drives
2. RAIDZ 11. Compression and Deduplication C. Why You Should Use ECC RAM
3. The ZFS Intent Log (ZIL) 12. Snapshots and Clones D. The True Cost Of Deduplication
4. The Adjustable Replacement Cache (ARC) 13. Sending and Receiving Filesystems
5. Exporting and Importing Storage Pools 14. ZVOLs
6. Scrub and Resilver 15. iSCSI, NFS and Samba
7. Getting and Setting Properties 16. Getting and Setting Properties
8. Best Practices and Caveats 17. Best Practices and Caveats

Now that you're a pro at snapshots, we can move to one of the crown jewels of ZFS- the ability to send and receive full filesystems from one host to another. This is epic, and I am not aware of any other filesystem that can do this without the help of 3rd party tools, such as "dd" and "nc".

ZFS Send

Sending a ZFS filesystem means taking a snapshot of a dataset, and sending the snapshot. This ensures that while sending the data, it will always remain consistent, which is crux for all things ZFS. By default, we send the data to a file. We then can move that single file to an offsite backup, another storage server, or whatever. The advantage a ZFS send has over "dd", is the fact that you do not need to take the filesystem offilne to get at the data. This is a Big Win IMO.

To send a filesystem to a file, you first must make a snapshot of the dataset. After the snapshot has been made, you send the snapshot. This produces an output stream, that must be redirected. As such, you would issue something like the following:

# zfs snapshot tank/test@tuesday
# zfs send tank/test@tuesday > /backup/test-tuesday.img

Now, your brain should be thinking. You have at your disposal a whole suite of Unix utilities to manipulate data. So, rather than storing the raw data, how about we compress it with the "xz" utility?

# zfs send tank/test@tuesday | xz > /backup/test-tuesday.img.xz

Want to encrypt the backup? You could use OpenSSL or GnuPG:

# zfs send tank/test@tuesday | xz | openssl enc -aes-256-cbc -a -salt > /backup/test-tuesday.img.xz.asc

ZFS Receive

Receiving ZFS filesystems is the other side of the coin. Where you have a data stream, you can import that data into a full writable filesystem. It wouldn't make much sense to send the filesystem to an image file, if you can't really do anything with the data in the file.

Just as "zfs send" operates on streams, "zfs receive" does the same. So, suppose we want to receive the "/backup/test-tuesday.img" filesystem. We can receive it into any storage pool, and it will create the necessary dataset.

# zfs receive tank/test2 < /backup/test-tuesday.img

Of course, in our sending example, I compressed and encrypted a sent filesystem. So, to reverse that process, I do the commands in the reverse order:

# openssl enc -d -aes-256-cbc -a -in /storage/temp/testzone.gz.ssl | unxz | zfs receive tank/test2

The "zfs recv" command can be used as a shortcut.

Combining Send and Receive

Both "zfs send" and "zfs receive" operate on streams of input and output. So, it would make sense that we can send a filesystem into another. Of course we can do this locally:

# zfs send tank/test@tuesday | zfs receive pool/test

This is perfectly acceptable, but it doesn't make a lot of sense to keep multiple copies of the filesystem on the same storage server. Instead, it would make better sense to send the filesystem to a remote box. You can do this trivially with OpenSSH:

# zfs send tank/test@tuesday | ssh "zfs receive pool/test"

Check out the simplicity of that command. You're taking live, running and consistent data from a snapshot, and sending that data to another box. This is epic for offsite storage backups. On your ZFS storage servers, you would run frequent snapshots of the datasets. Then, as a nightly cron job, you would "zfs send" the latest snapshot to an offsite storage server using "zfs receive". And because you are running a secure, tight ship, you encrypt the data with OpenSSL and XZ. Win.


Again, I can't stress the simplicity of sending and receiving ZFS filesystems. This is one of the biggest features in my book that makes ZFS a serious contender in the storage market. Put it in your nightly cron, and make offsite backups of your data with ZFS sending and receiving. You can send filesystems without unmounting them. You can change dataset properties on the receiving end. All your data remains consistent. You can combine it with other Unix utilities.

It's just pure win.

{ 12 } Comments

  1. Niek Bergboer | February 12, 2013 at 12:26 pm | Permalink

    Snapshots, and the ability to send and receive them, makes for automatic incremental backups: uses this, and you can trivially do hourly (or even more often) filesystem replication, keeping all snapshots.

  2. max | April 18, 2013 at 8:32 am | Permalink

    Thanks for zfs guide. Its very helpfull.
    But I have a trouble with send-receive.
    I have two pools in my system. I tried to send snapshot from first pool to second. Firest dataset had compression property.
    I tried to execute command such as:
    # zfs send pool750/data@01 | zfs receive -o compression=gzip pool250

    But system said:

    invalid option 'o'
    receive [-vnFu]
    receive [-vnFu] [-d | -e]

    For the property list, run: zfs set|get

    For the delegated permission list, run: zfs allow|unallow

    I have last version of ubuntu-zfs package.
    What is wrong?

    P.S. Sorry for my english

  3. Aaron Toponce | April 18, 2013 at 9:31 am | Permalink

    According to the zfs(8) manpage, "zfs receive -o" is not a valid option. "zfs receive | recv [-vnFu] filesystem|volume|snapshot" or "zfs receive | recv [-vnFu] [-d|-e] filesystem".

    By default, ZFS saves dataset settings when sending them to another location. If you want to change a setting, then you will need to run a separate "zfs set" command after the sending of the dataset has finished.

  4. max | April 18, 2013 at 11:25 pm | Permalink

    >>According to the zfs(8) manpage, “zfs receive -o” is not a valid option.

    Yes, but in your post was written about this feature.

    >> By default, ZFS saves dataset settings when sending them to another location.

    Hm... I tried to send about 280GB from gziped dataset to another pool yesterday. The pool was about 320GB free space. This was epic fail for me because compression wasn't automatically set on destination dataset.

  5. Aaron Toponce | April 19, 2013 at 2:27 am | Permalink

    "zfs create" has the -o switch for setting dataset parameters. "zfs send" and "zfs receive" do not have this switch, and no where in this post do I state that "-o" is valid for "zfs send" or "zfs receive". If you run "zfs get compress pool/dataset" on your dataset, and "gzip" is returned, then when using "zfs send" you need to use the " -p" switch:


    Include the dataset's properties in the stream. This flag is implicit when -R is specified. The receiving system must also support this feature.

  6. max | April 19, 2013 at 3:03 am | Permalink

    Thanks a lot.

  7. Warren Downs | July 1, 2013 at 5:40 pm | Permalink

    Regarding 'no where in this post do I state that “-o” is valid for “zfs send” or “zfs receive”', you may wish to change the above post to make that statement true (quoting original post):

    Further, you can change dataset properties on the receiving end. If the sending filesystem was not compressed, but you wish to compress the data on the receiving side, you can enable that as follows:

    # zfs send tank/test@tuesday | ssh "zfs receive -o compression=lzjb pool/test"

  8. syed | October 28, 2013 at 12:57 pm | Permalink

    Thanks, really appreciate it. Was very helpful.

  9. Broen van Besien | January 11, 2015 at 11:47 am | Permalink

    Sent and receive of entire snapshots is a big win indeed.
    And it's even more powerful:
    You can send incremental updates out-of-the-box by using the -i option, which will only send the difference between a current and a previous snapshot.

  10. NM | February 4, 2015 at 5:48 pm | Permalink

    The incremental updates suggested by Broen van Besien are a big plus. This could enable a portable/carryable system: send to laptop in the morning, work on it, send back to desktop when arriving home.

  11. Aaron Toponce | February 17, 2015 at 1:35 pm | Permalink

    The incremental updates suggested by Broen van Besien are a big plus. This could enable a portable/carryable system: send to laptop in the morning, work on it, send back to desktop when arriving home. Feasible?

    Sure. I send incremental snapshots from my workstation to my external backup drive every 15 minutes. Works like a charm. I don't see any problems with this for a laptop going to a backup server.

  12. Alan Galitz MD | December 8, 2015 at 9:27 pm | Permalink

    While it has been over 2 years, perhaps a how to on migrating a large pool from one array to another might be worth including.

    This link covers the basic steps, but I and others, judging by all the posts I found, hit several problems whose solutions are not well listed and thought since so many people use your site as a reference this might help. Perhaps this issue is cropping up with the addition and changes to the auto-snapshot implementation in zfs on linux.

    I have several home servers with 4-5TB if data each. They store identical copies of media, crashplan backup files, desktop/laptop files (yes the crashplan goes off site too, but the media is too large so 3 separate machines with redundant zfs is the backup plan for some the the large media). The send operation is the better part of a day.

    A few years ago I followed your excellent advice -THANKS- and setup Raidz with 3 x4TB on 2 machines and 4x3TB on one. Well now it turns out that this no longer best practice so after some drives began failing from age - no data loss- yea ZFS scrub! - time to covert to mirrors and stripes. So with some dual USB3 toasters, I build a 7TB array with mirrors 2x2 to temp store the data. So time to send the data to the temp array:

    # zfs snapshot -r zsrv@01
    # zfs send -R zsrv@01 | zfs receive -Fdvu zsrv-old

    Note, don't keep your terminal open while doing this as a sleep/hibenation might end the operation. Detach from the session to resume later. I use byobu since it comes with xubuntu.

    OK, to avoid massive fail, I found it necessary to turn OFF auto-shapshots. The new dataset started doing frequent snapshots and their presence killed the send/recv randomly. Perhaps this is a bug, the the first send/recv ( no "-i") doesn't like a pre-exiisting data set)

    The important error message was just before a long string of "...broken pipe..." in SSH session. I found it at the machine's display since the volume blew past the SSH buffer. The 'v' switch on the recv produces alot of output if you have hundreds of old snapshots.

    Also, since I was sending nfs shares, both the old and new datasets used the same folder, causing mounting errors, but that's later. I set the the top level dataset with the pool name to not mount or send shares but since the send -R causes properties to be copied the new pools switch them back on. The -u option is supposed to stop local mounting, but the nfs shares mounted locally anyway it seems. I'm not a pro so do not have the time to test much or know.

    It did not help that without the nfs share mounted by zfs, crashplan periodically would create a local folder, blocking the creation of the backup target dataset. So before the first full stream, check the usualy mounting place to ensure they really are empty and

    # zfs list -t snapshot | grep new # or a unique piece of the new pool

    OK, so for the final incremental send, ensure the disk isn't in use (shutdown iSCSI, NFS, syslog, crashplan,etc) see above
    Make another snapshot of 'zsrv' and incrementally send it to 'zsrv-new' (this should take a short time)

    # zfs snapshot -r zsrv@02
    # zfs send -R -i zsrv@01 zsrv@02 | zfs recv -dvu zsrv-new

    Now the instructions included a way to bring back the old pool and do some comparison. for me looked at data set sizes and opened many folders to see how things worked.
    So if you want to have both pools, pay attention to exported (nfs, smb etc) mounts and

    Set 'srv' as readonly
    Export the two zpools
    Rename and import the original zpool as'zsrv-old'
    Rename an import the new zpool as 'zsrv'

    Once all is well, then zpool to rebuild using the old drives and required replacements. Again, I went from Raidz to mirrored stripes.

    As you likely know, automatic snapshots were added to ZFS on linux as an external module and looks like now part of the main build. (Please pardon my terminology as I do this for my home servers not professionally). So, in summary, it looks like this has been leading or errors or perhaps a bug with at times the newly created pool starting to make auto-snapshots and cause the receive to fail. It may be that since I was sending the top level dataset, with its children with one command, that brought this out. I have seen with the "broken pipe" people saying it worked better to send datasets by themselves and that might prevent some of these problem at the cost if extra steps.

    I share this with you as adding this information would be helpful to many others judging by the many posts and frustration I’ve seen. I hope this helps others know to shut off any auto-snapshots while doing the initial send with large datasets which seem to bring this out.


{ 6 } Trackbacks

  1. [...] Sending and Receiving Filesystems [...]

  2. [...] Sending and Receiving Filesystems [...]

  3. [...] Sending and Receiving Filesystems [...]

  4. [...] Sending and Receiving Filesystems [...]

  5. [...] Sending and Receiving Filesystems [...]

  6. [...] Sending and Receiving Filesystems [...]

Post a Comment

Your email is never published nor shared.