Comments on: ZFS Administration, Part XIII- Sending and Receiving Filesystems https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/ Linux. GNU. Freedom. Sun, 13 May 2018 18:21:35 +0000 hourly 1 https://wordpress.org/?v=5.0-alpha-43006 By: Alan Galitz MD https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-256264 Wed, 09 Dec 2015 04:27:17 +0000 http://pthree.org/?p=2910#comment-256264 While it has been over 2 years, perhaps a how to on migrating a large pool from one array to another might be worth including.

This link https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/ covers the basic steps, but I and others, judging by all the posts I found, hit several problems whose solutions are not well listed and thought since so many people use your site as a reference this might help. Perhaps this issue is cropping up with the addition and changes to the auto-snapshot implementation in zfs on linux.

I have several home servers with 4-5TB if data each. They store identical copies of media, crashplan backup files, desktop/laptop files (yes the crashplan goes off site too, but the media is too large so 3 separate machines with redundant zfs is the backup plan for some the the large media). The send operation is the better part of a day.

A few years ago I followed your excellent advice -THANKS- and setup Raidz with 3 x4TB on 2 machines and 4x3TB on one. Well now it turns out that this no longer best practice so after some drives began failing from age - no data loss- yea ZFS scrub! - time to covert to mirrors and stripes. So with some dual USB3 toasters, I build a 7TB array with mirrors 2x2 to temp store the data. So time to send the data to the temp array:

# zfs snapshot -r zsrv@01
# zfs send -R zsrv@01 | zfs receive -Fdvu zsrv-old

Note, don't keep your terminal open while doing this as a sleep/hibenation might end the operation. Detach from the session to resume later. I use byobu since it comes with xubuntu.

OK, to avoid massive fail, I found it necessary to turn OFF auto-shapshots. The new dataset started doing frequent snapshots and their presence killed the send/recv randomly. Perhaps this is a bug, the the first send/recv ( no "-i") doesn't like a pre-exiisting data set)

The important error message was just before a long string of "...broken pipe..." in SSH session. I found it at the machine's display since the volume blew past the SSH buffer. The 'v' switch on the recv produces alot of output if you have hundreds of old snapshots.

Also, since I was sending nfs shares, both the old and new datasets used the same folder, causing mounting errors, but that's later. I set the the top level dataset with the pool name to not mount or send shares but since the send -R causes properties to be copied the new pools switch them back on. The -u option is supposed to stop local mounting, but the nfs shares mounted locally anyway it seems. I'm not a pro so do not have the time to test much or know.

It did not help that without the nfs share mounted by zfs, crashplan periodically would create a local folder, blocking the creation of the backup target dataset. So before the first full stream, check the usualy mounting place to ensure they really are empty and

# zfs list -t snapshot | grep new # or a unique piece of the new pool

OK, so for the final incremental send, ensure the disk isn't in use (shutdown iSCSI, NFS, syslog, crashplan,etc) see above
Make another snapshot of 'zsrv' and incrementally send it to 'zsrv-new' (this should take a short time)

# zfs snapshot -r zsrv@02
# zfs send -R -i zsrv@01 zsrv@02 | zfs recv -dvu zsrv-new

Now the instructions included a way to bring back the old pool and do some comparison. for me looked at data set sizes and opened many folders to see how things worked.
So if you want to have both pools, pay attention to exported (nfs, smb etc) mounts and

Set 'srv' as readonly
Export the two zpools
Rename and import the original zpool as'zsrv-old'
Rename an import the new zpool as 'zsrv'

Once all is well, then zpool to rebuild using the old drives and required replacements. Again, I went from Raidz to mirrored stripes.

As you likely know, automatic snapshots were added to ZFS on linux as an external module and looks like now part of the main build. (Please pardon my terminology as I do this for my home servers not professionally). So, in summary, it looks like this has been leading or errors or perhaps a bug with at times the newly created pool starting to make auto-snapshots and cause the receive to fail. It may be that since I was sending the top level dataset, with its children with one command, that brought this out. I have seen with the "broken pipe" people saying it worked better to send datasets by themselves and that might prevent some of these problem at the cost if extra steps.

I share this with you as adding this information would be helpful to many others judging by the many posts and frustration I’ve seen. I hope this helps others know to shut off any auto-snapshots while doing the initial send with large datasets which seem to bring this out.

cheers

]]>
By: Aaron Toponce https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-227902 Tue, 17 Feb 2015 20:35:52 +0000 http://pthree.org/?p=2910#comment-227902

The incremental updates suggested by Broen van Besien are a big plus. This could enable a portable/carryable system: send to laptop in the morning, work on it, send back to desktop when arriving home. Feasible?

Sure. I send incremental snapshots from my workstation to my external backup drive every 15 minutes. Works like a charm. I don't see any problems with this for a laptop going to a backup server.

]]>
By: NM https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-227048 Thu, 05 Feb 2015 00:48:59 +0000 http://pthree.org/?p=2910#comment-227048 The incremental updates suggested by Broen van Besien are a big plus. This could enable a portable/carryable system: send to laptop in the morning, work on it, send back to desktop when arriving home.
Feasible?

]]>
By: Broen van Besien https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-224310 Sun, 11 Jan 2015 18:47:25 +0000 http://pthree.org/?p=2910#comment-224310 Sent and receive of entire snapshots is a big win indeed.
And it's even more powerful:
You can send incremental updates out-of-the-box by using the -i option, which will only send the difference between a current and a previous snapshot.

]]>
By: syed https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-130307 Mon, 28 Oct 2013 18:57:41 +0000 http://pthree.org/?p=2910#comment-130307 Thanks, really appreciate it. Was very helpful.

]]>
By: Warren Downs https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-127208 Mon, 01 Jul 2013 23:40:29 +0000 http://pthree.org/?p=2910#comment-127208 Regarding 'no where in this post do I state that “-o” is valid for “zfs send” or “zfs receive”', you may wish to change the above post to make that statement true (quoting original post):

Further, you can change dataset properties on the receiving end. If the sending filesystem was not compressed, but you wish to compress the data on the receiving side, you can enable that as follows:

# zfs send tank/test@tuesday | ssh user@server.example.com "zfs receive -o compression=lzjb pool/test"

]]>
By: Aaron Toponce : ZFS Administration, Appendix A- Visualizing The ZFS Intent LOG (ZIL) https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124828 Fri, 19 Apr 2013 11:03:08 +0000 http://pthree.org/?p=2910#comment-124828 [...] Sending and Receiving Filesystems [...]

]]>
By: max https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124801 Fri, 19 Apr 2013 09:03:17 +0000 http://pthree.org/?p=2910#comment-124801 Thanks a lot.

]]>
By: Aaron Toponce https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124800 Fri, 19 Apr 2013 08:27:20 +0000 http://pthree.org/?p=2910#comment-124800 "zfs create" has the -o switch for setting dataset parameters. "zfs send" and "zfs receive" do not have this switch, and no where in this post do I state that "-o" is valid for "zfs send" or "zfs receive". If you run "zfs get compress pool/dataset" on your dataset, and "gzip" is returned, then when using "zfs send" you need to use the " -p" switch:

-p

Include the dataset's properties in the stream. This flag is implicit when -R is specified. The receiving system must also support this feature.

]]>
By: max https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124797 Fri, 19 Apr 2013 05:25:05 +0000 http://pthree.org/?p=2910#comment-124797 >>According to the zfs(8) manpage, “zfs receive -o” is not a valid option.

Yes, but in your post was written about this feature.

>> By default, ZFS saves dataset settings when sending them to another location.

Hm... I tried to send about 280GB from gziped dataset to another pool yesterday. The pool was about 320GB free space. This was epic fail for me because compression wasn't automatically set on destination dataset.

]]>
By: Aaron Toponce https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124789 Thu, 18 Apr 2013 15:31:00 +0000 http://pthree.org/?p=2910#comment-124789 According to the zfs(8) manpage, "zfs receive -o" is not a valid option. "zfs receive | recv [-vnFu] filesystem|volume|snapshot" or "zfs receive | recv [-vnFu] [-d|-e] filesystem".

By default, ZFS saves dataset settings when sending them to another location. If you want to change a setting, then you will need to run a separate "zfs set" command after the sending of the dataset has finished.

]]>
By: max https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124788 Thu, 18 Apr 2013 14:32:55 +0000 http://pthree.org/?p=2910#comment-124788 Thanks for zfs guide. Its very helpfull.
But I have a trouble with send-receive.
I have two pools in my system. I tried to send snapshot from first pool to second. Firest dataset had compression property.
I tried to execute command such as:
# zfs send pool750/data@01 | zfs receive -o compression=gzip pool250

But system said:

invalid option 'o'
usage:
receive [-vnFu]
receive [-vnFu] [-d | -e]

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow

I have last version of ubuntu-zfs package.
What is wrong?

P.S. Sorry for my english

]]>
By: Aaron Toponce : ZFS Administration, Part VII- Zpool Properties https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124249 Wed, 20 Feb 2013 15:34:00 +0000 http://pthree.org/?p=2910#comment-124249 [...] Sending and Receiving Filesystems [...]

]]>
By: Niek Bergboer https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-124231 Tue, 12 Feb 2013 19:26:09 +0000 http://pthree.org/?p=2910#comment-124231 Snapshots, and the ability to send and receive them, makes for automatic incremental backups: http://freecode.com/projects/zrep uses this, and you can trivially do hourly (or even more often) filesystem replication, keeping all snapshots.

]]>
By: Aaron Toponce : ZFS Administration, Part I- VDEVs https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-122323 Sat, 29 Dec 2012 13:19:54 +0000 http://pthree.org/?p=2910#comment-122323 [...] Sending and Receiving Filesystems [...]

]]>
By: Aaron Toponce : Install ZFS on Debian GNU/Linux https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-122034 Thu, 20 Dec 2012 15:10:04 +0000 http://pthree.org/?p=2910#comment-122034 [...] Sending and Receiving Filesystems [...]

]]>
By: Aaron Toponce : ZFS Administration, Part V- Exporting and Importing zpools https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-122028 Thu, 20 Dec 2012 15:08:17 +0000 http://pthree.org/?p=2910#comment-122028 [...] Sending and Receiving Filesystems [...]

]]>
By: Aaron Toponce : ZFS Administration, Part XI- Compression and Deduplication https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/#comment-122023 Thu, 20 Dec 2012 15:06:50 +0000 http://pthree.org/?p=2910#comment-122023 [...] Sending and Receiving Filesystems [...]

]]>