Image of the glider from the Game of Life by John Conway
Skip to content

ZFS Administration, Part XV- iSCSI, NFS and Samba

Table of Contents

Zpool Administration ZFS Administration Appendices
0. Install ZFS on Debian GNU/Linux 9. Copy-on-write A. Visualizing The ZFS Intent Log (ZIL)
1. VDEVs 10. Creating Filesystems B. Using USB Drives
2. RAIDZ 11. Compression and Deduplication C. Why You Should Use ECC RAM
3. The ZFS Intent Log (ZIL) 12. Snapshots and Clones D. The True Cost Of Deduplication
4. The Adjustable Replacement Cache (ARC) 13. Sending and Receiving Filesystems
5. Exporting and Importing Storage Pools 14. ZVOLs
6. Scrub and Resilver 15. iSCSI, NFS and Samba
7. Getting and Setting Properties 16. Getting and Setting Properties
8. Best Practices and Caveats 17. Best Practices and Caveats

I spent the previous week celebrating the Christmas holiday with family and friends, and as a result, took a break from blogging. However, other than the New Year, I'm finished with holidays for a while, and eager to get back to blogging, and finishing off this series. Only handful of posts left to go. So, let's continue our discussion with ZFS administration on GNU/Linux, by discussing sharing datasets.


I have been trying to keep these ZFS posts as operating system agnostic as much as possible. Even though they have had a slant towards Linux kernels, you should be able to take much of this to BSD or any of the Solaris derivatives, such as OpenIndiana or Nexenta. With this post, however, it's going to be Linux-kernel specific, and even Ubuntu and Debian specific at that. The reason being is iSCSI support is not compiled into ZFS on Linux as of the writing of this post, but sharing via NFS and SMB is. Further, the implementation details for sharing via NFS and SMB will be specific to Debian and Ubuntu in this post. So, you may need to make adjustments if using Fedora, openSUSE, Gentoo, Arch, et cetera.


You are probably asking why you would want to use ZFS specific sharing of datasets rather than using the "tried and true" methods with standard software. The reason is simple. When the system boots up, and goes through its service initialization process (typically by executing the shell scripts found in /etc/init.d/), it has a method to the madness on which service starts first. Loosely speaking, filesystems are mounted first, networking is enabled, then services are started at last. Some of these are tied together, such as NFS exports, which requires the filesystem to be mounted, a firewall in place, networking started, and the NFS daemon running. But, what happens when the filesystem is not mounted? If the directory is still accessible, it will be exported via NFS, and applications could begin dumping data into the export. This could lead to all sorts of issues, such as data inconsistencies. As such, administrators have put checks into place, such as exporting only nested directories in the mount point, which would not be available if the filesystem fails to mount. These are clever hacks, but certainly not elegant.

When tying the export directly into the filesystem, you can solve this beautifully, which ZFS does. In the case of ZFS, you can share a specific dataset via NFS, for example. However, if the dataset does not mount, then the export will not be available to the application, and the NFS client will block. Because the network share is inherent to the filesystem, there is no concern for data inconsistencies, and no need for silly check hacks or scripts. As a result, ZFS from Oracle has the ability to share a dataset via NFS, SMB (CIFS or Samba) and iSCSI. ZFS on Linux only supports NFS and SMB currently, with iSCSI support on the way.

In each case, you still must install the necessary daemon software to make the share available. For example, if you wish to share a dataset via NFS, then you need to install the NFS server software, and it must be running. Then, all you need to do is flip the sharing NFS switch on the dataset, and it will be immediately available.

Sharing via NFS

To share a dataset via NFS, you first need to make sure the NFS daemon is running. On Debian and Ubuntu, this is the "nfs-kernel-server" package. Further, with Debian and Ubuntu, the NFS daemon will not start unless there is an export in the /etc/exports file. So, you have two options: you can create a dummy export, only available to localhost, or you can edit the init script to start without checking for a current export. I prefer the former. Let's get that setup:

$ sudo aptitude install -R nfs-kernel-server
$ echo '/mnt localhost(ro)' >> /etc/exports
$ sudo /etc/init.d/nfs-kernel-server start
$ showmount -e
Export list for
/mnt localhost

With our NFS daemon running, we can now start sharing ZFS datasets. I'll assume already that you have created your dataset, it's mounted, and you're ready to start committing data to it. You'll notice in the zfs(8) manpage, that for the "sharenfs" property, it can be "on", "off" or "opts", where "opts" are valid NFS export options. So, if I wanted to share my "pool/srv" dataset, which is mounted to "/srv" to the network, I could do something like:

# zfs set sharenfs="rw=@" pool/srv
# zfs share pool/srv
# showmount -e
Export list for
/mnt localhost

If you want your ZFS datasets to be shared on boot, then you need to install the /etc/default/zfs config file. If using the Ubuntu PPA, this will be installed by default for you. If compiling from source, this will not be provided. Here are the contents of that file. I've added emphasis to the two lines that should be modified for persistence across boots, if you want to enable sharing via NFS. Default is 'no':

$ cat /etc/default/zfs
# /etc/default/zfs
# Instead of changing these default ZFS options, Debian systems should install
# the zfs-mount package, and Ubuntu systems should install the zfs-mountall
# package. The debian-zfs and ubuntu-zfs metapackages ensure a correct system
# configuration.
# If the system runs parallel init jobs, like upstart or systemd, then the
# `zfs mount -a` command races in a way that causes sporadic mount failures.

# Automatically run `zfs mount -a` at system start. Disabled by default.

# Automatically run `zfs share -a` at system start. Disabled by default.
# Requires nfsd and/or smbd. Incompletely implemented for Linux.

As mentioned in the comments, running a parallel init system creates problems for ZFS. This is something I recently banged my head against, as my /var/log/ and /var/cache/ datasets were not mounting on boot. To fix the problem, and run a serialized boot, thus ensuring that everything gets executed in the proper order, you need to touch a file:

# touch /etc/init.d/.legacy-bootordering

This will add time to your bootup, but given the fact that my system is up months at a time, I'm not worried about the extra 5 seconds this puts on my boot. This is documented in the /etc/init.d/rc script, setting the "CONCURRENCY=none" variable.

You should now be able to mount the NFS export from an NFS client:

(client)# mount -t nfs /mnt

Sharing via SMB

Currently, SMB integration is not working 100%. See bug #1170 I reported on Github. However, when things get working, this will likely be the way.

As with NFS, to share a ZFS dataset via SMB/CIFS, you need to have the daemon installed and running. Recently, the Samba development team released Samba version 4. This release gives the Free Software world a Free Software implementation of Active Directory running on GNU/Linux systems, SMB 2.1 file sharing support, clustered file servers, and much more. Currently, Debian testing has the beta 2 packages. Debian experimental has the stable release, and it may make its way up the chain for the next stable release. One can hope. Samba v4 is not needed to share ZFS datasets via SMB/CIFS, but it's worth mentioning. We'll stick with version 3 of the Samba packages, until version 4 stabilizes.

# aptitude install -R aptitude install samba samba-client samba-doc samba-tools samba-doc-pdf
# ps -ef | grep smb
root     22413     1  0 09:05 ?        00:00:00 /usr/sbin/smbd -D
root     22423 22413  0 09:05 ?        00:00:00 /usr/sbin/smbd -D
root     22451 21308  0 09:06 pts/1    00:00:00 grep smb

At this point, all we need to do is share the dataset, and verify that it's been shared. It is also worth noting that Microsoft Windows machines are not case sensitive, as things are in Unix. As such, if you are in a heterogenious environment, it may be worth disabling case sensitivity on the ZFS dataset. Setting this value can only be done on creation time. So, you may wish to issue the following when creating that dataset:

# zfs create -o casesensitivity=mixed pool/srv

Now you can continue with configuring the rest of the dataset:

# zfs set sharesmb=on pool/srv
# zfs share pool/srv
# smbclient -U guest -N -L localhost
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.6.6]

        Sharename       Type      Comment
        ---------       ----      -------
        print$          Disk      Printer Drivers
        sysvol          Disk      
        netlogon        Disk      
        IPC$            IPC       IPC Service (eightyeight server)
        Canon-imageRunner-3300 Printer   Canon imageRunner 3300
        HP-Color-LaserJet-3600 Printer   HP Color LaserJet 3600
        salesprinter    Printer   Canon ImageClass MF7460
        pool_srv        Disk      Comment: /srv
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.6.6]

        Server               Comment
        ---------            -------
        EIGHTYEIGHT          eightyeight server

        Workgroup            Master
        ---------            -------
        WORKGROUP            EIGHTYEIGHT

You can see that in this environment (my workstation hostname is 'eightyeight'), there are some printers being shared, and a couple disks. I've emphasized the disk that we are sharing in my output, to verify that it is working correctly. So, we should be able to mount that share as a CIFS mount, and access the data:

# aptitude install -R cifs-utils
# mount -t cifs -o username=USERNAME //localhost/srv /mnt
# ls /mnt

Sharing via iSCSI

Unfortunately, sharing ZFS datasets via iSCSI is not yet supported with ZFS on Linux. However, it is available in the Illumos source code upstream, and work is being done to get it working in GNU/Linux. As with SMB and NFS, you will need the iSCSI daemon installed and running. When support is enabled, I'll finish writing up this post on demonstrating how you can access iSCSI targets that are ZFS datasets. In the meantime, you would do something like the following:

# aptitude install -R openiscsi
# zfs set shareiscsi=on pool/srv

At which point, from the iSCSI client, you would access the target, format it, mount it, and start working with the data..

{ 12 } Comments

  1. David E. Anderson | December 31, 2012 at 8:29 am | Permalink

    thanks again for the series of docs and in particular the boot-spanning and serial tips!

  2. Roger Hunwicks | January 16, 2013 at 3:42 am | Permalink

    Thanks for the great series of posts - you've encouraged me to finally set up a ZFS server for experimenting with.

    On my Ubuntu 12.04.1 server I didn't need to create a dummy export in order to get NFS shares working - the supplied /etc/init.d/nfs-kernel-server checks for the existing of /etc/exports, but doesn't care what is in. The file exists by default but contains only comments, which is enough.

    Similarly, I am using the mountall from the Ubuntu ZFS PPA and that seems be working fine without changing ZFS_MOUNT in /etc/default/zfs

    Therefore the only change I needed was to change ZFS_SHARE to yes in /etc/default/zfs

  3. Roger Hunwicks | January 16, 2013 at 3:52 am | Permalink

    FWIW I am also not using .legacy-bootordering and I haven't had any problems. That may be because the zfs pool I am using is all shares (NFS and SMB) and virtual machines (run using libvirt/KVM on the same server), and I am not using ZFS for the OS files (on a separate ext4 mdadm mirror)

  4. Aaron Toponce | January 16, 2013 at 8:51 am | Permalink

    That's good to know about just the existence of the /etc/exports file, instead of its contents. Thanks for the heads up. Also, with regards to legacy booting, it's only an issue if using it for filesystem mounts, such as /var/, /home/, etc. Things outside of that, such as /kvm/ may or may not be affected, all depending on when your starting services need the mounts.

  5. Lloyd | June 21, 2013 at 9:29 pm | Permalink

    Great work!

    For Ubuntu 13.04 it looks like you do need a dummy export. Without it this happens.

    lloyd@tank:/var/log$ sudo /etc/init.d/nfs-kernel-server start
    * Not starting NFS kernel daemon: no exports.

  6. Fernando | February 20, 2014 at 1:17 pm | Permalink

    Hi there!

    I would like to share a ZFS dataset via SMB.

    At the end of the SMB section, you mounting the SMB share with: "mount -t cifs -o username=USERNAME //localhost/srv /mnt"

    Which username should I use here? My linux username? I'm asking because with my linux username I could not authenticate. Do I need to edit smb.conf in order to authenticate?



  7. Fernando | February 21, 2014 at 8:23 am | Permalink

    Hi guys!

    Nevermind my last question.. I solved my problem with:

    sudo smbpasswd -L -a your_username
    sudo smbpasswd -L -e your_username



  8. Mark | March 12, 2014 at 4:42 am | Permalink

    "Unfortunately, sharing ZFS datasets via iSCSI is not yet supported with ZFS on Linux"

    Doesn't this just refer to the meta data on the volume? If you create a zvol and hence a block device in the kernel, can't you just tell the IET (the iscsi target) to use that block device.

    Other than convenience, what other benefits are there for using the zfs command to create iscsi (and for that matter smb) shares?

  9. Chris Scott | August 9, 2014 at 8:09 am | Permalink

    How can I remove a NFS share I've created? I shared my entire ZFS drive (as per but now I think I'd prefer to share out only the folders I create within the ZFS pool.

    Can't figure out how to remove my ZFS shares altogether, though (apart from setting them to "off")

  10. Mike | September 29, 2014 at 12:58 pm | Permalink

    Thanks for this series. It's been really helpful.

    Can someone point me to the zfs config file in CentOS7? I've installed zfs from the EPEL repository, but I don't have /etc/default/zfs. I don't see anything similar in /etc/zfs/ either.

  11. NM | May 12, 2015 at 5:40 am | Permalink

    You closed bug #1170

  12. Doug | December 21, 2015 at 2:06 pm | Permalink

    Re: SMB sahres

    I can't get this to work with the instructions. SMB is running. When I go to set the smbshare=on, I get :

    cannot share 'mypool/test2': smb add share failed

    But when I check it, it does turn it on:

    zfs get sharesmb mypool/test2
    mypool/test2 sharesmb on local

    But then telling it to share it:

    zfs share mypool/test2
    cannot share 'mypool/test2': smb add share failed

    But running smbclient it does not show up as a share and therefore trying to mount with the smbclient fails.

{ 7 } Trackbacks

  1. [...] iSCSI, NFS and Samba [...]

  2. [...] iSCSI, NFS and Samba [...]

  3. [...] iSCSI, NFS and Samba [...]

  4. [...] iSCSI, NFS and Samba [...]

  5. [...] iSCSI, NFS and Samba [...]

  6. [...] iSCSI, NFS and Samba [...]

  7. [...] iSCSI, NFS and Samba [...]

Post a Comment

Your email is never published nor shared.