HowTo (state 04.12.2017)

The following guides are valid for CentOS 7.

Build Lustre

HowTo build a one-node Lustre file system with ZFS backend:

  1. Prepare System
    1. Install the kernel development tools
      yum -y groupinstall 'Development Tools'
      yum -y install epel-release
    2. Install additional dependencies
      yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed bison elfutils-devel  audit-libs-devel python-docutils sg3_utils expect attr lsof quilt libselinux-devel kernel-devel libyaml-devel
    3. Disable SELinux for older clients
      sed -i '/^SELINUX=/s/.*/SELINUX=disabled/' /etc/selinux/config 
  2. Prepare ZFS backend (follow the guide for packaged ZFS or go to this section for custom ZFS build)
    1. EPEL release
      yum -y install --nogpgcheck $URL/epel/zfs-release.el7.noarch.rpm 
    2. For the newest Lustre releases
      • Change /etc/yum.repos.d/zfs.repo to switch from dkms to kmod (more info here and here)
         name=ZFS on Linux for EL 7 - dkms
        @@ -9,7 +9,7 @@
         name=ZFS on Linux for EL 7 - kmod
    3. Install ZFS and its associated SPL packages
      • kmod packages for newer releases
        yum install -y zfs libzfs2-devel kmod-spl-devel kmod-zfs-devel 
      • dkms packages for older releases
        yum install -y zfs libzfs2-devel zfs-dkms 
  3. Build Lustre
    1. Get Lustre source code
      git clone git://
    2. Configure (--disable-ldiskfs for ZFS backend, --disable-server for client only)
      cd lustre-release/
      sh ./
      ./configure --disable-ldiskfs
    3. Make and install rpms
      make rpms
      yum -y install *.$(arch).rpm
  4. You may need to reboot and to explicitly load the ZFS and Lustre modules
    modprobe zfs
    modprobe lustre
  5. Format targets (change /tmp in this example to real devices or partitions)
    mkfs.lustre --mgs --backfstype=zfs --fsname=lustre --device-size=1048576 lustre-mgs/mgs /tmp/lustre-mgs
    mkfs.lustre --mdt --backfstype=zfs --fsname=lustre --index=0 --mgsnode=$(hostname)@tcp --device-size=1048576 lustre-mdt0/mdt0 /tmp/lustre-mdt0
    mkfs.lustre --ost --backfstype=zfs --fsname=lustre --index=0 --mgsnode=$(hostname)@tcp --device-size=1048576 lustre-ost0/ost0 /tmp/lustre-ost0
    1. Change /etc/ldev.conf
      hostname - mgs     zfs:lustre-mgs/mgs
      hostname - mdt0    zfs:lustre-mdt0/mdt0
      hostname - ost0    zfs:lustre-ost0/ost0
  6. Run Lustre
    1. Reconfigure the firewall to allow incoming connections on TCP port 988 (for socklnd only), or temporarily disable it
      systemctl stop firewalld
      systemctl disable firewalld 
    2. Start servers
      systemctl start lustre
    3. Mount client
      mkdir /mnt/lustre/client
      mount -t lustre $(hostname):/lustre /mnt/lustre/client

Debugging Lustre Source Code

HowTo debug Lustre:


The easiest way to print out a message or some content is the well known printf from userspace or printk from kernel space. Lustre defines several macros for printing out messages. The most usefull ones are CDEBUG and CERROR.

CDEBUG Prints message in debug log.
CDEBUG(D_INFO, "Debug message: rc=%d\n", number); 
CERROR Prints message in debug log and to console.
CERROR("Something bad happened: rc=%d\n", rc); 

Show debug log with lctl debug_kernel. CERROR messages are always printed; CDEBUG messages are printed depeding on level parameter (D_INFO in the example) and debugging level set iwth lctl.

Get debugging level with

 lctl get_param debug 

Set debugging level with

 lctl set_param debug="+info" 

More information can be found in the Lustre documentation.



HowTo build a custom ZFS:

  1. Prepare System (Example CentOS7)
    1. Disable SELinux
      sed -i '/^SELINUX=/s/.*/SELINUX=disabled/' /etc/selinux/config 
    2. Install the kernel development tools
      yum -y groupinstall 'Development Tools'
      yum -y install epel-release
    3. Install additional dependencies
      yum -y install parted lsscsi wget ksh
      yum -y install kernel-devel zlib-devel libattr-devel libuuid-devel libblkid-devel libselinux-devel libudev-devel
      yum -y install device-mapper-devel openssl-devel
  2. Clone both Git-repositories if you need 0.7 release or older (for newer SPL was merged in to the ZFS repository). See which ZFS version you need for Lustre in Lustre Support Matrix.
    git clone
    git clone
  3. Perform all the following steps for both directories (complete spl first)
    1. Configure for specific system
      cd <spl|zfs>
      ./configure --with-spec=redhat
    2. Build RPMs in both directories
      • kmod
        make pkg-utils pkg-kmod
    3. Install RPMs
      yum localinstall *.$(arch).rpm

Debugging ZFS Source Code

HowTo debug ZFS:


The easiest way to print an error message in kernel is the common printk function with KERN_ERR level. Some ZFS source files like dmu.c are built for user and kernel space and would cause compile errors when using printk. The following if statement avoids errors.

#ifdef _KERNEL
        printk(KERN_ERR "Error message \n");
research/projects/ipcc-l/howto.txt · Last modified: 2019-03-22 16:24 by Anna Fuchs