Toastys Rambles

In order to get me to write more and to simplify my blogging experience, I have a new Blog.

Ths Blog is now Powered by WriteFreely.

Follow me on the Fediverse via to for updates

When one of your servers has a ton of storage and the others all the RAM, then comes the time to share that storage via the network.

NFS is the first choice in most cases but as backing store of zones it has some drawbacks compared to ZFS. Unfortunately ZFS is not network shared like Ceph. But no problem ZFS is just a Filesystem and Volume Manager, it can be used ontop of any block storage. Thankfully exporting Block storage is easy in illumos with “scsi target mode framework (smtf)”. With it you can export any Block storage including of cource our beloved ZFS Volumes over the network. smtf also supports iSCSI. As iSCSI on OpenIndiana lacks guides I figured I would write up this post to guide people through the Process of setting up smtf with iSCSI.


We will have two nodes. A Target (Host/Server) with all the storage we could need and a Initiator (Client) where we want to run the zones.

Target Setup

Step 1 Create a LUN (Exported Disk)

First we need a block storage to export. Create a ZFS Volume for that. Please note that we will have to sync block sizes between stmf and zfs. Zfs by default uses 8K for Performance but stmf can only handle up to 4K. Thus we need to specify -o volblocksize=4K manually here. Otherwise you will get poor performance. I am planing on putting zones on the Lun so compression and deduplication here will count for the whole pool on the consumer/initiator side. Compression and dedup are up to you however. Use what you need. Only volblocksize is important.

zfs create -V 1000G -o volblocksize=4K -o compression=lz4 -o dedup=on rpool/zonelun0

This will create a Volume of one Terrabyte on the Rootpool with the name zonelun0. Now we need to let smtf know of the Block storage. Note the blk argument. this must be equal to the volblksize in the zfs volume.

stmfadm create-lu -p blk=4096 /dev/zvol/dsk/rpool/zonelun0

Now we can have a look at what smtf sees with list-lu. We should see something like the following.

~# stmfadm list-lu -v
LU Name: 600144F0AD85453600005B9188B40001
    Operational Status: Offline
    Provider Name     : sbd
    Alias             : /dev/zvol/dsk/rpool/zonelun0
    View Entry Count  : 0
    Data File         : /dev/zvol/dsk/rpool/zonelun0
    Meta File         : not set
    Size              : 1073741824000
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN     
    Product ID        : COMSTAR         
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

Note that our new LUN has a name of 600144F0AD85453600005B9188B40001. This will be used later on.

Step 2 Make LUN visible (View)

Now that we have a LUN we need to define who can see it. As we have no security needs atm we make it visible to everybody.

stmfadm add-view 600144F0AD85453600005B9188B40001 #Note the LUN name from step 1

~# stmfadm list-view -l 600144F0AD85453600005B9188B40001
View Entry: 0
    Host group   : All
    Target group : All
    LUN          : 0

Step 3 Create the Target

Now that we have the basic LUN setup we can create the iSCSI Target itself

First check if STMF service is running:

~# svcs -l stmf
fmri         svc:/system/stmf:default
name         STMF
enabled      true
state        online
next_state   none
state_time   September  7, 2018 at 12:09:11 AM CEST
logfile      /var/svc/log/system-stmf:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/system/filesystem/local:default (online)

if not enable it

svcadm enable stmf

Install and enable the iSCSI Target Service

pkg install network/iscsi/target

svcadm enable -r svc:/network/iscsi/target:default

create a new target

itadm create-target
~# itadm list-target -v
TARGET NAME                                                  STATE    SESSIONS online   0        
        alias:                  -
        auth:                   none (defaults)
        targetchapuser:         -
        targetchapsecret:       unset
        tpg-tags:               default

create target portal group (TGP) and stmf target group

itadm create-tpg iscsi01 #Note use your ip here

stmfadm create-tg iscsi-tg01

svcadm disable stmf
stmfadm add-tg-member -g iscsi-tg01
svcadm enable stmf

Initiator Setup

Step 1

Now that we have the target we just need to login into the target and get the LUN mapped.

iscsiadm add discovery-address

iscsiadm modify discovery -t enable

Now we have a list of targets exposed by our Server

~# iscsiadm list target
        Alias: -
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1

And we see our LUN

~# iscsiadm list target -S -v
        Alias: -
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
                CID: 0
                  IP address (Local):
                  IP address (Peer):
                  Discovery Method: SendTargets 
                  Login Parameters (Negotiated):
                        Data Sequence In Order: yes
                        Data PDU In Order: yes
                        Default Time To Retain: 20
                        Default Time To Wait: 2
                        Error Recovery Level: 0
                        First Burst Length: 65536
                        Immediate Data: yes
                        Initial Ready To Transfer (R2T): yes
                        Max Burst Length: 262144
                        Max Outstanding R2T: 1
                        Max Receive Data Segment Length: 32768
                        Max Connections: 1
                        Header Digest: NONE
                        Data Digest: NONE

        LUN: 0
             Vendor:  SUN     
             Product: COMSTAR         
             OS Device Name: /dev/rdsk/c0t600144F0AD85453600005B9188B40001d0s2

Step Format and Profit

zpool create zonespool c0t600144F0AD85453600005B9188B40001d0

And now have fun with your iSCSI backed ZFS Pool

This initial post to see if the theme renders properly and to say I have started blogging.