A faster memstat for mdb

I have implemented a version of the ::memstat dcmd for mdb that gives results in less than half the time of the ::memstat currently in mdb. If you are interested, it is available for download here.

So, how does it work? The current version of memstat uses the page walker (::walk page) to walk all cached pages in the system. The new version simply examines all pages.

Each page-able page in the system is represented by a page_t structure. Basically all of memory except for the unix and genunix kernel modules, and a few other odds and ends is considered "page-able". Every page, that is in use (either by the kernel, user process, anonymous, or cached) has an identity. This is a vnode/offset pair. The identify uniquely determines how the page is being used. For instance, a page for the code of a running bash process will have the vnode_t for /usr/bin/bash, and the offset within /usr/bin/bash of where the page comes from. For a kernel page, there is a special vnode_t (kvp). For anonymous space, the page has a swapfsvnode (also used for shared memory and tmpfs files). When a process gets a page fault, the fault handling code first checks to see if the faulting address is mapped in the process' address space. If not, a segmentation violation (SIGSEGV) is sent to the process. If the address is within the address space, the fault handling code sees if the page is already in memory. It does this by retrieving the vnode/offset for the faulting page, and hashing into an array called page_hash. Each entry in page_hash is the beginning of a linked list of page_t structures. So the fault handling code does a hash to get a page_hash array entry, then walks the page_t structures starting at that entry to look for a matching vnode_t/offset. If the page_t is in the hash, the fault handling code sets up a page table entry (translation table entry on SPARC) to map to the corresponding physical page.

The page_hash array is sized so the the average search, given a page_hash bucket, is no longer than 4 (PAGE_HASHAVELEN in vm/page.h) entries. This makes searching for a cached page fairly fast.

The page walker that mdb uses to do memstat walks every hash bucket looking for pages. Basically, if the page is found from a hash bucket, the page is either in use by the kernel, some process, or tmpfs, or, the page is free but cached. Any page not hanging off of a page_hash bucket is considered free (i.e., the free (freelist) statistic).

The new memstat takes a different approach. Rather than scanning each hash bucket for pages, it simply reads all of the page_t structures on the system, then examines each one to determine if it is a kernel page, executable page, anonymous page, and so on. Any page_t that does not have a vnode_t is considered a free page, and is counted in the free (freelist) statistic.

How are the page_t structures found on the system? There is a linked list of memseg structures that are bookkeeping for page-able page structures. The list is headed by a pointer, memsegs, and is built early on in the startup code when the system is booting. I suspect the list could change due to dynamic reconfiguration events, but I'll leave that as an exercise for the reader...
To see the list, you can do "::memseg_list" in mdb. On my system, this gives:


> ::memseg_list
ADDR PAGES EPAGES BASE END
fbe00028 fbe94160 fe652260 00000c00 0007fed0
fbe00014 fbe85160 fbe94160 00000100 00000400
fbe00000 fbe82050 fbe85160 00000002 0000009f
>


The ADDR column is the address of a memseg structure. PAGES is the address of the first page_t in an array. EPAGES points to the end of the page_t array. BASE is the starting page frame number of the memseg, and END is the ending page frame number. So, on my system, page frames between 2 and 9f, 100 and 400, and c00 and 7fed0 have page_t structures. Physical page 0 and 1 are not in the list. Also pages between 9f and 100, and between 400 and c00 are not in the list. This is either because the physical memory does not exist, or it is not considered pageable.

The new memstat uses the memseg list to read in all of the page_t structures. On my system, this means 3 read calls (though the read from fbe94160 to fe652260 is quite large), versus thousands of read calls in the existing memstat via the page walker. The new memstat assumes there will never be more than 256 memseg structures. (This was arbitrarily chosen. I have not seen machines with more than 6 memseg structures, but I don't get on very large machines very often). A more correct way would be to build the memseg list within the dcmd, but I am lazy.

Using dtrace and counting system calls during running of the two version of memstat shows that the original memstat makes 1741225 system calls, while the new memstat makes 737344. So over 10000 fewer system calls in the new memstat.

I think memstat performance could be improved even more by using mmap to map in the page arrays. Then there would be no need for using mdb_alloc, and no need to mdb_vread the page_t structs.

Update to bruningsystems.com website

I have added a section called "articles", which has links to various articles, presentations, and some course materials on OpenSolaris. You can see it here.

recovering removed file on zfs disk

I have used my modified mdb and zdb (see
http://www.osdevcon.org/2008/files/osdevcon2008-proceedings.pdf and
http://www.osdevcon.org/2008/files/osdevcon2008-max.pdf)
to recover a file that was removed from a zfs file system.
The technique is to locate the active uberblock_t after the file
was created, but before the file was removed, and follow the data
structures from that uberblock_t. This technique would probably not
work on a near full file system, and probably not on a very busy file
system, but it works here. Also, this will not work with RAID-z,
but works fine for mirrors. (I shall get around to figuring out
raid-z, but not now...).

It is possible to follow all of the steps and still not have the right
data because you chose the wrong uberblock_t, or one of the blocks containing
metadata (or the data itself) has been re-used.

The modified mdb and zdb have been updated to work with Nevada,
build 94. It took about 15 minutes to merge the versions I was using
for build 79 into build94. For source for the changes, and
the zfs dmod, send mail to me at max@bruningsystems.com.

It might be possible, with a bit more clever use of mdb
and some shell scripting, to automate this... Also, it might be
useful to add an option to zdb so that a different transaction
id other than the active one be used for it's traversals.
Then you might be able to do everything using zdb.

The following describes the steps taken.

First, I copy a file with known contents to the zfs file system.

# cp /usr/dict/words /zfs_fs/words
#

We'll get the object id (inumber) for /zfs_fs. We'll use it later.

# ls -aid /zfs_fs
         3 /zfs_fs
#

Next, I'll try to make sure everything is on the disk.

# sync
#

Now, I'll use zdb to get the root blkptr from the uberblock.
This will also give me a transaction ID. Generally, you would not
use zdb to get the uberblock_t every time that you add/remove a
file to a zfs file system. That is ok. I have written a dcmd
(output shown below), that walks the uberblock_t array on disk.
Then you can, by trial and error, locate the uberblock_t you need
(assuming it still exists in the array, and assuming the metadata
it points to has not been re-used for another purpose).

# ./zdb -uuu zfs_fs
Uberblock

 magic = 0000000000bab10c
 version = 11
 txg = 1282  <-- transaction id in decimal
 guid_sum = 8876692711396000182
 timestamp = 1218963748 UTC = Sun Aug 17 11:02:28 2008
 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:11a00:200>
            DVA[1]=<0:c010e00:200> DVA[2]=<0:18008e00:200> fletcher4
            lzjb LE contiguous birth=1282 fill=27
        cksum=81f780ec5:361b52dda06:b6f3f410036f:1a2b8b10bfdb5c

Next, I'll remove the file I just created.

# rm /zfs_fs/words 
#

Let's take a look at the active uberblock_t.

# ./zdb -uuu zfs_fs
Uberblock

 magic = 0000000000bab10c
 version = 11
 txg = 1282  <-- nothing has changed
 guid_sum = 8876692711396000182
 timestamp = 1218963748 UTC = Sun Aug 17 11:02:28 2008
 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:11a00:200> 
            DVA[1]=<0:c010e00:200> DVA[2]=<0:18008e00:200> fletcher4
            lzjb LE contiguous birth=1282 fill=27 
        cksum=81f780ec5:361b52dda06:b6f3f410036f:1a2b8b10bfdb5c

Let's try to make sure it is on the disk.

# sync
#

And check the active uberblock_t again.

# ./zdb -uuu zfs_fs
Uberblock

 magic = 0000000000bab10c
 version = 11
 txg = 1284  <-- new transaction id, after file was removed
 guid_sum = 8876692711396000182
 timestamp = 1218963808 UTC = Sun Aug 17 11:03:28 2008
 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:15200:200> 
            DVA[1]=<0:c014600:200> DVA[2]=<0:1800a000:200> fletcher4
            lzjb LE contiguous birth=1284 fill=27 
        cksum=87431704e:37f154f58d7:bbddb76e9703:1aaf346847004f

Now, let's make sure nothing changes in the file system.

# zfs umount zfs_fs
#

And look at the active uberblock_t again.

# ./zdb -uuu zfs_fs
Uberblock

 magic = 0000000000bab10c
 version = 11
 txg = 1284  <-- ok.  nothing changed
 guid_sum = 8876692711396000182
 timestamp = 1218963808 UTC = Sun Aug 17 11:03:28 2008
 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:15200:200> 
            DVA[1]=<0:c014600:200> DVA[2]=<0:1800a000:200> fletcher4 
            lzjb LE contiguous birth=1284 fill=27 
        cksum=87431704e:37f154f58d7:bbddb76e9703:1aaf346847004f

Ok. So nothing changed when the file system was unmounted.
Now, we'll use the modified mdb to examine the uberblock_t array on disk.
The uberblock_t we want has transaction id 1282 decimal.

# ./mdb /export/home/max/zfsfile 

First, convert decimal 1282 to hex.

> 0t1282=X
                502             

Now, load kernel CTF and a few dcmds that work with zfs on disk.

> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so 

Walk the uberblock_t array on disk. This shows all possible 1024 uberblocks.
Here, we'll only show the entry with ub_txg = 0x502. Again, if I had not
retrieved the value of the active uberblock_t after the file was created,
and before the file was removed, I could dump all uberblock_t using the
following command, and then searched backwards, trying all transaction ids
that are less than the current (i.e., current after the file was removed
and the file system unmounted).

> ::walk uberblock | ::print -a -t zfs`uberblock_t
...
{
    20800 uint64_t ub_magic = 0xbab10c
    20808 uint64_t ub_version = 0xb
    20810 uint64_t ub_txg = 0x502  <-- the correct transaction id
    20818 uint64_t ub_guid_sum = 0x7b3058fd830ec1b6
    20820 uint64_t ub_timestamp = 0x48a7e924
    20828 blkptr_t ub_rootbp = {  <-- blkptr is at 0x20828 on disk
        20828 dva_t [3] blk_dva = [
            {
                20828 uint64_t [2] dva_word = [ 0x1, 0x8d ]
            }
            {
                20838 uint64_t [2] dva_word = [ 0x1, 0x60087 ]
            }
            {
                20848 uint64_t [2] dva_word = [ 0x1, 0xc0047 ]
            }
        ]
        20858 uint64_t blk_prop = 0x800b070300000001
        20860 uint64_t [3] blk_pad = [ 0, 0, 0 ]
        20878 uint64_t blk_birth = 0x502
        20880 uint64_t blk_fill = 0x1b
        20888 zio_cksum_t blk_cksum = {
            20888 uint64_t [4] zc_word = [ 0x81f780ec5, 0x361b52dda06, 
0xb6f3f410036f, 0x1a2b8b10bfdb5c ]
        }
    }
}
...

Let's dump the blkptr_t for this uberblock_t.

> 20828::blkptr
DVA[0]: vdev_id 0 / 11a00
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[0]: :0:11a00:200:d
DVA[1]: vdev_id 0 / c010e00
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[1]: :0:c010e00:200:d
DVA[2]: vdev_id 0 / 18008e00
DVA[2]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[2]: :0:18008e00:200:d
LSIZE:  400                             PSIZE: 200
ENDIAN: LITTLE                                  TYPE:  DMU objset
BIRTH:  502                LEVEL: 0     FILL:  1b00000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  81f780ec5:361b52dda06:b6f3f410036f:1a2b8b10bfdb5c
$q
#

Now, using the modified zdb, let's dump the mos objset_phys_t.

# ./zdb -R zfs_fs:0:11a00:200:d,lzjb,400 2> /tmp/mos
Found vdev: /export/home/max/zfsfile
#

Back to mdb to examine the objset_phys_t for the meta object set (mos).

# ./mdb /tmp/mos
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so 
> 0::print -a -t zfs`objset_phys_t
{
    0 dnode_phys_t os_meta_dnode = {
        0 uint8_t dn_type = 0xa  <-- this is DMU_OT_DNODE
 ...
        40 blkptr_t [1] dn_blkptr = [
            {
                40 dva_t [3] blk_dva = [
                    {
                        40 uint64_t [2] dva_word = [ 0x5, 0x88 ]
                    }
                    {
                        50 uint64_t [2] dva_word = [ 0x5, 0x60082 ]
                    }
                    {
                        60 uint64_t [2] dva_word = [ 0x5, 0xc0042 ]
                    }
                ]
                70 uint64_t blk_prop = 0x800a07030004001f
                78 uint64_t [3] blk_pad = [ 0, 0, 0 ]
                90 uint64_t blk_birth = 0x502
                98 uint64_t blk_fill = 0x1a
                a0 zio_cksum_t blk_cksum = {
                    a0 uint64_t [4] zc_word = [ 0xa9af50f215, 0xec01e192b95e, 
0xc523efad092ebc, 0x7a3a8be19416f454 ]
                }
            }
        ]
 ...

And dump the blkptr_t in the objset_phys_t.

> 40::blkptr
DVA[0]: vdev_id 0 / 11000
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: a0000000000
DVA[0]: :0:11000:a00:d
DVA[1]: vdev_id 0 / c010400
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: a0000000000
DVA[1]: :0:c010400:a00:d
DVA[2]: vdev_id 0 / 18008400
DVA[2]:       GANG: FALSE  GRID:  0000  ASIZE: a0000000000
DVA[2]: :0:18008400:a00:d
LSIZE:  4000                            PSIZE: a00
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 0     FILL:  1a00000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  a9af50f215:ec01e192b95e:c523efad092ebc:7a3a8be19416f454
$q
#

Using zdb with the offset specified for the first ditto block
in the above blkptr output, we get the mos dnode array.
Note that the "LEVEL: 0" blkptr output means there are
no levels of indirection. On larger zfs file systems, you may
need to go through block(s) of indirect blkptr_t's. An example of this
is shown a bit later.

# ./zdb -R zfs_fs:0:11000:a00:d,lzjb,4000 2> /tmp/metadnode
Found vdev: /export/home/max/zfsfile
#

Now, we'll look at the metadnode for the DMU_OT_OBJECT_DIRECTORY. This
will tell us about objects in the zfs file system. For every zfs file
system that I have tried this on, this is dnode number 1, (starting from
0). Regardless, the field to check is "dn_type = 0x1". It is possible,
(I assume), for this to be at a different index into the metadnode array,
and, possibly not in the 0x4000 bytes read and decompressed from 0x11000.
In this case, the LEVEL field would not have been 0, and you would have to
look at indirect blkptr_t's. But not here...

# ./mdb /tmp/metadnode 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0,20::print -a -t zfs`dnode_phys_t  <-- dnode_phys_t is 0x200 bytes, so 0x20
{          <-- of these in 0x4000.
    0 uint8_t dn_type = 0  <-- First entry is not used (DMU_OT_NONE)
...
}
{   <-- start of the second entry
    200 uint8_t dn_type = 0x1  <-- DMU_OT_OBJECT_DIRECTORY (see dmu.h)
    ...
    240 blkptr_t [1] dn_blkptr = [ <-- blkptr_t is 0x240 in /tmp/metadnode
    ...  <-- lots of output omitted, we'll look at some of this later.
}

Now we'll look at the blkptr_t for the Object Directory.

240::blkptr
DVA[0]: vdev_id 0 / 2400
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[0]: :0:2400:200:d
DVA[1]: vdev_id 0 / c002400
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[1]: :0:c002400:200:d
DVA[2]: vdev_id 0 / 18000000
DVA[2]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[2]: :0:18000000:200:d
LSIZE:  200                             PSIZE: 200
ENDIAN: LITTLE                                  TYPE:  object directory
BIRTH:  4                  LEVEL: 0     FILL:  100000000
CKFUNC: fletcher4                       COMP:  uncompressed
CKSUM:  5a40238b4:1cd8f9f7e19:522eab3e03f0:a9c92410b009e
$q

#

Now, we'll read the (uncompressed) 0x200 bytes of the object directory using zdb. The "2400" is the (hex) offset from the blkptr_t above.

# ./zdb -R zfs_fs:0:2400:200:r 2> /tmp/objdir
Found vdev: /export/home/max/zfsfile
#

Back to mdb to look at the object directory. Object directories are "zap"
objects. Zap objects contain name/value pairs. The first 64 bits
identify the type of the zap (micro zap or fat zap). A "fat zap" is a zap
object that uses indirection. Micro zaps contain name/value pairs directly
(i.e., no indirection). I have not seen a fat zap (but the largest zfs
file system I have used is only ~140GB, and I have not examined large
directories. (Directory entries are stored in zap objects).

# ./mdb /tmp/objdir 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so

> 0/J  <-- look at the first 64 bits as hex
0:              8000000000000003  <-- a "signature" for a micro zap

> 0::print -a -t zfs`mzap_phys_t  <-- the beginning of the microzap is
{                                 <-- an mzap_phys_t
    0 uint64_t mz_block_type = 0x8000000000000003
    8 uint64_t mz_salt = 0x129c2c3
    10 uint64_t mz_normflags = 0
    18 uint64_t [5] mz_pad = [ 0, 0, 0, 0, 0 ]
    40 mzap_ent_phys_t [1] mz_chunk = [  <-- there may be more than one
        {                                <-- mzap_ent_phys_t starting here
            40 uint64_t mze_value = 0x2  <-- object id of "root_dataset"
            48 uint32_t mze_cd = 0
            4c uint16_t mze_pad = 0
            4e char [50] mze_name = [ "root_dataset" ]  
        }
    ]
}
$q
#

Now, we go back to the mos metadnode array in /tmp/metadnode, and
examine object id 2 (the third entry in the array).
Each entry is 0x200 bytes, so we want the dnode_phys_t starting
at (2*200) bytes into the file.

# ./mdb /tmp/metadnode 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 2*200::print -a -t zfs`dnode_phys_t  <-- get object id 2
{
    400 uint8_t dn_type = 0xc  <-- DMU_OT_DSL_DIR (a dataset directory object)
    ...
    404 uint8_t dn_bonustype = 0xc  <-- bonus buffer contains a dsl_dir_phys_t
    ...
    440 blkptr_t [1] dn_blkptr = [  <-- not used for this object
        {
            440 dva_t [3] blk_dva = [
                {
                    440 uint64_t [2] dva_word = [ 0, 0 ]
      ...
    4c0 uint8_t [320] dn_bonus = [ 0xe5, 0xa9, 0xa6, 0x48, 0, 0, 0, 0, 0x10,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xe, 0, 0, 0, 0, 0, 0, 0, ... ]
}

And dump the bonus buffer at 0x4c0.

> 4c0::print -a -t zfs`dsl_dir_phys_t
{
    4c0 uint64_t dd_creation_time = 0x48a6a9e5
    4c8 uint64_t dd_head_dataset_obj = 0x10  <-- object id for dataset head
    ...
}

Let's get object id 10 from the metadnode array.

> 10*200::print -a -t zfs`dnode_phys_t
{
    2000 uint8_t dn_type = 0x10  <-- DMU_OT_DSL_DATASET
    ...
    2004 uint8_t dn_bonustype = 0x10  <-- bonus buffer contains dsl_dataset_phys_t
    ...
    2040 blkptr_t [1] dn_blkptr = [  <-- again, not used here
        {
            2040 dva_t [3] blk_dva = [
                {
                    2040 uint64_t [2] dva_word = [ 0, 0 ]
      ...
    20c0 uint8_t [320] dn_bonus = [ 0x2, 0, 0, 0, 0, 0, 0, 0, 0xe, 0, 0, 0, 0, 0
, 0, 0, 0x1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ... ]
}

At 0x20c0 in the /tmp/metadnode file is a dsl_dataset_phys_t (the bonus
buffer).

> 20c0::print -a -t zfs`dsl_dataset_phys_t
{
    20c0 uint64_t ds_dir_obj = 0x2
    ...
    2140 blkptr_t ds_bp = {
        2140 dva_t [3] blk_dva = [
            {
                2140 uint64_t [2] dva_word = [ 0x1, 0x79 ]
            }
            {
                2150 uint64_t [2] dva_word = [ 0x1, 0x60073 ]
            }
            {
                2160 uint64_t [2] dva_word = [ 0, 0 ]
            }
        ]
 ...
}

Let's look at the blkptr_t in the dsl_dataset_phys_t.

> 2140::blkptr
DVA[0]: vdev_id 0 / f200
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[0]: :0:f200:200:d
DVA[1]: vdev_id 0 / c00e600
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[1]: :0:c00e600:200:d
LSIZE:  400                             PSIZE: 200
ENDIAN: LITTLE                                  TYPE:  DMU objset
BIRTH:  502                LEVEL: 0     FILL:  600000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  9cb4e346a:403aa7532bf:d688fac60e1e:1e67a933734ea5
$q
#

The blkptr_t for the dsl_dataset_phys_t is for another DMU objset.
(The first DMU objset was from the uberblock_t rootbp and describe
the set of objects. The objset described by the dsl_dataset_phys_t describes
the set of objects in the file system (i.e., files and directories (and...?)).
Back to zdb to get this data.

# ./zdb -R zfs_fs:0:f200:200:d,lzjb,400 2> /tmp/root_dataset_mos
Found vdev: /export/home/max/zfsfile
#

And back to mdb to display the objset_phys_t for the root dataset.

# ./mdb /tmp/root_dataset_mos 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::print -a -t zfs`objset_phys_t
{
    0 dnode_phys_t os_meta_dnode = {
        0 uint8_t dn_type = 0xa  <-- the second object directory
 ...
        40 blkptr_t [1] dn_blkptr = [  <-- blkptr_t is 0x40 bytes into the file
 ...

And dump the blkptr_t...

> 40::blkptr
DVA[0]: vdev_id 0 / 10800
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:10800:400:id
DVA[1]: vdev_id 0 / c00fc00
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00fc00:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 6     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  58461f1c5e:3c7272ace4a1:15e2cf555fd2ac:58b9cdc6bcd0b54
$q
#

Note the "LEVEL: 6" in the above output. There are 6 levels of indirection
to get to another array of dnode_phys_t. We will follow the levels, always
using the first indirect blkptr_t at each level since the file was in
a directory whose object id is 3 (from "ls -aid /zfs_fs" back at the
beginning). If I want the dnode_phys_t for a different object id, I
can use the technique explained in the paper and slides referenced
at the beginning.

# ./zdb -R zfs_fs:0:10800:400:d,lzjb,4000 2> /tmp/dnode_l6
Found vdev: /export/home/max/zfsfile
#

Each indirect blkptr_t array contains 0x80 blkptr_t structures. (The size
of a blkptr_t is 0x80 bytes. 0x4000 (i.e., the size of the decompressed
data) divided by 0x80 = 0x80). We'll use mdb to examine blkptr 0 in the
array.

# ./mdb /tmp/dnode_l6 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::blkptr
DVA[0]: vdev_id 0 / 10400
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:10400:400:id
DVA[1]: vdev_id 0 / c00f800
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00f800:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 5     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  593bf7cd50:3d5bdfbff40e:1652191251855c:5af2260f72aa12a
$q
#

Great. Let's get the indirect array for level 5.

# ./zdb -R zfs_fs:0:10400:400:d,lzjb,4000 2> /tmp/dnode_l5
Found vdev: /export/home/max/zfsfile
#

And back to mdb to display it...

# ./mdb /tmp/dnode_l5
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::blkptr
DVA[0]: vdev_id 0 / 10000
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:10000:400:id
DVA[1]: vdev_id 0 / c00f400
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00f400:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 4     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  5a4787d7ae:3e5a99501b03:16cbd983ac6802:5d616f6513864cb
$q
#

And now to level 4. Note that BIRTH value corresponds to the
transaction id we want... (0x502 = 0t1282).

# ./zdb -R zfs_fs:0:10000:400:d,lzjb,4000 2> /tmp/dnode_l4
Found vdev: /export/home/max/zfsfile
#

Back to mdb...

# ./mdb /tmp/dnode_l4
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::blkptr
DVA[0]: vdev_id 0 / fc00
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:fc00:400:id
DVA[1]: vdev_id 0 / c00f000
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00f000:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 3     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  580321bd90:3cf0cb3827a9:1647f21a4fee83:5b2042e25b8771b
$q
#

Now to level 3.

# ./zdb -R zfs_fs:0:fc00:400:d,lzjb,4000 2> /tmp/dnode_l3
Found vdev: /export/home/max/zfsfile
#
# ./mdb /tmp/dnode_l3
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::blkptr
DVA[0]: vdev_id 0 / f800
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:f800:400:id
DVA[1]: vdev_id 0 / c00ec00
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00ec00:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 2     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  58e75640d6:3d0bc696c0c2:162c02c075c9ab:5a30099a876cabe
$q
#

And level 2...

# ./zdb -R zfs_fs:0:f800:400:d,lzjb,4000 2> /tmp/dnode_l2
Found vdev: /export/home/max/zfsfile
#
# ./mdb /tmp/dnode_l2
mdb: no terminal data available for TERM=emacs
mdb: term init failed: command-line editing and prompt will not be available
::loadctf
::load /export/home/max/source/mdb/i386/rawzfs.so
0::blkptr
DVA[0]: vdev_id 0 / f400
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:f400:400:id
DVA[1]: vdev_id 0 / c00e800
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00e800:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 1     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  5763205f2d:3c57f7df68a9:15fea170721af5:59a7686491c24a7
$q
#

And level 1.

# ./zdb -R zfs_fs:0:f400:400:d,lzjb,4000 2> /tmp/dnode_l1
Found vdev: /export/home/max/zfsfile
#
# ./mdb /tmp/dnode_l1
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0::blkptr
DVA[0]: vdev_id 0 / ec00
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 60000000000
DVA[0]: :0:ec00:600:d
DVA[1]: vdev_id 0 / c00e000
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 60000000000
DVA[1]: :0:c00e000:600:d
LSIZE:  4000                            PSIZE: 600
ENDIAN: LITTLE                                  TYPE:  DMU dnode
BIRTH:  502                LEVEL: 0     FILL:  500000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  6fb5c84271:61d7d7ffe6a4:2f9cbc90dcaa4c:10f07885852e2558
$q
#

Level 0 will contain the beginning of the array of dnode_phys_t
for files and directories within the file system.
We'll again use zdb to retrieve the block containing the first
0x20 entries. (Again, decompressed size is 0x4000, dnode_phys_t size
is 0x200, so there are 0x20 entries in the first level 0 block).

# ./zdb -R zfs_fs:0:ec00:600:d,lzjb,4000 2> /tmp/dnode_l0
Found vdev: /export/home/max/zfsfile
#

# ./mdb /tmp/dnode_l0
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0,20::print -t -a zfs`dnode_phys_t
{
    0 uint8_t dn_type = 0 <-- first entry is not used
    ...
}
{  <-- second entry (object id 1)
    200 uint8_t dn_type = 0x15  <-- DMU_OT_MASTER_NODE
    ...
    240 blkptr_t [1] dn_blkptr = [
        {
 ...
}
{  <-- third entry (object id 2)
    400 uint8_t dn_type = 0x16
    ...
{  <-- fourth entry (object id 3, should be root directory for the fs)
    600 uint8_t dn_type = 0x14  <-- DMU_OT_DIRECTORY_CONTENTS
    ...
    604 uint8_t dn_bonustype = 0x11  <-- bonus buffer contains znode_phys_t
    ...
    640 blkptr_t [1] dn_blkptr = [  <-- this blkptr_t is a zap for directory entries
        {
            640 dva_t [3] blk_dva = [
                {
                    640 uint64_t [2] dva_word = [ 0x1, 0x73 ]
                }
  ...
     6c0 uint8_t [320] dn_bonus = [ 0x1e, 0xe9, 0xa7, 0x48, 0, 0, 0, 0,
      0xc3, 0x61, 0x34, 0xf, 0, 0, 0, 0, 0x1f, 0xe9, 0xa7, 0x48, 0,
      0, 0, 0, 0x1, 0x43, 0x79, 0x3a, 0, 0, 0, 0, ... ]
 ... <-- lots omitted
}

At this point, we could go the the fourth entry in the above output
(object id 3 at 0x600 bytes into the file) and look at the directory
contents to see if the removed file is there. (Remember, ls -aid on
the directory containing the removed file shows inumber 3).
However, we'll be safe and examine the master node to get to
the root directory of the file system. The master node
is object id 1 (at 0x200 in the above output). The block pointer
for this dnode_phys_t is for a zap object.
We'll use mdb to dump the master node blkptr_t.

> 240::blkptr
DVA[0]: vdev_id 0 / 0
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[0]: :0:0:200:d
DVA[1]: vdev_id 0 / c000000
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[1]: :0:c000000:200:d
LSIZE:  200                             PSIZE: 200
ENDIAN: LITTLE                                  TYPE:  ZFS master node
BIRTH:  4                  LEVEL: 0     FILL:  100000000
CKFUNC: fletcher4                       COMP:  uncompressed
CKSUM:  233dfc135:e10dd7aa27:2e8c1eba771e:6a380d575d3d6
$q
#

And now back to zdb to get the zfs master node zap object. Note
that this is not compressed, and is at the beginning of the disk
(following label 0 and label 1.

# ./zdb -R zfs_fs:0:0:200:r 2> /tmp/master_node
Found vdev: /export/home/max/zfsfile
#

Back to mdb to examine the master node zap.

# ./mdb /tmp/master_node 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0/J  <-- let's see what kind of zap it is
0:              8000000000000003 <-- micro zap
> 0::print -a -t zfs`mzap_phys_t
{
    0 uint64_t mz_block_type = 0x8000000000000003
    8 uint64_t mz_salt = 0x3d3b
    10 uint64_t mz_normflags = 0
    18 uint64_t [5] mz_pad = [ 0, 0, 0, 0, 0 ]
    40 mzap_ent_phys_t [1] mz_chunk = [
        {
            40 uint64_t mze_value = 0x3
            48 uint32_t mze_cd = 0
            4c uint16_t mze_pad = 0
            4e char [50] mze_name = [ "VERSION" ]
        }
    ]
}

The mzap_phys_t is 0x80 bytes large. Following this are zero or more
mzap_ent_phys_t. Each mzap_ent_phys_t is 0x40 bytes. The following
will dump all mzap_ent_phys_t following the mzap_phys_t in the block.

> 80,((200-80)%40)::print -a -t zfs`mzap_ent_phys_t
...
{
    c0 uint64_t mze_value = 0x3  <-- the object id for the root of the fs
    c8 uint32_t mze_cd = 0
    cc uint16_t mze_pad = 0
    ce char [50] mze_name = [ "ROOT" ]  <-- this is root
}
$q
#

Now, back to the level 0 dnode_phys_t array to look at the root directory
dnode_phys_t.

# ./mdb /tmp/dnode_l0
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 3*200::print -a -t zfs`dnode_phys_t
{
    600 uint8_t dn_type = 0x14  <-- DMU_OT_DIRECTORY_CONTENTS
    ...
    604 uint8_t dn_bonustype = 0x11 <-- bonus buffer contains znode_phys_t
    ...
    640 blkptr_t [1] dn_blkptr = [
        {
            640 dva_t [3] blk_dva = [
                {
                    640 uint64_t [2] dva_word = [ 0x1, 0x73 ]
                }
  ...
}

The blkptr_t is for a zap object containing filename/object id
values for files in the root directory of the file system.

> 640::blkptr
DVA[0]: vdev_id 0 / e600
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[0]: :0:e600:200:d
DVA[1]: vdev_id 0 / c00da00
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 20000000000
DVA[1]: :0:c00da00:200:d
LSIZE:  200                             PSIZE: 200
ENDIAN: LITTLE                                  TYPE:  ZFS directory
BIRTH:  502                LEVEL: 0     FILL:  100000000
CKFUNC: fletcher4                       COMP:  uncompressed
CKSUM:  25f50a2fc:fe963fd84e:36937666328d:7f9475424708c
$q
#

Now read the root directory zap object.

# ./zdb -R zfs_fs:0:e600:200:r 2> /tmp/rootdir
Found vdev: /export/home/max/zfsfile
#

And use mdb to look at the zap entries.

# ./mdb /tmp/rootdir 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> 0/J
0:              8000000000000003  <-- a micro zap


> 0::print -a -t zfs`mzap_phys_t
{
    0 uint64_t mz_block_type = 0x8000000000000003
    8 uint64_t mz_salt = 0x3e0f
    10 uint64_t mz_normflags = 0
    18 uint64_t [5] mz_pad = [ 0, 0, 0, 0, 0 ]
    40 mzap_ent_phys_t [1] mz_chunk = [
        {
            40 uint64_t mze_value = 0x8000000000000004
            48 uint32_t mze_cd = 0
            4c uint16_t mze_pad = 0
            4e char [50] mze_name = [ "foo" ]
        }
    ]
}

And dump the rest of the zap entries.

> 80,((200-80)%40)::print -a -t zfs`mzap_ent_phys_t
{
    80 uint64_t mze_value = 0x8000000000000005
    88 uint32_t mze_cd = 0
    8c uint16_t mze_pad = 0
    8e char [50] mze_name = [ "words" ] <-- here is the removed file
}
...
5*200=X  <-- we want dnode_phys_t object id 5.
                a00    <-- Offset within /tmp/dnode_l0 where the object resides
$q
#

We'll go back and get the dnode for object id 5.

# ./mdb /tmp/dnode_l0
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
> a00::print -a -t zfs`dnode_phys_t
{
    a00 uint8_t dn_type = 0x13  <-- DMU_OT_PLAIN_FILE_CONTENTS
    ...
    a04 uint8_t dn_bonustype = 0x11 <-- znode_phys_t for "words" file
    ...
    a40 blkptr_t [1] dn_blkptr = [  <-- blkptr_t for data or indirect blocks
        {
 ...
    ac0 uint8_t [320] dn_bonus = [ 0x1f, 0xe9, 0xa7, 0x48, 0, 0, 0, 0, 0xcb, 
0x96, 0x78, 0x3a, 0, 0, 0, 0, 0x1f, 0xe9, 0xa7, 0x48, 0, 0, 0, 0, 0xd1, 0xb1, 
0x83, 0x3a, 0, 0, 0, 0, ... ]
}

Now, let's take a quick look at the znode_phys_t for this file.
It is in the bonus buffer at 0xac0.

> ac0::print -a -t zfs`znode_phys_t
{
    ac0 uint64_t [2] zp_atime = [ 0x48a7e91f, 0x3a7896cb ]
    ad0 uint64_t [2] zp_mtime = [ 0x48a7e91f, 0x3a83b1d1 ]
    ae0 uint64_t [2] zp_ctime = [ 0x48a7e91f, 0x3a83b1d1 ]
    af0 uint64_t [2] zp_crtime = [ 0x48a7e91f, 0x3a7896cb ]
    b00 uint64_t zp_gen = 0x502
    b08 uint64_t zp_mode = 0x8124
    b10 uint64_t zp_size = 0x32752 <-- should be same size as /usr/dict/words
    b18 uint64_t zp_parent = 0x3
    b20 uint64_t zp_links = 0x1
    ...
}

> 32752=D
                206674          
> !ls -l /usr/dict/words
-r--r--r--   1 root     bin       206674 Jul 11 02:57 /usr/dict/words

Looks good. Let's look at the blkptr_t for this dnode_phys_t.

> a40::blkptr
DVA[0]: vdev_id 0 / e800
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[0]: :0:e800:400:id
DVA[1]: vdev_id 0 / c00dc00
DVA[1]:       GANG: FALSE  GRID:  0000  ASIZE: 40000000000
DVA[1]: :0:c00dc00:400:id
LSIZE:  4000                            PSIZE: 400
ENDIAN: LITTLE                                  TYPE:  ZFS plain file
BIRTH:  502                LEVEL: 1     FILL:  200000000
CKFUNC: fletcher4                       COMP:  lzjb
CKSUM:  5e9a82c0c2:3ff97cbecacc:1714169599f4c8:5dd02ff967dd42c
$q
#

Note the "LEVEL: 1". This means there is one level of indirect blocks.
We'll use zdb to retrieve the indirect block.

# ./zdb -R zfs_fs:0:e800:400:d,lzjb,4000 2> /tmp/iblock
Found vdev: /export/home/max/zfsfile
#

And mdb to look at the indirect block.

# ./mdb /tmp/iblock 
> ::loadctf
> ::load /export/home/max/source/mdb/i386/rawzfs.so
0::blkptr
DVA[0]: vdev_id 0 / 20000
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 2000000000000
DVA[0]: :0:20000:20000:d
LSIZE:  20000                           PSIZE: 20000
ENDIAN: LITTLE                                  TYPE:  ZFS plain file
BIRTH:  502                LEVEL: 0     FILL:  100000000
CKFUNC: fletcher2                       COMP:  uncompressed
CKSUM:  f5cbf93a151abcac:5b5d6ca83588d8ad:574d9b8bf334944b:ad78d30af51771d8

The blkptr_t above is for the first 0x20000 (128k) of the file.
The next blkptr_t in the indirect block should contain the remainder
of the file. (The file is less than 256k large).

> 80::blkptr
DVA[0]: vdev_id 0 / 40000
DVA[0]:       GANG: FALSE  GRID:  0000  ASIZE: 2000000000000
DVA[0]: :0:40000:20000:d
LSIZE:  20000                           PSIZE: 20000
ENDIAN: LITTLE                                  TYPE:  ZFS plain file
BIRTH:  502                LEVEL: 0     FILL:  100000000
CKFUNC: fletcher2                       COMP:  uncompressed
CKSUM:  f39ae34f048ae079:de2ef1af7d1fb495:ec3ae3f7985b2a98:c6d33ac68cb042b6
$q
#

Now we'll use zdb to retrieve the data blocks.

# ./zdb -R zfs_fs:0:20000:20000:r 2> /tmp/data  <-- first data block
Found vdev: /export/home/max/zfsfile
# ./zdb -R zfs_fs:0:40000:20000:r 2> /tmp/data1 <-- second data block
Found vdev: /export/home/max/zfsfile
#

# cat /tmp/data /tmp/data1 > /tmp/foo  <-- concatenate them

Size of the file, according to the znode_phys_t is 206674 bytes.
We'll lop off remaining bytes.

# dd if=/tmp/foo bs=206674 count=1 of=/tmp/finalwords
1+0 records in
1+0 records out

Now, let's see if we have the correct data.

# diff /tmp/finalwords /usr/dict/words
#   <-- no differences

new opensolaris course material

I recently wrote the material and taught the second day of a three day course
on OpenSolaris for a group of university professors in Bangalore, India. The
first day was largely a "getting started on OpenSolaris" session, along with
an introduction to dtrace, and the third day was mostly administration
specifically about zones and zfs. The second day was an introduction to
OpenSolaris Internals. The topics I covered included processes and threads,
synchronization, memory management, and file systems. Those who have taken
the 5 day Solaris Internals course with me will find that several of the
diagrams and hands-on exercises that I use during that course are now written
up in this material. Note that the material for the second day uses only two
pages that are come from pre-existing sources. So this is new material.
The material contains both overhead slides and accompanying text.

I did something similar for professors in China almost 2 years ago, except
that session was a 5 day internals session. The India professors looked at
materials prepared by the Chinese professors, and decided that the depth of
coverage in the materials was too deep to use as a starting point for a course
on operating systems. Having looked at the Chinese professor's materials, I
am inclined to agree with the Indian professors. The Chinese material is
an excellent guide through the source code and through McDougall and Mauro's
Solaris Internals book. As a study guide for people trying to learn about
the way OpenSolaris works, it is quite good and complete. As a tool for
teaching, especially classes without prior operating system experience, I feel
it misses the mark. While a professor that has good operating system knowledge
can use the Chinese material to learn OpenSolaris internals for him or her
self, I think the materials may assume too much prior knowledge of the
students. The new material tries to give professors a starting point that can
be used to teach OpenSolaris, not just to learn it.

For those of you who already have an Operating Systems background, (though not
in OpenSolaris), I think you'll find this new material quite useful. The
new material is not elementary (there is plenty of useful information, even
for people who have extensive OS and even extensive Solaris kernel experience).
It makes assumptions that, for instance, you understand why one needs locks,
or why virtual memory is useful, among others. The material explains the
implementation of various concepts/mechanisms using a combination of tools,
mostly mdb and dtrace, and various diagrams. In a 1 day session, there are
many topics not covered, and some are covered very superficially, but most
of the major OS topics are covered in a good amount of detail.

The new material is at: http://www.opensolaris.org/os/community/edu/curriculum_development.
Look at the OpenSolaris Curriculum "Plugins Preparation" section for overheads
and course guides.

Using kernel ctf with raw disk

The following shows a use of a modified version of mdb
which allows one to use the CTF information from the kernel
to examine data structures on disk. The disk used below contains
a ufs file system. The same techniques can be used to examine
zfs file system on disk, which is why I did this in the first place.
Once I have "mapped out" the on-disk format of zfs using this modified
version of mdb, I'll write about it.

In the meantime, I'll probably add a few dcmds and walkers for ufs that
use the kernel CTF information.

In the following, annotation starts with "<--" except for a
few places where I have embedded code from header files.
Also, the output has been truncated in a few places.
I am assuming that you have some knowledge of mdb, (for instance,
"value1 % value2 = X" returns the (hex) value of value1 divided by value2.
If you need more mdb, read the Modular Debugger Guide on docs.sun.com,
or, even better, take a course.

This example will start with the superblock, and from there examine the root
inode and then the root directory. From there, the example gets the /var inode
and then the /var directory. From there, we go to the /var/sadm directory
and look at the contents of the /var/sadm/README file. All of this is
done by examining the relevant data structures on disk.

# ./mdb /dev/rdsk/c0d0s0 <-- this is the root fs
mdb: no terminal data available for TERM=emacs
mdb: term init failed: command-line editing and prompt will not be available
mdb: no module 'mdb_ks' could be found <-- kernel support module not loaded(?)
mdb: failed to load kernel support module -- some modules may not load
::print struct anon <-- try ::print, normally this does not work with raw disk
{
an_vp <-- it works!
an_pvp
an_off
an_poff
an_hash
an_refcnt
}
2000::print struct fs <-- superblock should be 8192 bytes into fs (see sys/fs/ufs_fs.h)
{
fs_link = 0 <-- see fs_magic below for sanity check
fs_rolled = 0x2
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_dblkno = 0x2f8
fs_cgoffset = 0x40
fs_cgmask = 0xffffffc0
fs_time = 0x46eb90d4
fs_size = 0x32e3519
fs_dsize = 0x321e0c8
fs_ncg = 0x43e
fs_bsize = 0x2000
fs_fsize = 0x400
<-- output omitted
fs_fsmnt = [ "/" ]
<-- output omitted
fs_magic = 0x11954 <-- check against FS_MAGIC in sys/fs/ufs_fs.h (correct)
fs_space = [ 0x8 ]
}

::status <-- what does mdb say I'm debugging
debugging file '/dev/rdsk/c0d0s0' (object file)

<-- The following only prints the fields I am interested in:
2000::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic fs_ipg
fs_sblkno = 0x10 <-- location of the superblock in the cylinder group
fs_cblkno = 0x18 <-- location of the cylinder group block (struct cg)
fs_iblkno = 0x20 <-- location of start of inodes (in cylinder group)
fs_cgoffset = 0x40 <-- offset of cylinder group
fs_magic = 0x11954 <-- magic number
fs_ipg = 0x16c0 <-- inodes per cylinder group

<-- immediately following superblock is back up
4000::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_cgoffset = 0x40
fs_magic = 0x11954

6000::print struct cg <-- next block should be first cylinder group block
{
cg_link = 0
cg_magic = 0x90255 <-- magic number is good
cg_time = 0x46e78f3e
<-- output omitted
}

::sizeof struct icommon <-- how big is the disk inode
sizeof (struct icommon) = 0x80

<-- the following is the root inode. Root for ufs is inumber 2. The fs_iblkno
<-- value (0x20) is multiplied by the fragment size to get the start of the
<-- inodes (in the first cylinder group), each inode is 0x80 bytes large. The
<-- second (i.e., root inode) is then at disk location (20*400+(2*80)
(20*400)+(2*80)::print -a struct icommon
{
8100 ic_smode = 0x41ed
8102 ic_nlink = 0x30
8104 ic_suid = 0
8106 ic_sgid = 0
8108 ic_lsize = 0x600
8110 ic_atime = {
8110 tv_sec = 0x46eba57f
8114 tv_usec = 0x5a69b
}
<-- output omitted
8128 ic_db = [ 0x2ff410, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ] <-- one data block for / directory
8158 ic_ib = [ 0, 0, 0 ]
<-- output omitted
}

8110\Y <-- this is disk address of atime stamp on root inode
0x8110: 2007 Sep 15 11:27:27 <-- looks good

!date <-- current time
Sat Sep 15 11:36:07 CEST 2007

<-- now the block number from ic_db[0] is used to dump the contents of the
<-- root directory
2ff410*400::print struct direct
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(2ff410*400)+c::print struct direct <-- second entry (first+d_reclen)
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(2ff410*400)+c+c::print struct direct <-- third entry (first + d_reclen of first and second)
{
d_ino = 0x3
d_reclen = 0x14
d_namlen = 0xa
d_name = [ "lost+found" ]
}
(2ff410*400)+c+c+14::print struct direct <-- fourth entry (easily made into walker?)
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x3
d_name = [ "var" ]
}
<-- check "/var" inumber
!ls -id /var <-- get inumber
5824 /var
16c0=D <-- d_ino from direct entry
5824 <-- match

<-- the following is a back up superblock in the second cylinder group.
<-- The relevant macros for this are in sys/fs/ufs_fs.h and are shown here:
/*
* Cylinder group macros to locate things in cylinder groups.
* They calc file system addresses of cylinder group data structures.
*/
#define cgbase(fs, c) ((daddr32_t)((fs)->fs_fpg * (c)))

#define cgstart(fs, c) \
(cgbase(fs, c) + (fs)->fs_cgoffset * ((c) & ~((fs)->fs_cgmask)))

#define cgsblock(fs, c) (cgstart(fs, c) + (fs)->fs_sblkno) /* super blk */

#define cgtod(fs, c) (cgstart(fs, c) + (fs)->fs_cblkno) /* cg block */

#define cgimin(fs, c) (cgstart(fs, c) + (fs)->fs_iblkno) /* inode blk */

#define cgdmin(fs, c) (cgstart(fs, c) + (fs)->fs_dblkno) /* 1st data */

/*
* Macros for handling inode numbers:
* inode number to file system block offset.
* inode number to cylinder group number.
* inode number to file system block address.
*/
#define itoo(fs, x) ((x) % (uint32_t)INOPB(fs))

#define itog(fs, x) ((x) / (uint32_t)(fs)->fs_ipg)
<-- So. Here the fs_fpg field from the superblock (= 0xc000) is used to
<-- get the fragments per group. This is multiplied times the fragment size (0x400)
<-- Then the fs_cgoffset field (cylinder group offset) is added (40*400), then the
<-- fs_sblkno offset (10*400). The resulting address is the location on the
<-- disk of the backup superblock in the second cylinder group. To see the third,
<-- use (c000*2*400)+(40*400)+(10*400)::print struct fs
<-- To see the fourth, (c000*3*400)+(40*400)+(10*400)::print struct fs, etc.

(c000*400)+(40*400)+(10*400)::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_cgoffset = 0x40
fs_magic = 0x11954

<-- Now, let's take a look at the inode for the "/var" directory.
<-- Above, the direct structure for /var says the inumber is 0x16c0.
<-- There are 16c0 inodes per cylinder group (the fs_ipg field in the
<-- superblock), so this inode should be the first inode in the
<-- second cylinder group. (c000*400) is the base of the second cylinder
<-- group. (40*400) is the starting offset. (20*400) is the starting
<-- inode offset. Given an inumber, the formula for finding the inode
<-- on disk is:
<-- (inumber % fs_ipg)=X This returns an index indicating which cylinder
<-- group the inode is in. This is "cg_index" in the next calculation.
<-- (inumber - (cg_index * fs_ipg))=X This returns the index
<-- (offset) within the cylinder group ("cg_offset") (Actually, inumber mod fs_ipg).
<-- Then: ((fs_fpg*400)*cg_index)+((fs_cgoffset*400)*cg_index)+(fs_iblkno*400)+(cg_offset*80).
<-- Here, 400 is the fragment size (from fs_fsize) and 80 is the sizeof the
<-- disk inode.

(16c0%16c0)=X
1 <-- the second cylinder group
16c0-(1*16c0)=X
0 <-- the first inode in the group

<-- this is the inode for /var
(c000*400*1)+(40*400)+(20*400)+(0*80)::print struct icommon
{
ic_smode = 0x41ed
ic_nlink = 0x2c
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x400
ic_atime = {
tv_sec = 0x46eb30e8
tv_usec = 0x992bc
}
<-- output omitted
ic_db = [ 0xc348, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}

<-- So /var has one block (ic_db[0] = 0xc348). This should be a directory. Now dump out some
<-- entries. The following is a good candidate for a walker...
(c348*400)::print struct direct
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(c348*400)+c::print struct direct
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(c348*400)+c+c::print struct direct
{
d_ino = 0x16c1
d_reclen = 0x10
d_namlen = 0x4
d_name = [ "sadm" ]
}

<-- let's check the work...
!ls -id /var/sadm
5825 /var/sadm
16c1=D
5825 <-- match looks good
<-- Ok. Now lets look at the inode for /var/sadm. This is
<-- inumber 16c1.

16c1%16c0=X
1 <-- the second cylinder group
16c1-(16c0*1)=X
1 <-- the second inode in the group

(c000*400*1)+(40*400)+(20*400)+(1*80)::print struct icommon
{
ic_smode = 0x41ed
ic_nlink = 0xd
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x200
<-- output omitted
ic_db = [ 0xc349, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}
c349*400::print struct direct
{
d_ino = 0x16c1
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(c349*400)+c::print struct direct
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(c349*400)+c+c::print struct direct
{
d_ino = 0x16c2
d_reclen = 0x10
d_namlen = 0x7
d_name = [ "install" ]
}
(c349*400)+c+c+10::print struct direct
{
d_ino = 0x16c5
d_reclen = 0xc
d_namlen = 0x3
d_name = [ "pkg" ]
}
(c349*400)+c+c+10+c::print struct direct
{
d_ino = 0x1d41
d_reclen = 0x10
d_namlen = 0x6
d_name = [ "system" ]
}
(c349*400)+c+c+10+c+10::print struct direct
{
d_ino = 0x7e4d
d_reclen = 0x18
d_namlen = 0xc
d_name = [ "install_data" ]
}
(c349*400)+c+c+10+c+10+18::print struct direct
{
d_ino = 0x7e4e
d_reclen = 0x14
d_namlen = 0x8
d_name = [ "softinfo" ]
}
(c349*400)+c+c+10+c+10+18+14::print struct direct
{
d_ino = 0x27d9
d_reclen = 0x10
d_namlen = 0x6
d_name = [ "README" ] <-- here is the file we want to examine
}

27d9%16c0=X
1 <-- the second cylinder group
27d9-(1*16c0)=X
1119 <-- the 1120th inode
(1*c000*400)+(40*400)+(20*400)+(1119*80)::print struct icommon
{
ic_smode = 0x8124
ic_nlink = 0x1
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x444
<-- output omitted
ic_db = [ 0x111cc, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}
111cc*400,200/c <-- ok, let's dump the first 512 bytes
0x4473000: -------------------------------
/var/sadm DIRECTORY RESTRUCTURE
-------------------------------

The system administration directory has been reorganized to bett
er
service the needs of Solaris administrators and administration s
oftware.
The old and new locations for files important to our customers a
re:

OLD LOCATION NEW LOCATION
------------ ------------
install_data/install_log system/logs/install_log
install_data/upgrade_log system/log

<-- check the work
!head /var/sadm/README
-------------------------------
/var/sadm DIRECTORY RESTRUCTURE
-------------------------------

The system administration directory has been reorganized to better
service the needs of Solaris administrators and administration software.
The old and new locations for files important to our customers are:

OLD LOCATION NEW LOCATION
------------ ------------
$q
#

bash-3.00$

Using dtrace and mdb to examine virtual memory

here is a short example using
dtrace and mdb to examine page faults and process address spaces.
This will be used in a workshop being given to professors teaching operating systems
within China. The workshop will cover Solaris internals using the opensolaris source code
and tools such as dtrace, mdb, and kmdb.

new web site for Bruning Systems

Hi,
I have finally gotten around to updating the web site. Still need to
add the dtrace scripts, but first I need to document them.

www.bruningsystems.com

max

snooping gld-based NIC drivers using dtrace

Hi.
I have just posted a script at http://www.bruningsystems.com/rtlsio.p
that allows one to snoop incoming/outgoing packets on a Realtek NIC.
The script is very easy to change for any other GLD-based NIC (see
the comment at the beginning of the script to determine what needs
to be changed.)
To run the script, save it and then:

# dtrace -q -C -s ./rtlsio.p

Let me know what you think.

max

dtrace script to trace kernel thread state changes

I just posted a script that traces kernel thread state changes to the dtrace
forum on forum.sun.com. You can also find it at http://www.bruningsystems.com/runq.d

Have fun.
max

solaris kernel memory usage

This is a test. I am currently working on answering a question
from a former student about kernel memory. The question and
answer will be here shortly.