new opensolaris course material

I recently wrote the material and taught the second day of a three day course
on OpenSolaris for a group of university professors in Bangalore, India. The
first day was largely a "getting started on OpenSolaris" session, along with
an introduction to dtrace, and the third day was mostly administration
specifically about zones and zfs. The second day was an introduction to
OpenSolaris Internals. The topics I covered included processes and threads,
synchronization, memory management, and file systems. Those who have taken
the 5 day Solaris Internals course with me will find that several of the
diagrams and hands-on exercises that I use during that course are now written
up in this material. Note that the material for the second day uses only two
pages that are come from pre-existing sources. So this is new material.
The material contains both overhead slides and accompanying text.

I did something similar for professors in China almost 2 years ago, except
that session was a 5 day internals session. The India professors looked at
materials prepared by the Chinese professors, and decided that the depth of
coverage in the materials was too deep to use as a starting point for a course
on operating systems. Having looked at the Chinese professor's materials, I
am inclined to agree with the Indian professors. The Chinese material is
an excellent guide through the source code and through McDougall and Mauro's
Solaris Internals book. As a study guide for people trying to learn about
the way OpenSolaris works, it is quite good and complete. As a tool for
teaching, especially classes without prior operating system experience, I feel
it misses the mark. While a professor that has good operating system knowledge
can use the Chinese material to learn OpenSolaris internals for him or her
self, I think the materials may assume too much prior knowledge of the
students. The new material tries to give professors a starting point that can
be used to teach OpenSolaris, not just to learn it.

For those of you who already have an Operating Systems background, (though not
in OpenSolaris), I think you'll find this new material quite useful. The
new material is not elementary (there is plenty of useful information, even
for people who have extensive OS and even extensive Solaris kernel experience).
It makes assumptions that, for instance, you understand why one needs locks,
or why virtual memory is useful, among others. The material explains the
implementation of various concepts/mechanisms using a combination of tools,
mostly mdb and dtrace, and various diagrams. In a 1 day session, there are
many topics not covered, and some are covered very superficially, but most
of the major OS topics are covered in a good amount of detail.

The new material is at: http://www.opensolaris.org/os/community/edu/curriculum_development.
Look at the OpenSolaris Curriculum "Plugins Preparation" section for overheads
and course guides.

Using kernel ctf with raw disk

The following shows a use of a modified version of mdb
which allows one to use the CTF information from the kernel
to examine data structures on disk. The disk used below contains
a ufs file system. The same techniques can be used to examine
zfs file system on disk, which is why I did this in the first place.
Once I have "mapped out" the on-disk format of zfs using this modified
version of mdb, I'll write about it.

In the meantime, I'll probably add a few dcmds and walkers for ufs that
use the kernel CTF information.

In the following, annotation starts with "<--" except for a
few places where I have embedded code from header files.
Also, the output has been truncated in a few places.
I am assuming that you have some knowledge of mdb, (for instance,
"value1 % value2 = X" returns the (hex) value of value1 divided by value2.
If you need more mdb, read the Modular Debugger Guide on docs.sun.com,
or, even better, take a course.

This example will start with the superblock, and from there examine the root
inode and then the root directory. From there, the example gets the /var inode
and then the /var directory. From there, we go to the /var/sadm directory
and look at the contents of the /var/sadm/README file. All of this is
done by examining the relevant data structures on disk.

# ./mdb /dev/rdsk/c0d0s0 <-- this is the root fs
mdb: no terminal data available for TERM=emacs
mdb: term init failed: command-line editing and prompt will not be available
mdb: no module 'mdb_ks' could be found <-- kernel support module not loaded(?)
mdb: failed to load kernel support module -- some modules may not load
::print struct anon <-- try ::print, normally this does not work with raw disk
{
an_vp <-- it works!
an_pvp
an_off
an_poff
an_hash
an_refcnt
}
2000::print struct fs <-- superblock should be 8192 bytes into fs (see sys/fs/ufs_fs.h)
{
fs_link = 0 <-- see fs_magic below for sanity check
fs_rolled = 0x2
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_dblkno = 0x2f8
fs_cgoffset = 0x40
fs_cgmask = 0xffffffc0
fs_time = 0x46eb90d4
fs_size = 0x32e3519
fs_dsize = 0x321e0c8
fs_ncg = 0x43e
fs_bsize = 0x2000
fs_fsize = 0x400
<-- output omitted
fs_fsmnt = [ "/" ]
<-- output omitted
fs_magic = 0x11954 <-- check against FS_MAGIC in sys/fs/ufs_fs.h (correct)
fs_space = [ 0x8 ]
}

::status <-- what does mdb say I'm debugging
debugging file '/dev/rdsk/c0d0s0' (object file)

<-- The following only prints the fields I am interested in:
2000::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic fs_ipg
fs_sblkno = 0x10 <-- location of the superblock in the cylinder group
fs_cblkno = 0x18 <-- location of the cylinder group block (struct cg)
fs_iblkno = 0x20 <-- location of start of inodes (in cylinder group)
fs_cgoffset = 0x40 <-- offset of cylinder group
fs_magic = 0x11954 <-- magic number
fs_ipg = 0x16c0 <-- inodes per cylinder group

<-- immediately following superblock is back up
4000::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_cgoffset = 0x40
fs_magic = 0x11954

6000::print struct cg <-- next block should be first cylinder group block
{
cg_link = 0
cg_magic = 0x90255 <-- magic number is good
cg_time = 0x46e78f3e
<-- output omitted
}

::sizeof struct icommon <-- how big is the disk inode
sizeof (struct icommon) = 0x80

<-- the following is the root inode. Root for ufs is inumber 2. The fs_iblkno
<-- value (0x20) is multiplied by the fragment size to get the start of the
<-- inodes (in the first cylinder group), each inode is 0x80 bytes large. The
<-- second (i.e., root inode) is then at disk location (20*400+(2*80)
(20*400)+(2*80)::print -a struct icommon
{
8100 ic_smode = 0x41ed
8102 ic_nlink = 0x30
8104 ic_suid = 0
8106 ic_sgid = 0
8108 ic_lsize = 0x600
8110 ic_atime = {
8110 tv_sec = 0x46eba57f
8114 tv_usec = 0x5a69b
}
<-- output omitted
8128 ic_db = [ 0x2ff410, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ] <-- one data block for / directory
8158 ic_ib = [ 0, 0, 0 ]
<-- output omitted
}

8110\Y <-- this is disk address of atime stamp on root inode
0x8110: 2007 Sep 15 11:27:27 <-- looks good

!date <-- current time
Sat Sep 15 11:36:07 CEST 2007

<-- now the block number from ic_db[0] is used to dump the contents of the
<-- root directory
2ff410*400::print struct direct
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(2ff410*400)+c::print struct direct <-- second entry (first+d_reclen)
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(2ff410*400)+c+c::print struct direct <-- third entry (first + d_reclen of first and second)
{
d_ino = 0x3
d_reclen = 0x14
d_namlen = 0xa
d_name = [ "lost+found" ]
}
(2ff410*400)+c+c+14::print struct direct <-- fourth entry (easily made into walker?)
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x3
d_name = [ "var" ]
}
<-- check "/var" inumber
!ls -id /var <-- get inumber
5824 /var
16c0=D <-- d_ino from direct entry
5824 <-- match

<-- the following is a back up superblock in the second cylinder group.
<-- The relevant macros for this are in sys/fs/ufs_fs.h and are shown here:
/*
* Cylinder group macros to locate things in cylinder groups.
* They calc file system addresses of cylinder group data structures.
*/
#define cgbase(fs, c) ((daddr32_t)((fs)->fs_fpg * (c)))

#define cgstart(fs, c) \
(cgbase(fs, c) + (fs)->fs_cgoffset * ((c) & ~((fs)->fs_cgmask)))

#define cgsblock(fs, c) (cgstart(fs, c) + (fs)->fs_sblkno) /* super blk */

#define cgtod(fs, c) (cgstart(fs, c) + (fs)->fs_cblkno) /* cg block */

#define cgimin(fs, c) (cgstart(fs, c) + (fs)->fs_iblkno) /* inode blk */

#define cgdmin(fs, c) (cgstart(fs, c) + (fs)->fs_dblkno) /* 1st data */

/*
* Macros for handling inode numbers:
* inode number to file system block offset.
* inode number to cylinder group number.
* inode number to file system block address.
*/
#define itoo(fs, x) ((x) % (uint32_t)INOPB(fs))

#define itog(fs, x) ((x) / (uint32_t)(fs)->fs_ipg)
<-- So. Here the fs_fpg field from the superblock (= 0xc000) is used to
<-- get the fragments per group. This is multiplied times the fragment size (0x400)
<-- Then the fs_cgoffset field (cylinder group offset) is added (40*400), then the
<-- fs_sblkno offset (10*400). The resulting address is the location on the
<-- disk of the backup superblock in the second cylinder group. To see the third,
<-- use (c000*2*400)+(40*400)+(10*400)::print struct fs
<-- To see the fourth, (c000*3*400)+(40*400)+(10*400)::print struct fs, etc.

(c000*400)+(40*400)+(10*400)::print struct fs fs_sblkno fs_cblkno fs_iblkno fs_cgoffset fs_magic
fs_sblkno = 0x10
fs_cblkno = 0x18
fs_iblkno = 0x20
fs_cgoffset = 0x40
fs_magic = 0x11954

<-- Now, let's take a look at the inode for the "/var" directory.
<-- Above, the direct structure for /var says the inumber is 0x16c0.
<-- There are 16c0 inodes per cylinder group (the fs_ipg field in the
<-- superblock), so this inode should be the first inode in the
<-- second cylinder group. (c000*400) is the base of the second cylinder
<-- group. (40*400) is the starting offset. (20*400) is the starting
<-- inode offset. Given an inumber, the formula for finding the inode
<-- on disk is:
<-- (inumber % fs_ipg)=X This returns an index indicating which cylinder
<-- group the inode is in. This is "cg_index" in the next calculation.
<-- (inumber - (cg_index * fs_ipg))=X This returns the index
<-- (offset) within the cylinder group ("cg_offset") (Actually, inumber mod fs_ipg).
<-- Then: ((fs_fpg*400)*cg_index)+((fs_cgoffset*400)*cg_index)+(fs_iblkno*400)+(cg_offset*80).
<-- Here, 400 is the fragment size (from fs_fsize) and 80 is the sizeof the
<-- disk inode.

(16c0%16c0)=X
1 <-- the second cylinder group
16c0-(1*16c0)=X
0 <-- the first inode in the group

<-- this is the inode for /var
(c000*400*1)+(40*400)+(20*400)+(0*80)::print struct icommon
{
ic_smode = 0x41ed
ic_nlink = 0x2c
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x400
ic_atime = {
tv_sec = 0x46eb30e8
tv_usec = 0x992bc
}
<-- output omitted
ic_db = [ 0xc348, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}

<-- So /var has one block (ic_db[0] = 0xc348). This should be a directory. Now dump out some
<-- entries. The following is a good candidate for a walker...
(c348*400)::print struct direct
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(c348*400)+c::print struct direct
{
d_ino = 0x2
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(c348*400)+c+c::print struct direct
{
d_ino = 0x16c1
d_reclen = 0x10
d_namlen = 0x4
d_name = [ "sadm" ]
}

<-- let's check the work...
!ls -id /var/sadm
5825 /var/sadm
16c1=D
5825 <-- match looks good
<-- Ok. Now lets look at the inode for /var/sadm. This is
<-- inumber 16c1.

16c1%16c0=X
1 <-- the second cylinder group
16c1-(16c0*1)=X
1 <-- the second inode in the group

(c000*400*1)+(40*400)+(20*400)+(1*80)::print struct icommon
{
ic_smode = 0x41ed
ic_nlink = 0xd
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x200
<-- output omitted
ic_db = [ 0xc349, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}
c349*400::print struct direct
{
d_ino = 0x16c1
d_reclen = 0xc
d_namlen = 0x1
d_name = [ "." ]
}
(c349*400)+c::print struct direct
{
d_ino = 0x16c0
d_reclen = 0xc
d_namlen = 0x2
d_name = [ ".." ]
}
(c349*400)+c+c::print struct direct
{
d_ino = 0x16c2
d_reclen = 0x10
d_namlen = 0x7
d_name = [ "install" ]
}
(c349*400)+c+c+10::print struct direct
{
d_ino = 0x16c5
d_reclen = 0xc
d_namlen = 0x3
d_name = [ "pkg" ]
}
(c349*400)+c+c+10+c::print struct direct
{
d_ino = 0x1d41
d_reclen = 0x10
d_namlen = 0x6
d_name = [ "system" ]
}
(c349*400)+c+c+10+c+10::print struct direct
{
d_ino = 0x7e4d
d_reclen = 0x18
d_namlen = 0xc
d_name = [ "install_data" ]
}
(c349*400)+c+c+10+c+10+18::print struct direct
{
d_ino = 0x7e4e
d_reclen = 0x14
d_namlen = 0x8
d_name = [ "softinfo" ]
}
(c349*400)+c+c+10+c+10+18+14::print struct direct
{
d_ino = 0x27d9
d_reclen = 0x10
d_namlen = 0x6
d_name = [ "README" ] <-- here is the file we want to examine
}

27d9%16c0=X
1 <-- the second cylinder group
27d9-(1*16c0)=X
1119 <-- the 1120th inode
(1*c000*400)+(40*400)+(20*400)+(1119*80)::print struct icommon
{
ic_smode = 0x8124
ic_nlink = 0x1
ic_suid = 0
ic_sgid = 0x3
ic_lsize = 0x444
<-- output omitted
ic_db = [ 0x111cc, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ic_ib = [ 0, 0, 0 ]
<-- output omitted
}
111cc*400,200/c <-- ok, let's dump the first 512 bytes
0x4473000: -------------------------------
/var/sadm DIRECTORY RESTRUCTURE
-------------------------------

The system administration directory has been reorganized to bett
er
service the needs of Solaris administrators and administration s
oftware.
The old and new locations for files important to our customers a
re:

OLD LOCATION NEW LOCATION
------------ ------------
install_data/install_log system/logs/install_log
install_data/upgrade_log system/log

<-- check the work
!head /var/sadm/README
-------------------------------
/var/sadm DIRECTORY RESTRUCTURE
-------------------------------

The system administration directory has been reorganized to better
service the needs of Solaris administrators and administration software.
The old and new locations for files important to our customers are:

OLD LOCATION NEW LOCATION
------------ ------------
$q
#

bash-3.00$

Using dtrace and mdb to examine virtual memory

here is a short example using
dtrace and mdb to examine page faults and process address spaces.
This will be used in a workshop being given to professors teaching operating systems
within China. The workshop will cover Solaris internals using the opensolaris source code
and tools such as dtrace, mdb, and kmdb.

new web site for Bruning Systems

Hi,
I have finally gotten around to updating the web site. Still need to
add the dtrace scripts, but first I need to document them.

www.bruningsystems.com

max

snooping gld-based NIC drivers using dtrace

Hi.
I have just posted a script at http://www.bruningsystems.com/rtlsio.p
that allows one to snoop incoming/outgoing packets on a Realtek NIC.
The script is very easy to change for any other GLD-based NIC (see
the comment at the beginning of the script to determine what needs
to be changed.)
To run the script, save it and then:

# dtrace -q -C -s ./rtlsio.p

Let me know what you think.

max

dtrace script to trace kernel thread state changes

I just posted a script that traces kernel thread state changes to the dtrace
forum on forum.sun.com. You can also find it at http://www.bruningsystems.com/runq.d

Have fun.
max

solaris kernel memory usage

This is a test. I am currently working on answering a question
from a former student about kernel memory. The question and
answer will be here shortly.