Debugging Tools
This section describes two debuggers that can be applied to device drivers. Both
debuggers are described in detail in the Solaris Modular Debugger Guide.
The kmdb(1) kernel debugger provides typical runtime debugger facilities, such as breakpoints, watch points, and single-stepping. The kmdb debugger supersedes kadb, which was available in previous releases. The commands that were previously available from kadb are used in kmdb, in addition to new functionality. Where kadb could only be loaded at boot time, kmdb can be loaded at any time. The kmdb debugger is preferred for live, interactive debugging due to its execution controls.
The mdb(1) modular debugger is more limited than kmdb as a real-time debugger, but mdb has rich facilities for postmortem debugging.
The kmdb and mdb debuggers mostly share the same user interface. Many debugging
techniques therefore can be applied with the same commands in both tools. Both
debuggers support macros, dcmds, and dmods. A dcmd (pronounced dee-command) is a routine in
the debugger that can access any of the properties of the current target
program. A dcmd can be dynamically loaded at runtime. A dmod, which is
short for debugger module, is a package of dcmds that can be loaded
to provide non-standard behavior.
Both mdb and kmdb are backward-compatible with legacy debuggers such as adb and kadb.
The mdb debugger can execute all of the macros that are available to
kmdb as well as any legacy user-defined macros for adb. See the Solaris Modular Debugger Guide
for information about where to find standard macro sets.
Postmortem Debugging
Postmortem analysis offers numerous advantages to driver developers. More than one developer can
examine a problem in parallel. Multiple instances of the debugger can be used
simultaneously on a single crash dump. The analysis can be performed offline so
that the crashed system can be returned to service, if possible. Postmortem analysis
enables the use of user-developed debugger functionality in the form of dmods. Dmods
can bundle functionality that would be too memory-intensive for real-time debuggers, such as
kmdb.
When a system panics while kmdb is loaded, control is passed to the
debugger for immediate investigation. If kmdb does not seem appropriate for analyzing the
current problem, a good strategy is to use :c to continue execution and
save the crash dump. When the system reboots, you can perform postmortem analysis
with mdb on the saved crash dump. This process is analogous to debugging
an application crash from a process core file.
Note - In earlier versions of the Solaris operating system, adb(1) was the recommended tool
for postmortem analysis. In the current Solaris operating system, mdb(1) is the recommended tool
for postmortem analysis. The mdb() feature set surpasses the set of commands from
the legacy crash(1M) utility. The crash utility is no longer available in
the Solaris operating system.
Using the kmdb Kernel Debugger
The kmdb debugger is an interactive kernel debugger that provides the following capabilities:
Control of kernel execution
Inspection of the kernel state
Live modifications to the code
This section assumes that you are already familiar with the kmdb debugger. The
focus in this section is on kmdb capabilities that are useful in device
driver design. To learn how to use kmdb in detail, refer to the
kmdb(1) man page and to the Solaris Modular Debugger Guide. If you are familiar with kadb,
refer to the kadb(1M) man page for the major differences between kadb and
kmdb.
The kmdb debugger can be loaded and unloaded at will. Instructions for loading
and unloading kmdb are in the Solaris Modular Debugger Guide. For safety and convenience, booting with
an alternate kernel is highly encouraged. The boot process is slightly different between
the SPARC platform and the x86 platform, as described in this section.
Note - By default, kmdb uses the CPU ID as the prompt when kmdb is
running. In the examples in this chapter [0] is used as the prompt
unless otherwise noted.
Booting kmdb With an Alternate Kernel on the SPARC Platform
Use either of the following commands to boot a SPARC system with
both kmdb and an alternate kernel:
boot kmdb -D kernel.test/sparcv9/unix
boot kernel.test/sparcv9/unix -k
Booting kmdb With an Alternate Kernel on the x86 Platform
Use either of the following commands to boot an x86 system with
both kmdb and an alternate kernel:
b kmdb -D kernel.test/unix
b kernel.test/unix -k
Setting Breakpoints in kmdb
Use the bp command to set a breakpoint, as shown in the following
example.
Example 22-7 Setting Standard Breakpoints in kmdb
[0]> myModule`myBreakpointLocation::bp
If the target module has not been loaded, then an error message
that indicates this condition is displayed, and the breakpoint is not created. In this
case you can use a deferred breakpoint. A deferred breakpoint activates automatically when the
specified module is loaded. Set a deferred breakpoint by specifying the target location
after the bp command. The following example demonstrates a deferred breakpoint.
Example 22-8 Setting Deferred Breakpoints in kmdb
[0]>::bp myModule`myBreakpointLocation
For more information on using breakpoints, see the Solaris Modular Debugger Guide. You can also
get help by typing either of the following two lines:
> ::help bp
> ::bp dcmd
kmdb Macros for Driver Developers
The kmdb(1M) debugger supports macros that can be used to display kernel data
structures. Use $M to display kmdb macros. Macros are used in the
form:
[ address ] $<macroname
Note - Neither the information displayed by these macros nor the format in which the
information is displayed, constitutes an interface. Therefore, the information and format can change
at any time.
The kmdb macros in the following table are particularly useful to developers of
device drivers. For convenience, legacy macro names are shown where applicable.
Table 22-1 kmdb Macros
Dcmd |
Legacy Macro |
Description |
|---|
::devinfo |
devinfo devinfo_brief devinfo.prop |
Print a
summary of a device node |
::walk devinfo_parents |
devinfo.parent |
Walk the ancestors of a device node |
::walk devinfo_sibling |
devinfo.sibling |
Walk the
siblings of a device node |
::minornodes |
devinfo.minor |
Print the minor nodes that correspond to the
given device node |
::major2name |
|
Print the name of a device that is bound to
a given device node. |
::devbindings |
|
Print the device nodes that are bound to a
given device node or major number. |
The ::devinfo dcmd displays a node state that can have one of the
following values:
- DS_ATTACHED
The driver's attach(9E) routine returned successfully.
- DS_BOUND
The node is bound to a driver, but the driver's probe(9E) routine has not yet been called.
- DS_INITIALIZED
The parent nexus has assigned a bus address for the driver. The implementation-specific initializations have been completed. The driver's probe(9E) routine has not yet been called at this point.
- DS_LINKED
The device node has been linked into the kernel's device tree, but the system has not yet found a driver for this node.
- DS_PROBED
The driver's probe(9E) routine returned successfully.
- DS_READY
The device is fully configured.
Using the mdb Modular Debugger
The mdb(1) modular debugger can be applied to the following types of files:
The mdb debugger provides sophisticated debugging support for analyzing kernel problems. This section
provides an overview of mdb features. For a complete discussion of mdb, refer
to the Solaris Modular Debugger Guide.
Although mdb can be used to alter live kernel state, mdb lacks
the kernel execution control that is provided by kmdb. As a result
kmdb is preferred for runtime debugging. The mdb debugger is used more for
static situations.
Note - The prompt for mdb is >.
Getting Started With the Modular Debugger
The mdb debugger provides an extensive programming API for implementing debugger modules so
that driver developers can implement custom debugging support. The mdb debugger also provides many
usability features, such as command-line editing, command history, an output pager, and online
help.
Note - The adb macros should no longer be used. That functionality has largely been
superseded by the dcmds in mdb.
The mdb debugger provides a rich set of modules and dcmds. With these
tools, you can debug the Solaris kernel, any associated modules, and device drivers.
These facilities enable you to perform tasks such as:
Formulate complex debugging queries
Locate all the memory allocated by a particular thread
Print a visual picture of a kernel STREAM
Determine what type of structure a particular address refers to
Locate leaked memory blocks in the kernel
Analyze memory to locate stack traces
Assemble dcmds into modules called dmods for creating customized operations
To get started, switch to the crash directory and type mdb, specifying a
system crash dump, as illustrated in the following example.
Example 22-9 Invoking mdb on a Crash Dump
% cd /var/crash/testsystem
% ls
bounds unix.0 vmcore.0
% mdb unix.0 vmcore.0
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 cpc nfs ]
> ::status
debugging crash dump vmcore.0 (64-bit) from testsystem
operating system: 5.10 Generic (sun4u)
panic message: zero
dump content: kernel pages only
When mdb responds with the > prompt, you can run commands.
To examine the running kernel on a live system, run mdb from the
system prompt as follows.
Example 22-10 Invoking mdb on a Running Kernel
# mdb -k
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 ptm cpc ipc nfs ]
> ::status
debugging live kernel (64-bit) on testsystem
operating system: 5.10 Generic (sun4u)
Useful Debugging Tasks With kmdb and mdb
This section provides examples of useful debugging tasks. The tasks in this section
can be performed with either mdb or kmdb unless specifically noted. This section assumes
a basic knowledge of the use of kmdb and mdb. Note that
the information presented here is dependent on the type of system used. A
Sun BladeTM 100 workstation running the 64-bit kernel was used to produce these examples.
Caution - Because irreversible destruction of data can result from modifying data in kernel structures,
you should exercise extreme caution. Do not modify or rely on data in
structures that are not part of the Solaris DDI. See the Intro(9S)
man page for information on structures that are part of the Solaris DDI.
Exploring System Registers With kmdb
The kmdb debugger can display machine registers as a group or individually. To
display all registers as a group, use $r as shown in the following
example.
Example 22-11 Reading All Registers on a SPARC Processor With kmdb
[0]: $r
g0 0 l0 0
g1 100130a4 debug_enter l1 edd00028
g2 10411c00 tsbmiss_area+0xe00 l2 10449c90
g3 10442000 ti_statetbl+0x1ba l3 1b
g4 3000061a004 l4 10474400 ecc_syndrome_tab+0x80
g5 0 l5 3b9aca00
g6 0 l6 0
g7 2a10001fd40 l7 0
o0 0 i0 0
o1 c i1 10449e50
o2 20 i2 0
o3 300006b2d08 i3 10
o4 0 i4 0
o5 0 i5 b0
sp 2a10001b451 fp 2a10001b521
o7 1001311c debug_enter+0x78 i7 1034bb24 zsa_xsint+0x2c4
y 0
tstate: 1604 (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4)
pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0
winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14
tba 0x10000000
pc edd000d8 edd000d8: ta %icc,%g0 + 125
npc edd000dc edd000dc: nop
The debugger exports each register value to a variable with the same name
as the register. If you read the variable, the current value of
the register is returned. If you write to the variable, the value of
the associated machine register is changed. The following example changes the value of
the %o0 register from 0 to 1 on an x86 machine.
Example 22-12 Reading and Writing Registers on an x86 Machine With kmdb
[0]> <eax=K
c1e6e0f0
[0]> 0>eax
[0]> <eax=K
0
[0]> c1e6e0f0>eaxIf you need to inspect the registers of a different processor, you
can use the ::cpuregs dcmd. The ID of the processor to be examined can
be supplied as either the address to the dcmd or as the
value of the -c option, as shown in the following example.
Example 22-13 Inspecting the Registers of a Different Processor
[0]> 0::cpuregs
%cs = 0x0158 %eax = 0xc1e6e0f0 kmdbmod`kaif_dvec
%ds = 0x0160 %ebx = 0x00000000
The following example switches from processor 0 to processor 3 on a SPARC
machine. The %g3 register is inspected and then cleared. To confirm the new
value, %g3 is read again.
Example 22-14 Retrieving the Value of an Individual Register From a Specified Processor
[0]> 3::switch
[3]> <g3=K
24
[3]> 0>g3
[3]> <g3
0
Detecting Kernel Memory Leaks
The ::findleaks dcmd provides powerful, efficient detection of memory leaks in kernel crash
dumps. The full set of kernel-memory debugging features must be enabled for ::findleaks
to be effective. For more information, see Setting kmem_flags Debugging Flags. Run ::findleaks during driver development and
testing to detect code that leaks memory, thus wasting kernel resources. See Chapter 9, Debugging With the Kernel Memory Allocator, in Solaris Modular Debugger Guide
for a complete discussion of ::findleaks.
Note - Code that leaks kernel memory can render the system vulnerable to denial-of-service attacks.
Writing Debugger Commands With mdb
The mdb debugger provides a powerful API for implementing debugger facilities that you
customize to debug your driver. The Solaris Modular Debugger Guide explains the programming API in detail.
The SUNWmdbdm package installs sample mdb source code in the directory /usr/demo/mdb. You
can use mdb to automate lengthy debugging chores or help to validate
that your driver is behaving properly. You can also package your mdb debugging modules
with your driver product. With packaging, these facilities are available to service personnel
at a customer site.
Obtaining Kernel Data Structure Information
The Solaris kernel provides data type information in structures that can be inspected
with either kmdb or mdb.
Note - The kmdb and mdb dcmds can be used only with objects that contain
compressed symbolic debugging information that has been designed for use with mdb. This
information is currently available only for certain Solaris kernel modules. The SUNWzlib package
must be installed to process the symbolic debugging information.
The following example demonstrates how to display the data in the scsi_pkt
structure.
Example 22-15 Displaying Kernel Data Structures With a Debugger
> 7079ceb0::print -t 'struct scsi_pkt'
{
opaque_t pkt_ha_private = 0x7079ce20
struct scsi_address pkt_address = {
struct scsi_hba_tran *a_hba_tran = 0x70175e68
ushort_t a_target = 0x6
uchar_t a_lun = 0
uchar_t a_sublun = 0
}
opaque_t pkt_private = 0x708db4d0
int (*)() *pkt_comp = sd_intr
uint_t pkt_flags = 0
int pkt_time = 0x78
uchar_t *pkt_scbp = 0x7079ce74
uchar_t *pkt_cdbp = 0x7079ce64
ssize_t pkt_resid = 0
uint_t pkt_state = 0x37
uint_t pkt_statistics = 0
uchar_t pkt_reason = 0
}The size of a data structure can be useful in debugging. Use
the ::sizeof dcmd to obtain the size of a structure, as shown
in the following example.
Example 22-16 Displaying the Size of a Kernel Data Structure
> ::sizeof struct scsi_pkt
sizeof (struct scsi_pkt) = 0x58
The address of a specific member within a structure is also useful
in debugging. Several methods are available for determining a member's address.
Use the ::offsetof dcmd to obtain the offset for a given member of
a structure, as in the following example.
Example 22-17 Displaying the Offset to a Kernel Data Structure
> ::offsetof struct scsi_pkt pkt_state
offsetof (struct pkt_state) = 0x48
Use the ::print dcmd with the -a option to display the addresses of
all members of a structure, as in the following example.
Example 22-18 Displaying the Relative Addresses of a Kernel Data Structure
> ::print -a struct scsi_pkt
{
0 pkt_ha_private
8 pkt_address {
...
}
18 pkt_private
...
}If an address is specified with ::print in conjunction with the -a option,
the absolute address for each member is displayed.
Example 22-19 Displaying the Absolute Addresses of a Kernel Data Structure
> 10000000::print -a struct scsi_pkt
{
10000000 pkt_ha_private
10000008 pkt_address {
...
}
10000018 pkt_private
...
}The ::print, ::sizeof and ::offsetof dcmds enable you to debug problems when
your driver interacts with the Solaris kernel.
Caution - This facility provides access to raw kernel data structures. You can examine any
structure whether or not that structure appears as part of the DDI. Therefore,
you should refrain from relying on any data structure that is not explicitly
part of the DDI.
Note - These dcmds should be used only with objects that contain compressed symbolic debugging
information that has been designed for use with mdb. Symbolic debugging information is
currently available for certain Solaris kernel modules only. The SUNWzlib (32-bit) or SUNWzlibx
(64-bit) decompression software must be installed to process the symbolic debugging information. The
kmdb debugger can process symbolic type data with or without the SUNWzlib or
SUNWzlibx packages.
Obtaining Device Tree Information
The mdb debugger provides the ::prtconf dcmd for displaying the kernel device tree.
The output of the ::prtconf dcmd is similar to the output of the
prtconf(1M) command.
Example 22-20 Using the ::prtconf Dcmd
> ::prtconf
300015d3e08 SUNW,Sun-Blade-100
300015d3c28 packages (driver not attached)
300015d3868 SUNW,builtin-drivers (driver not attached)
300015d3688 deblocker (driver not attached)
300015d34a8 disk-label (driver not attached)
300015d32c8 terminal-emulator (driver not attached)
300015d30e8 obp-tftp (driver not attached)
300015d2f08 dropins (driver not attached)
300015d2d28 kbd-translator (driver not attached)
300015d2b48 ufs-file-system (driver not attached)
300015d3a48 chosen (driver not attached)
300015d2968 openprom (driver not attached)You can display the node by using a macro, such as
the ::devinfo dcmd, as shown in the following example.
Example 22-21 Displaying Device Information for an Individual Node
> 300015d3e08::devinfo
300015d3e08 SUNW,Sun-Blade-100
System properties at 0x300015abdc0:
name='relative-addressing' type=int items=1
value=00000001
name='MMU_PAGEOFFSET' type=int items=1
value=00001fff
name='MMU_PAGESIZE' type=int items=1
value=00002000
name='PAGESIZE' type=int items=1
value=00002000
Driver properties at 0x300015abe00:
name='pm-hardware-state' type=string items=1
value='no-suspend-resume'Use ::prtconf to see where your driver has attached in the device tree,
and to display device properties. You can also specify the verbose (-v) flag
to ::prtconf to display the properties for each device node, as follows.
Example 22-22 Using the ::prtconf Dcmd in Verbose Mode
> ::prtconf -v
DEVINFO NAME
300015d3e08 SUNW,Sun-Blade-100
System properties at 0x300015abdc0:
name='relative-addressing' type=int items=1
value=00000001
name='MMU_PAGEOFFSET' type=int items=1
value=00001fff
name='MMU_PAGESIZE' type=int items=1
value=00002000
name='PAGESIZE' type=int items=1
value=00002000
Driver properties at 0x300015abe00:
name='pm-hardware-state' type=string items=1
value='no-suspend-resume'
...
300015ce798 pci10b9,5229, instance #0
Driver properties at 0x300015ab980:
name='target2-dcd-options' type=any items=4
value=00.00.00.a4
name='target1-dcd-options' type=any items=4
value=00.00.00.a2
name='target0-dcd-options' type=any items=4
value=00.00.00.a4Another way to locate instances of your driver is the ::devbindings dcmd.
Given a driver name, the command displays a list of all instances of
the named driver as demonstrated in the following example.
Example 22-23 Using the ::devbindings Dcmd to Locate Driver Instances
> ::devbindings dad
300015ce3d8 ide-disk (driver not attached)
300015c9a60 dad, instance #0
System properties at 0x300015ab400:
name='lun' type=int items=1
value=00000000
name='target' type=int items=1
value=00000000
name='class_prop' type=string items=1
value='ata'
name='type' type=string items=1
value='ata'
name='class' type=string items=1
value='dada'
...
300015c9880 dad, instance #1
System properties at 0x300015ab080:
name='lun' type=int items=1
value=00000000
name='target' type=int items=1
value=00000002
name='class_prop' type=string items=1
value='ata'
name='type' type=string items=1
value='ata'
name='class' type=string items=1
value='dada'
Retrieving Driver Soft State Information
A common problem when debugging a driver is retrieving the soft state for
a particular driver instance. The soft state is allocated with the ddi_soft_state_zalloc(9F) routine.
The driver can obtain the soft state through ddi_get_soft_state(9F). The name of the
soft state pointer is the first argument to ddi_soft_state_init(9F)). With the name, you can use
mdb to retrieve the soft state for a particular driver instance through the
::softstate dcmd:
> *bst_state::softstate 0x3
702b7578
In this case, ::softstate is used to fetch the soft state for instance
3 of the bst sample driver. This pointer references a bst_soft structure that
is used by the driver to track state for this instance.
Modifying Kernel Variables
You can use both kmdb and mdb to modify kernel variables or other
kernel state. Kernel state modification with mdb should be done with care,
because mdb does not stop the kernel before making modifications. Groups of modifications
can be made atomically by using kmdb, because kmdb stops the kernel before allowing
access by the user. The mdb debugger is capable of making single atomic
modifications only.
Be sure to use the proper format specifier to perform the
modification. The formats are:
w – Writes the lowest two bytes of the value of each expression to the target beginning at the location specified by dot
W – Writes the lowest 4 bytes of the value of each expression to the target beginning at the location specified by dot
Z – Write the complete 8 bytes of the value of each expression to the target beginning at the location specified by dot
Use the ::sizeof dcmd to determine the size of the variable to be
modified.
The following example overwrites the value of moddebug with the value 0x80000000.
Example 22-24 Modifying a Kernel Variable With a Debugger
> moddebug/W 0x80000000
moddebug: 0 = 0x80000000
