DTrace

Pointers

• http://docs.sun.com/app/docs/doc/817-6223 - solaris dynamic tracing guide
• http://docs.sun.com/app/docs/doc/819-5488?a=load - dtrace user guide
• http://www.opensolaris.org/os/community/dtrace/ - community page
• http://www.sun.com/bigadmin/content/dtrace/ - bigadmin page
• http://www.sun.com/bigadmin/content/dtrace/dtrace_usenix.pdf - USENIX dtrace paper
• http://blogs.sun.com/roller/page/ahl/dtracesched

• DTraceJava
• DTraceOSX
• DTracePerl
• UnixGeekTools - other great tools for unix geeks
• http://blogs.sun.com/samf/entry/a_dtrace_provider_for_nfs - lots of pointers

DTraceToolkit

• http://www.opensolaris.org/os/community/dtrace/dtracetoolkit/
• http://www.context-switch.com/performance/dtrace.htm
• http://www.nbl.fi/~nbl97/solaris/dtrace/index.html
• http://www.samag.com/documents/s=9915/sam0512a/0512a.htm
• http://www.sun.com/bigadmin/content/submitted/dtrace_defunct_process.html
• http://www.sun.com/bigadmin/content/submitted/dtrace_with_ftp.html

Cookbook

• SolarisPerformance
• SolarisCPU

Java

Perl

• http://blogs.sun.com/alanbur/entry/dtrace_and_perl

Freebsd port

• http://people.freebsd.org/~jb/dtrace/index.html
• http://www.sitetronics.com/wordpress/?p=19 - blog
• http://www.zdnet.com.au/news/software/soa/Developer_aims_for_Dtrace_on_FreeBSD/0,2000061733,39210618,00.htm
• http://bsd.slashdot.org/bsd/05/09/08/1217229.shtml?tid=102&tid=7&tid=218

One-liners

• http://blogs.sun.com/bud/category/one-line-dtrace

# New processes with arguments,
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# Files opened by process,
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# Syscall count by program,
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# Syscall count by syscall,
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# Read bytes by process,
dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

# Write bytes by process,
dtrace -n 'sysinfo:::writech { @bytes[execname] = sum(arg0); }'

# Read size distribution by process,
dtrace -n 'sysinfo:::readch { @dist[execname] = quantize(arg0); }'

# Write size distribution by process,
dtrace -n 'sysinfo:::writech { @dist[execname] = quantize(arg0); }'

# Disk size by process,
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

# Pages paged in by process,
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

# Minor faults by process,
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'

note from sun

I've spoken to some of you about the new DTrace facility in Solaris 10. DTrace provides detailed and dynamic system and application (including Java) performance info even on a live production system. Customers and ISVs are using DTrace to get 2-1000x performance improvement out of their applications on Solaris and identify longstanding system performance issues that were previously impossible to troubleshoot. Technology is available on both the SPARC and Intel versions of S10. If you're interested in this technology a good starting point is:

http://www.sun.com/bigadmin/content/dtrace/

Since S10 does not FCS until December, some customers are using Solaris 10 DTrace (via Solaris http://wwws.sun.com/software/solaris/solaris-express/sol_index.html Express) in their test labs to do detailed analysis on application binaries deployed on S8 or earlier, fixing the bottlenecks using Dtrace on S10, and then returning the optimized code to production S8 systems until they are ready to move to S10.

I've also attached a technical whitepaper that explains how DTrace works and compares to other commercial and public-domain performance analysis technologies. Paper was presented at this year's USENIX Technical http://www.usenix.org/events/usenix04/index.html conference in Boston.

Let me know if you would be interested in a webinar on this technology.

--michael