[dtrace-discuss] University of Toronto's JIFL OS instrumentation system

[Bryan Cantrill]

On Sat, Jul 28, 2007 at 06:40:39PM -0700, Adam Leventhal wrote:
> On Thu, Jul 12, 2007 at 04:49:13PM -0600, Peter Schow wrote:
> > DTracers might be interested in reading about other dynamic OS 
> > instrumentation technologies, like the University of Toronto's 
> > JIFL system:
> > 
> >    http://individual.utoronto.ca/marek/jifl-eurosys07.pdf
> > 
> > They actually mention DTrace a few times, including in the "Related
> > Work" section but seem to imply that they haven't tried it.
> 
> They did more than not try it -- they pretty much wrote it off. The focus
> on this paper was efficient instrumention, and they asserted that trap-
> based instrumentation would be slower than what they had developed; it
> would have been nice for them to confirm this about DTrace.
> 
> The paper had some interesting ideas (though I'm not sure how much they
> went beyond the work done UW-Madison by Barton Miller et al.), but proposed
> some kernel instrumentation techniques which were a bit too exhilarating for
> my tastes.

I think Adam is being too charitable; at best, this paper represents
dated, pre-DTrace notions of instrumentation.  In particular, I believe
that our most important contribution is the importance of instrumenting
systems in _production_.  The performance ramifications of this are clear,
and they are exactly the opposite of the prevailing wisdom prior to DTrace:
a dynamic instrumentation framework must have zero _disabled_ probe effect.
The _enabled_ probe effect is actually not nearly as important as getting
the disabled probe effect to zero, for it is the zero disabled probe effect
that allows you to have the instrumentation that you need when you're going
to need it.  (Like this work, virtually all work prior to DTrace focussed
exclusively on minimizing the enabled probe effect, while cheerfully
ignoring a non-zero disabled probe effect.)  Indeed, this paragraph from
the paper sums up their misunderstanding of DTrace's fundamental
contribution:

   JIFL incurs a performance cost of approximately 2% running with
   no instrumentation due to JIT overhead and cost of executing from
   the code cache.  As noted earlier, this result it included to
   illustrate the cost of the JIFL framework -- if no instrumentation
   is wanted JIFL would be disabled, incurring no overhead.

This represents an old developer mindset around dynamic instrumentation --
that one should make an upfront, non-reversible (for the lifetime of the
running application/system) decision about whether instrumentation is
"wanted."  As many of you already know, you often don't know that you
want instrumentation until you need it -- and with their dated approach,
it's too late.

I hasten to add that there are all also safety ramifications of running
in production, none of which they seem to understand or appreciate.  They
blow off these ramifications as "straightforward to add" in their section
discussing DTrace -- which is pretty rich considering that they acknowledge
in the next section that their work is not even preemption-safe.

Will researchers ever start looking at the many new problems that DTrace
has opened instead of the old problems that DTrace has solved?  One hopes.

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc