[DTrace] Trace Memory Access
pdu at cse.unl.edu
pdu at cse.unl.edu
Sat Feb 14 12:22:23 PST 2009
Dear Michael
Thank you for you informative and helpful post. I really appreciate the
hard work you have put into this.
With respect to the solution, I have some questions. First, so pid
provider can provide user process instruction level tracing. However, what
if I want to monitor the memory access in the entire process, not some
particular functions.
I suspect this has to be an easier way to achieve this instrumentation.
Actually, the application I am tracing is the OpenJDK Hotspot virtual
machine. I've been trying to approach this in several directions:
1. Dtrace
2. Dynamic Binary Instrumentation (Valgrind, Pin, Dyninst)
3. Hotspot Dtrace Provider
4. Modify Linux kernel.
5. Modify Xen virtual machine.
6. Add a read-barrier to Hotspot.
So far, Dtrace seems not directly support this. I could get some results
by using DBI frameworks, though not very desirable. I consulted the manual
of the Hotspot provider manual
(http://java.sun.com/javase/6/docs/technotes/guides/vm/dtrace.html), no
luck either.
The 4th and 5th options need lot of work and frankly speaking, there are
way too many things going on inside the kernel/VM that I can capture. The
6th option is my last resort, my colleague might have already implemented
a read-barrier, but it works only in the interpreter mode, not native
mode. Still not very desirable and need some integration work.
Anyway, thanks a lot for your help! Looking forward to your further thoughts.
Peng Du
Feb 14, 2009
> Hey Peng -
>
> I might have asked about what you've done so far. :) Thanks for sharing
> that
> information.
>
> I've been playing a bit with the pid provider lately, so I thought about
> your question at the instruction-level. Of course Dtrace lets you trace
> user-level function boundaries:
>
> dtrace -n 'pid{PID}:a.out:main:entry ....'
>
> Just specify the PID you want to scope, omit the brackets. In this example
> I
> scope the text segment and main function within it. The probe names entry
> and return apply to any function and give respective access to the
> incoming
> args and return values. You can also pinpoint an instruction at it's
> byte-aligned address:
>
> dtrace -n 'pid{PID}:a.out:main:8...'
>
> at which point you can read register values through the uregs[] constants
> provided by Dtrace. There's a list on p. 351 in the Solaris Dynamic
> Tracing
> Guide (part #817-6223-12), or somewhere in the Dtrace wiki at
> http://solarisinternals.com.
>
> I've been playing with this (and absorbing a bit of assemby along the
> way),
> so I wondered how to retrieve the instruction itself, grab the memory
> address arg if relevant, etc. Unnecessary, perhaps, for the question at
> hand. I suspect it would also intrude on your application heavily,
> although
> some combination of speculative tracing and buffer tuning might mitigate
> that. But this is guesswork at the moment on my part.
>
> Unfortunately most of the messages to this list I have seen are spam. It
> may
> be that no one with more deft Dtrace ability is listening. But I'll chew
> on
> this a little and see if I cannot provide a concrete and working example
> for
> you.
>
> Regards,
>
> Michael
>
> On Fri, Feb 13, 2009 at 10:10 PM, Peng Du <pdu at cse.unl.edu> wrote:
>
>> Thanks, Michael
>>
>> Though I can't fully understand your post as a newbie, I got the idea.
>> Can I assume tracing memory read/write is doable using Dtrace?
>>
>> Let me state my problem more specifically. Here I have a very large and
>> complex C++ application. And I want to understand the memory usage
>> pattern within this app. Page level tracing, say 4K in x86, is totally
>> sufficient. I want to create an array of counters, each per page in the
>> 4G address space. Whenever a memory is accessed, the counter of the page
>> that contains the address will be incremented.
>>
>> I've done this in Valgrind in Linux. However the Valgrind trace slowed
>> down the execution significantly and it serializes thread executions,
>> which is not very realistic in the normal scenario. So, the result I got
>> might not be representative.
>>
>> Would you elaborate more on your idea? A more concrete example will be
>> much appreciated!
>>
>> Thanks a lot!
>>
>>
>> Peng Du
>> Feb 14, 2009
>>
>>
>>
>>
>>
>> On Fri, 2009-02-13 at 21:50 -0800, Michael Ernest wrote:
>> > Hello Peng -
>> >
>> > It is possible to instrument an application at the instruction level
>> > and record register values in view at any point. I suppose you could
>> > guess read the register that would contain a memory address and send
>> > it to buffer.
>> >
>> > I haven't thought this completely through, but I imagine you'd want to
>> > use speculative tracing to commit certain register values to your
>> > Dtrace buffers when they are interesting, and discard them otherwise.
>> > I suspect in this kind of exercise you will fire a lot of probes and
>> > overrun the buffers quickly if you aren't careful.
>> >
>> > Regards,
>> >
>> > Michael
>> >
>> >
>> > On Thu, Feb 12, 2009 at 8:23 PM, Peng Du <pdu at cse.unl.edu> wrote:
>> > Hello, everyone
>> >
>> > I am a newbie of DTrace and OpenSolaris. Just wondering is it
>> > possible
>> > to trace every memory read/write of a specific process given
>> > the pid?
>> > Basically speaking, I want to record the virtual addresses
>> > being
>> > accessed by the process, using which I will try to do some
>> > online
>> > analysis.
>> >
>> > If DTrace does not support this type of instrumentation, will
>> > other
>> > tools in OpenSolaris help me achieve that, such as mdb, and
>> > etc?
>> >
>> > Thanks!
>> >
>> > Peng Du
>> > Feb 12, 2009
>> >
>> > _______________________________________________
>> > DTrace mailing list
>> > DTrace at opensolaris.org
>> > http://mail.opensolaris.org/mailman/listinfo/dtrace
>> >
>> >
>> >
>> > --
>> > Michael Ernest
>> > Inkling Research, Inc.
>>
>>
>
>
> --
> Michael Ernest
> Inkling Research, Inc.
>
More information about the DTrace
mailing list