From pdu at cse.unl.edu  Thu Feb 12 20:23:24 2009
From: pdu at cse.unl.edu (Peng Du)
Date: Thu, 12 Feb 2009 22:23:24 -0600
Subject: [DTrace] Trace Memory Access
Message-ID: <1234499004.22413.31.camel@localhost>

Hello, everyone

I am a newbie of DTrace and OpenSolaris. Just wondering is it possible
to trace every memory read/write of a specific process given the pid?
Basically speaking, I want to record the virtual addresses being
accessed by the process, using which I will try to do some online
analysis. 

If DTrace does not support this type of instrumentation, will other
tools in OpenSolaris help me achieve that, such as mdb, and etc? 

Thanks! 

Peng Du
Feb 12, 2009



From michael.ernest at gmail.com  Fri Feb 13 21:50:35 2009
From: michael.ernest at gmail.com (Michael Ernest)
Date: Fri, 13 Feb 2009 21:50:35 -0800
Subject: [DTrace] Trace Memory Access
In-Reply-To: <1234499004.22413.31.camel@localhost>
References: <1234499004.22413.31.camel@localhost>
Message-ID: <25db11d0902132150u48f683at3581f5471291367c@mail.gmail.com>

Hello Peng -

It is possible to instrument an application at the instruction level and
record register values in view at any point. I suppose you could guess read
the register that would contain a memory address and send it to buffer.

I haven't thought this completely through, but I imagine you'd want to use
speculative tracing to commit certain register values to your Dtrace buffers
when they are interesting, and discard them otherwise. I suspect in this
kind of exercise you will fire a lot of probes and overrun the buffers
quickly if you aren't careful.

Regards,

Michael


On Thu, Feb 12, 2009 at 8:23 PM, Peng Du <pdu at cse.unl.edu> wrote:

> Hello, everyone
>
> I am a newbie of DTrace and OpenSolaris. Just wondering is it possible
> to trace every memory read/write of a specific process given the pid?
> Basically speaking, I want to record the virtual addresses being
> accessed by the process, using which I will try to do some online
> analysis.
>
> If DTrace does not support this type of instrumentation, will other
> tools in OpenSolaris help me achieve that, such as mdb, and etc?
>
> Thanks!
>
> Peng Du
> Feb 12, 2009
>
> _______________________________________________
> DTrace mailing list
> DTrace at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/dtrace
>



-- 
Michael Ernest
Inkling Research, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.opensolaris.org/pipermail/dtrace/attachments/20090213/6fe82fbb/attachment.html>

From pdu at cse.unl.edu  Fri Feb 13 22:10:03 2009
From: pdu at cse.unl.edu (Peng Du)
Date: Sat, 14 Feb 2009 00:10:03 -0600
Subject: [DTrace] Trace Memory Access
In-Reply-To: <25db11d0902132150u48f683at3581f5471291367c@mail.gmail.com>
References: <1234499004.22413.31.camel@localhost>
	<25db11d0902132150u48f683at3581f5471291367c@mail.gmail.com>
Message-ID: <1234591803.2228.9.camel@localhost>

Thanks, Michael

Though I can't fully understand your post as a newbie, I got the idea.
Can I assume tracing memory read/write is doable using Dtrace? 

Let me state my problem more specifically. Here I have a very large and
complex C++ application. And I want to understand the memory usage
pattern within this app. Page level tracing, say 4K in x86, is totally
sufficient. I want to create an array of counters, each per page in the
4G address space. Whenever a memory is accessed, the counter of the page
that contains the address will be incremented. 

I've done this in Valgrind in Linux. However the Valgrind trace slowed
down the execution significantly and it serializes thread executions,
which is not very realistic in the normal scenario. So, the result I got
might not be representative. 

Would you elaborate more on your idea? A more concrete example will be
much appreciated! 

Thanks a lot! 


Peng Du
Feb 14, 2009





On Fri, 2009-02-13 at 21:50 -0800, Michael Ernest wrote:
> Hello Peng -
> 
> It is possible to instrument an application at the instruction level
> and record register values in view at any point. I suppose you could
> guess read the register that would contain a memory address and send
> it to buffer.
> 
> I haven't thought this completely through, but I imagine you'd want to
> use speculative tracing to commit certain register values to your
> Dtrace buffers when they are interesting, and discard them otherwise.
> I suspect in this kind of exercise you will fire a lot of probes and
> overrun the buffers quickly if you aren't careful.
> 
> Regards,
> 
> Michael
> 
> 
> On Thu, Feb 12, 2009 at 8:23 PM, Peng Du <pdu at cse.unl.edu> wrote:
>         Hello, everyone
>         
>         I am a newbie of DTrace and OpenSolaris. Just wondering is it
>         possible
>         to trace every memory read/write of a specific process given
>         the pid?
>         Basically speaking, I want to record the virtual addresses
>         being
>         accessed by the process, using which I will try to do some
>         online
>         analysis.
>         
>         If DTrace does not support this type of instrumentation, will
>         other
>         tools in OpenSolaris help me achieve that, such as mdb, and
>         etc?
>         
>         Thanks!
>         
>         Peng Du
>         Feb 12, 2009
>         
>         _______________________________________________
>         DTrace mailing list
>         DTrace at opensolaris.org
>         http://mail.opensolaris.org/mailman/listinfo/dtrace
> 
> 
> 
> -- 
> Michael Ernest
> Inkling Research, Inc.



From pdu at cse.unl.edu  Sat Feb 14 12:22:23 2009
From: pdu at cse.unl.edu (pdu at cse.unl.edu)
Date: Sat, 14 Feb 2009 14:22:23 -0600 (CST)
Subject: [DTrace] Trace Memory Access
In-Reply-To: <25db11d0902141001h45b65a0s4b7e2e339b07456a@mail.gmail.com>
References: <1234499004.22413.31.camel@localhost>
	<25db11d0902132150u48f683at3581f5471291367c@mail.gmail.com>
	<1234591803.2228.9.camel@localhost>
	<25db11d0902141001h45b65a0s4b7e2e339b07456a@mail.gmail.com>
Message-ID: <33238.129.93.158.28.1234642943.squirrel@cse-mail.unl.edu>

Dear Michael

Thank you for you informative and helpful post. I really appreciate the
hard work you have put into this.

With respect to the solution, I have some questions. First, so pid
provider can provide user process instruction level tracing. However, what
if I want to monitor the memory access in the entire process, not some
particular functions.

I suspect this has to be an easier way to achieve this instrumentation.
Actually, the application I am tracing is the OpenJDK Hotspot virtual
machine. I've been trying to approach this in several directions:

1. Dtrace
2. Dynamic Binary Instrumentation (Valgrind, Pin, Dyninst)
3. Hotspot Dtrace Provider
4. Modify Linux kernel.
5. Modify Xen virtual machine.
6. Add a read-barrier to Hotspot.

So far, Dtrace seems not directly support this. I could get some results
by using DBI frameworks, though not very desirable. I consulted the manual
of the Hotspot provider manual
(http://java.sun.com/javase/6/docs/technotes/guides/vm/dtrace.html), no
luck either.

The 4th and 5th options need lot of work and frankly speaking, there are
way too many things going on inside the kernel/VM that I can capture. The
6th option is my last resort, my colleague might have already implemented
a read-barrier, but it works only in the interpreter mode, not native
mode. Still not very desirable and need some integration work.

Anyway, thanks a lot for your help! Looking forward to your further thoughts.


Peng Du
Feb 14, 2009


> Hey Peng -
>
> I might have asked about what you've done so far. :) Thanks for sharing
> that
> information.
>
> I've been playing a bit with the pid provider lately, so I thought about
> your question at the instruction-level. Of course Dtrace lets you trace
> user-level function boundaries:
>
> dtrace -n 'pid{PID}:a.out:main:entry ....'
>
> Just specify the PID you want to scope, omit the brackets. In this example
> I
> scope the text segment and main function within it. The probe names entry
> and return apply to any function and give respective access to the
> incoming
> args and return values. You can also pinpoint an instruction at it's
> byte-aligned address:
>
> dtrace -n 'pid{PID}:a.out:main:8...'
>
> at which point you can read register values through the uregs[] constants
> provided by Dtrace. There's a list on p. 351 in the Solaris Dynamic
> Tracing
> Guide (part #817-6223-12), or somewhere in the Dtrace wiki at
> http://solarisinternals.com.
>
> I've been playing with this (and absorbing a bit of assemby along the
> way),
> so I wondered how to retrieve the instruction itself, grab the memory
> address arg if relevant, etc. Unnecessary, perhaps, for the question at
> hand. I suspect it would also intrude on your application heavily,
> although
> some combination of speculative tracing and buffer tuning might mitigate
> that. But this is guesswork at the moment on my part.
>
> Unfortunately most of the messages to this list I have seen are spam. It
> may
> be that no one with more deft Dtrace ability is listening. But I'll chew
> on
> this a little and see if I cannot provide a concrete and working example
> for
> you.
>
> Regards,
>
> Michael
>
> On Fri, Feb 13, 2009 at 10:10 PM, Peng Du <pdu at cse.unl.edu> wrote:
>
>> Thanks, Michael
>>
>> Though I can't fully understand your post as a newbie, I got the idea.
>> Can I assume tracing memory read/write is doable using Dtrace?
>>
>> Let me state my problem more specifically. Here I have a very large and
>> complex C++ application. And I want to understand the memory usage
>> pattern within this app. Page level tracing, say 4K in x86, is totally
>> sufficient. I want to create an array of counters, each per page in the
>> 4G address space. Whenever a memory is accessed, the counter of the page
>> that contains the address will be incremented.
>>
>> I've done this in Valgrind in Linux. However the Valgrind trace slowed
>> down the execution significantly and it serializes thread executions,
>> which is not very realistic in the normal scenario. So, the result I got
>> might not be representative.
>>
>> Would you elaborate more on your idea? A more concrete example will be
>> much appreciated!
>>
>> Thanks a lot!
>>
>>
>> Peng Du
>> Feb 14, 2009
>>
>>
>>
>>
>>
>> On Fri, 2009-02-13 at 21:50 -0800, Michael Ernest wrote:
>> > Hello Peng -
>> >
>> > It is possible to instrument an application at the instruction level
>> > and record register values in view at any point. I suppose you could
>> > guess read the register that would contain a memory address and send
>> > it to buffer.
>> >
>> > I haven't thought this completely through, but I imagine you'd want to
>> > use speculative tracing to commit certain register values to your
>> > Dtrace buffers when they are interesting, and discard them otherwise.
>> > I suspect in this kind of exercise you will fire a lot of probes and
>> > overrun the buffers quickly if you aren't careful.
>> >
>> > Regards,
>> >
>> > Michael
>> >
>> >
>> > On Thu, Feb 12, 2009 at 8:23 PM, Peng Du <pdu at cse.unl.edu> wrote:
>> >         Hello, everyone
>> >
>> >         I am a newbie of DTrace and OpenSolaris. Just wondering is it
>> >         possible
>> >         to trace every memory read/write of a specific process given
>> >         the pid?
>> >         Basically speaking, I want to record the virtual addresses
>> >         being
>> >         accessed by the process, using which I will try to do some
>> >         online
>> >         analysis.
>> >
>> >         If DTrace does not support this type of instrumentation, will
>> >         other
>> >         tools in OpenSolaris help me achieve that, such as mdb, and
>> >         etc?
>> >
>> >         Thanks!
>> >
>> >         Peng Du
>> >         Feb 12, 2009
>> >
>> >         _______________________________________________
>> >         DTrace mailing list
>> >         DTrace at opensolaris.org
>> >         http://mail.opensolaris.org/mailman/listinfo/dtrace
>> >
>> >
>> >
>> > --
>> > Michael Ernest
>> > Inkling Research, Inc.
>>
>>
>
>
> --
> Michael Ernest
> Inkling Research, Inc.
>