From max at axiomax.com Sun Jan 2 15:07:47 2005 From: max at axiomax.com (Max Bruning) Date: Sun, 02 Jan 2005 15:07:47 PST Subject: [DTrace] Interesting discussion on dtrace printf issues Message-ID: <29846703.1104707267517.JavaMail.nobody@swforum2> Hello, I thought you might be interested in this topic in the Solaris Forums: "dtrace printf issues", http://forum.sun.com/thread.jspa?threadID=22110 From philip.beevers at fidessa.com Sat Jan 1 13:12:00 2005 From: philip.beevers at fidessa.com (Philip Beevers) Date: Sat, 1 Jan 2005 21:12:00 -0000 Subject: [DTrace] using C preprocessor in dtrace scripts Message-ID: <8B5A5401A2DE59489F1438BBC587A586042C8BD1@ukwdc-mail3.royalblue.com> Hi Max/DTrace list, > At any rate, without the -C, I can't use #include . > Without the #include , I can't use the M_DATA. > As it is, I get the following: > > # ./strrput.d 0xd595a6c0 <-- this is a vnode for a socket > ftp is using (not important here) > dtrace: failed to compile script ./strrput.d: line 7: > sd_vnode is not a member of struct stdata > # > > sd_vnode is definitely a member, so I tried using #define > _KERNEL in the script, and then I get an error: > dtrace: failed to compile script ./strrput.d: > "/usr/include/sys/kstat.h", line 439: invalid type combination Checking line 439 of that file... 423 typedef struct kstat_named { 424 char name[KSTAT_STRLEN]; /* name of counter */ 425 uchar_t data_type; /* data type */ 426 union { 427 char c[16]; /* enough for 128-bit ints */ 428 int32_t i32; 429 uint32_t ui32; 430 struct { 431 union { 432 char *ptr; /* NULL-term string */ 433 #if defined(_KERNEL) && defined(_MULTI_DATAMODEL) 434 caddr32_t ptr32; 435 #endif 436 char __pad[8]; /* 64-bit padding */ 437 } addr; 438 uint32_t len; /* # bytes for strlen + '\0' */ 439 } string; What seems to be happening is that DTrace is getting confused with this definition of a type called "string", and the internal D type "string". I can't take the credit for finding this though - a friend within Sun pointed out he's raised this already (thanks Jon!) as CR 6207207 (Synopsis: string definition in kstat.h conflicts with built in string type), and you can work around it with something like: #define string __C_string to change the typedef so it defines a different type. As you say, -D_KERNEL is necessary to pull in the definition of struct stdata - it isn't there otherwise, which I think is why you're seeing this: > dtrace: failed to compile script ./strrput.d: line 7: > sd_vnode is not a member of struct stdata Alternatively you can just include sys/strsubr.h before sys/stream.h, which has the same effect as -D_KERNEL in terms of ensuring struct stdata is defined; you then get the same problem with the duplicate string definition, though. But going back to the original problem: > At any rate, without the -C, I can't use #include . > Without the #include , I can't use the M_DATA. As the typing works magically without the include, maybe it's better just to hard-code the value of M_DATA, or if you want to continue to use the name, simply copy the definition, e.g. #!/usr/sbin/dtrace -Cs /* Copied from /usr/include/sys/stream.h */ #define M_DATA 0x00 strrput:entry ... etc etc ... -- Philip Beevers Fidessa Infrastructure Development mailto:philip.beevers at fidessa.com phone: +44 1483 206571 > -----Original Message----- > From: Max Bruning [mailto:max at bruningsystems.com] > Sent: 29 December 2004 01:42 > To: dtrace at opensolaris.org > Subject: [DTrace] using C preprocessor in dtrace scripts > > > Hi. > I am trying to use the -C option to include a file so > I can use #define's in the file. > > This is part of a larger script: > ------------------------------------------------------- > > #!/usr/sbin/dtrace -Cs > > #include > > strrput:entry > / > args[0]->q_stream->sd_vnode == (struct vnode *)$1 > && args[1]->b_datap->db_type == M_DATA > / > { > printf("strrput mp = %p\n", args[1]); > printf("mp->b_rptr = %*.*s\n", > args[1]->b_wptr-args[1]->b_rptr, > args[1]->b_wptr-args[1]->b_rptr, stringof(args[1]->b_rptr)); > } > > ----------------------------------------------------- > > At any rate, without the -C, I can't use #include . > Without the #include , I can't use the M_DATA. > As it is, I get the following: > > # ./strrput.d 0xd595a6c0 <-- this is a vnode for a socket > ftp is using (not important here) > dtrace: failed to compile script ./strrput.d: line 7: > sd_vnode is not a member of struct stdata > # > > sd_vnode is definitely a member, so I tried using #define > _KERNEL in the script, and then I get an error: > dtrace: failed to compile script ./strrput.d: > "/usr/include/sys/kstat.h", line 439: invalid type combination > > Anyone have any luck with using the preprocessor? > > thanks, > max > > > > > _______________________________________________ > DTrace mailing list > DTrace at opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtrace > From max at bruningsystems.com Mon Jan 3 09:12:19 2005 From: max at bruningsystems.com (Max Bruning) Date: Mon, 03 Jan 2005 09:12:19 -0800 Subject: [DTrace] using C preprocessor in dtrace scripts In-Reply-To: <8B5A5401A2DE59489F1438BBC587A586042C8BD1@ukwdc-mail3.royalblue.com> References: <8B5A5401A2DE59489F1438BBC587A586042C8BD1@ukwdc-mail3.royalblue.com> Message-ID: <1104772337.2322.8.camel@unknown> Thanks Philip. The easy fix (use #define M_DATA 0x00) I had done before I sent the original email. I don't mind work-arounds, I just wanted to make sure I wasn't missing anything. max On Sat, 2005-01-01 at 13:12, Philip Beevers wrote: > Hi Max/DTrace list, > > > At any rate, without the -C, I can't use #include . > > Without the #include , I can't use the M_DATA. > > As it is, I get the following: > > > > # ./strrput.d 0xd595a6c0 <-- this is a vnode for a socket > > ftp is using (not important here) > > dtrace: failed to compile script ./strrput.d: line 7: > > sd_vnode is not a member of struct stdata > > # > > > > sd_vnode is definitely a member, so I tried using #define > > _KERNEL in the script, and then I get an error: > > dtrace: failed to compile script ./strrput.d: > > "/usr/include/sys/kstat.h", line 439: invalid type combination > > Checking line 439 of that file... > > 423 typedef struct kstat_named { > > 424 char name[KSTAT_STRLEN]; /* name of counter */ > > 425 uchar_t data_type; /* data type */ > > 426 union { > > 427 char c[16]; /* enough for 128-bit ints */ > > 428 int32_t i32; > > 429 uint32_t ui32; > > 430 struct { > > 431 union { > > 432 char *ptr; /* NULL-term string */ > > 433 #if defined(_KERNEL) && defined(_MULTI_DATAMODEL) > > 434 caddr32_t ptr32; > > 435 #endif > > 436 char __pad[8]; /* 64-bit padding */ > > 437 } addr; > > 438 uint32_t len; /* # bytes for strlen + '\0' */ > > 439 } string; > > > What seems to be happening is that DTrace is getting confused with this > definition of a type called "string", and the internal D type "string". > > I can't take the credit for finding this though - a friend within Sun > pointed out he's raised this already (thanks Jon!) as CR 6207207 (Synopsis: > string definition in kstat.h conflicts with built in string type), and you > can work around it with something like: > > #define string __C_string > > to change the typedef so it defines a different type. > > As you say, -D_KERNEL is necessary to pull in the definition of struct > stdata - it isn't there otherwise, which I think is why you're seeing this: > > > dtrace: failed to compile script ./strrput.d: line 7: > > sd_vnode is not a member of struct stdata > > Alternatively you can just include sys/strsubr.h before sys/stream.h, which > has the same effect as -D_KERNEL in terms of ensuring struct stdata is > defined; you then get the same problem with the duplicate string definition, > though. But going back to the original problem: > > > At any rate, without the -C, I can't use #include . > > Without the #include , I can't use the M_DATA. > > As the typing works magically without the include, maybe it's better just to > hard-code the value of M_DATA, or if you want to continue to use the name, > simply copy the definition, e.g. > > #!/usr/sbin/dtrace -Cs > > /* Copied from /usr/include/sys/stream.h */ > #define M_DATA 0x00 > > strrput:entry > ... etc etc ... > > -- > > Philip Beevers > Fidessa Infrastructure Development > > mailto:philip.beevers at fidessa.com > phone: +44 1483 206571 > > > > -----Original Message----- > > From: Max Bruning [mailto:max at bruningsystems.com] > > Sent: 29 December 2004 01:42 > > To: dtrace at opensolaris.org > > Subject: [DTrace] using C preprocessor in dtrace scripts > > > > > > Hi. > > I am trying to use the -C option to include a file so > > I can use #define's in the file. > > > > This is part of a larger script: > > ------------------------------------------------------- > > > > #!/usr/sbin/dtrace -Cs > > > > #include > > > > strrput:entry > > / > > args[0]->q_stream->sd_vnode == (struct vnode *)$1 > > && args[1]->b_datap->db_type == M_DATA > > / > > { > > printf("strrput mp = %p\n", args[1]); > > printf("mp->b_rptr = %*.*s\n", > > args[1]->b_wptr-args[1]->b_rptr, > > args[1]->b_wptr-args[1]->b_rptr, stringof(args[1]->b_rptr)); > > } > > > > ----------------------------------------------------- > > > > At any rate, without the -C, I can't use #include . > > Without the #include , I can't use the M_DATA. > > As it is, I get the following: > > > > # ./strrput.d 0xd595a6c0 <-- this is a vnode for a socket > > ftp is using (not important here) > > dtrace: failed to compile script ./strrput.d: line 7: > > sd_vnode is not a member of struct stdata > > # > > > > sd_vnode is definitely a member, so I tried using #define > > _KERNEL in the script, and then I get an error: > > dtrace: failed to compile script ./strrput.d: > > "/usr/include/sys/kstat.h", line 439: invalid type combination > > > > Anyone have any luck with using the preprocessor? > > > > thanks, > > max > > > > > > > > > > _______________________________________________ > > DTrace mailing list > > DTrace at opensolaris.org > > https://www.opensolaris.org/mailman/listinfo/dtrace > > > _______________________________________________ > DTrace mailing list > DTrace at opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtrace > From philip.beevers at fidessa.com Tue Jan 4 13:01:02 2005 From: philip.beevers at fidessa.com (Philip Beevers) Date: Tue, 4 Jan 2005 21:01:02 -0000 Subject: [DTrace] Userland providers Message-ID: <8B5A5401A2DE59489F1438BBC587A586042C8BE7@ukwdc-mail3.royalblue.com> Hi everyone, I've been having a go at defining a userland provider, as it's one of the four things on Bryan's initial list. I've taken most of my cues from the plockstat provider Adam's put into libc, and the makefiles around it; I thought I'd document the process of defining a userland provider as I've been doing it, as it might be useful to someone else out there. And, of course, I've got a problem which I thought someone might be able to help with. OK, so here's how you define a userland DTrace provider: 1. Work out what your probes are, where they should be, and what their arguments are. Sounds obvious, but I suspect this will be pretty hard. Within the applications I work on, we have a whole host of statistics, and working out which statistics we should have and where they should be maintained is the toughest part. Anyway, I'm just testing, so I've skipped that step for now ;-) 2. Create a D script which defines your provider. Adam's example for the plockstat provider is in the source tree at /usr/src/lib/libc/port/threads/plockstat.d. My example, defining a single probe called bar from a provider called foo with a couple of integer arguments, is: provider foo { probe bar(int a, int b) : (int a, int b); }; #pragma D attributes Evolving/Evolving/ISA provider foo provider #pragma D attributes Private/Private/Unknown provider foo module #pragma D attributes Private/Private/Unknown provider foo function #pragma D attributes Evolving/Evolving/ISA provider foo name #pragma D attributes Evolving/Evolving/ISA provider foo args This appears to define the probes you're going to code up, and their stability attributes. 3. Write your code, using the preprocessor macros DTRACE_PROBE and DTRACE_PROBE1 to DTRACE_PROBE5 to define the points in the code at which the probes should fire. The first two args for all these macros are the provider and probe name - the DTRACE_PROBE1 - 5 probes allow you to specify 1 to 5 arguments to the probe. They're all defined in /usr/include/sys/sdt.h. So, to follow my example, the code looks like: DTRACE_PROBE2(foo, bar, 1, 2); 4. Compile your code into object code format normally - but don't link or build an archive just yet. [Note: you'll spot the above macros compile down to calls to magic, extern'd functions which are unique to each probe. These don't exist anywhere so your code won't link into an executable anyway] 5. Wave the DTrace magic wand. This step compiles an object code file from the D script you created at step 2, and fixes up the objects you compiled in step 4 by removing the extern'd function calls. I guess it must record the offsets in the object code of the probe points somewhere, too, so DTrace knows where to enable probes at runtime. Anyway, you do this by invoking dtrace(1M) as follows: dtrace -G -xlazyload -s -o So, let's say I put the D script from section 2 in a file b.d, and I've got a t.c which looks like this: #include #include #include int main(int argc, char** argv) { for (int i = 0; i < 10; i++) { printf("Hello world %d\n", i); DTRACE_PROBE2(foo, bar, 1, 2); } sleep(10); for (int i = 0; i < 10; i++) { printf("Hello world %d\n", i); DTRACE_PROBE2(foo, bar, 1, 2); } sleep(10); return (0); } Having compiled this to t.o, I'd then invoke: dtrace -G -xlazyload -s b.d -o b.o t.o This creates b.o and fixes up t.o. It's important to include all the object files here so all those function calls get fixed up - if you look at the makefiles in libc, you can see that Adam includes all the threads objects on the command line. 6. Build your archive/binary from the objects. So in my example: cc -o a t.o b.o ... and hey presto, you've got some probes you can enable: # dtrace -l -P foo\$target -c ./a ID PROVIDER MODULE FUNCTION NAME 29535 foo14420 a main bar But here's the problem - although DTrace knows the probes are there, they don't seem to fire! For example... # dtrace -P foo\$target -c ./a dtrace: description 'foo$target' matched 1 probe Hello world 0 Hello world 1 Hello world 2 Hello world 3 Hello world 4 Hello world 5 Hello world 6 Hello world 7 Hello world 8 Hello world 9 Hello world 0 Hello world 1 Hello world 2 Hello world 3 Hello world 4 Hello world 5 Hello world 6 Hello world 7 Hello world 8 Hello world 9 dtrace: pid 15369 has exited # Can anyone see where I'm going wrong? Anyone else had a go at this? -- Philip Beevers Fidessa Infrastructure Development mailto:philip.beevers at fidessa.com phone: +44 1483 206571 From mws at eng.sun.com Tue Jan 4 18:56:07 2005 From: mws at eng.sun.com (Michael Shapiro) Date: Tue, 4 Jan 2005 18:56:07 -0800 (PST) Subject: [DTrace] Userland providers In-Reply-To: <8B5A5401A2DE59489F1438BBC587A586042C8BE7@ukwdc-mail3.royalblue.com> from Philip Beevers at "Jan 4, 2005 09:01:02 pm" Message-ID: <200501050256.j052u7Ek107489@poptart.sfbay.sun.com> > Hi everyone, > > I've been having a go at defining a userland provider, as it's one of the > four things on Bryan's initial list. I've taken most of my cues from the > plockstat provider Adam's put into libc, and the makefiles around it; I > thought I'd document the process of defining a userland provider as I've > been doing it, as it might be useful to someone else out there. And, of > course, I've got a problem which I thought someone might be able to help > with. > > ... > > Having compiled this to t.o, I'd then invoke: > > dtrace -G -xlazyload -s b.d -o b.o t.o > > This creates b.o and fixes up t.o. It's important to include all the object > files here so all those function calls get fixed up - if you look at the > makefiles in libc, you can see that Adam includes all the threads objects on > the command line. > > 6. Build your archive/binary from the objects. So in my example: > > cc -o a t.o b.o > > > ... and hey presto, you've got some probes you can enable: > > # dtrace -l -P foo\$target -c ./a > ID PROVIDER MODULE FUNCTION NAME > 29535 foo14420 a main bar > > But here's the problem - although DTrace knows the probes are there, they > don't seem to fire! For example... > > # dtrace -P foo\$target -c ./a > dtrace: description 'foo$target' matched 1 probe > Hello world 0 > Hello world 1 > Hello world 2 > Hello world 3 > Hello world 4 > Hello world 5 > Hello world 6 > Hello world 7 > Hello world 8 > Hello world 9 > Hello world 0 > Hello world 1 > Hello world 2 > Hello world 3 > Hello world 4 > Hello world 5 > Hello world 6 > Hello world 7 > Hello world 8 > Hello world 9 > dtrace: pid 15369 has exited > > # > > Can anyone see where I'm going wrong? Anyone else had a go at this? Phil, This is the correct procedure, and this does work properly on the latest bits, unless I'm missing some aspect of your configuration. But Adam and I made a bunch of changes in this area in the late builds of S10, so perhaps there is a discrepancy there. What architecture and build of S10 are you on? Are you using the Sun compiler or gcc? Incidentally, for reference, the final version of the answerbook includes Chapter 34, "Statically Defined Tracing for User Applications", that describes the procedure (essentially the same as the part of the mail that I elided, except that typically you don't need to use -xlazyload). -Mike -- Mike Shapiro, Solaris Kernel Development. From philip.beevers at fidessa.com Tue Jan 4 23:32:35 2005 From: philip.beevers at fidessa.com (Philip Beevers) Date: Wed, 5 Jan 2005 07:32:35 -0000 Subject: [DTrace] Userland providers Message-ID: <8B5A5401A2DE59489F1438BBC587A586042C8BE9@ukwdc-mail3.royalblue.com> Hi Mike, Thanks for that. > This is the correct procedure, and this does work properly on > the latest bits, > unless I'm missing some aspect of your configuration. But > Adam and I made a > bunch of changes in this area in the late builds of S10, so > perhaps there is a > discrepancy there. What architecture and build of S10 are > you on? Are you > using the Sun compiler or gcc? My environment's not all that clean - it's build 71 on Opteron with the Sun compiler, but with a few tweaks here and there to solve some issues. I'll clean it up and see if things improve. > > Incidentally, for reference, the final version of the > answerbook includes > Chapter 34, "Statically Defined Tracing for User Applications", that > describes the procedure (essentially the same as the part of the mail > that I elided, except that typically you don't need to use > -xlazyload). Oh, that's embarrassing - really sorry about this, Mike, I should have looked on docs.sun.com first. Got to say, though, I'm fairly pleased the section in the doc wasn't all that different to my mail! -- Philip Beevers Fidessa Infrastructure Development mailto:philip.beevers at fidessa.com phone: +44 1483 206571 > -----Original Message----- > From: Michael Shapiro [mailto:mws at eng.sun.com] > Sent: 05 January 2005 02:56 > To: philip.beevers at fidessa.com > Cc: dtrace at opensolaris.org > Subject: Re: [DTrace] Userland providers > > > > Hi everyone, > > > > I've been having a go at defining a userland provider, as > it's one of the > > four things on Bryan's initial list. I've taken most of my > cues from the > > plockstat provider Adam's put into libc, and the makefiles > around it; I > > thought I'd document the process of defining a userland > provider as I've > > been doing it, as it might be useful to someone else out > there. And, of > > course, I've got a problem which I thought someone might be > able to help > > with. > > > > ... > > > > Having compiled this to t.o, I'd then invoke: > > > > dtrace -G -xlazyload -s b.d -o b.o t.o > > > > This creates b.o and fixes up t.o. It's important to > include all the object > > files here so all those function calls get fixed up - if > you look at the > > makefiles in libc, you can see that Adam includes all the > threads objects on > > the command line. > > > > 6. Build your archive/binary from the objects. So in my example: > > > > cc -o a t.o b.o > > > > > > ... and hey presto, you've got some probes you can enable: > > > > # dtrace -l -P foo\$target -c ./a > > ID PROVIDER MODULE > FUNCTION NAME > > 29535 foo14420 a > main bar > > > > But here's the problem - although DTrace knows the probes > are there, they > > don't seem to fire! For example... > > > > # dtrace -P foo\$target -c ./a > > dtrace: description 'foo$target' matched 1 probe > > Hello world 0 > > Hello world 1 > > Hello world 2 > > Hello world 3 > > Hello world 4 > > Hello world 5 > > Hello world 6 > > Hello world 7 > > Hello world 8 > > Hello world 9 > > Hello world 0 > > Hello world 1 > > Hello world 2 > > Hello world 3 > > Hello world 4 > > Hello world 5 > > Hello world 6 > > Hello world 7 > > Hello world 8 > > Hello world 9 > > dtrace: pid 15369 has exited > > > > # > > > > Can anyone see where I'm going wrong? Anyone else had a go at this? > > Phil, > > This is the correct procedure, and this does work properly on > the latest bits, > unless I'm missing some aspect of your configuration. But > Adam and I made a > bunch of changes in this area in the late builds of S10, so > perhaps there is a > discrepancy there. What architecture and build of S10 are > you on? Are you > using the Sun compiler or gcc? > > Incidentally, for reference, the final version of the > answerbook includes > Chapter 34, "Statically Defined Tracing for User Applications", that > describes the procedure (essentially the same as the part of the mail > that I elided, except that typically you don't need to use > -xlazyload). > > -Mike > > -- > Mike Shapiro, Solaris Kernel Development. > From brendan.gregg at tpg.com.au Wed Jan 5 09:26:18 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Thu, 6 Jan 2005 04:26:18 +1100 (EST) Subject: [DTrace] dtrace script request In-Reply-To: <200412212032.PAA25992@smc.vnet.net> Message-ID: G'Day Folks, I agree with Max, these are all good ideas. And although I don't know the best way ahead here are some further thoughts. The likelyhood of /usr/demo/dtrace/contrib appearing may rest on Sun's internal policy for adding things to Solaris, espically things written by external people (us). I've no idea - but I'd guess the chances are slim! DTrace does indeed need an easy to find "DTrace CookBook" website for extra example scripts, on BigAdmin or Sunfreeware. If we are contributing scripts to a single website, then any particularly useful script will probably find it's way into /usr/demo/dtrace eventually anyway (perhaps rewritten). Any DTrace enabled application (eg a monitoring GUI) probably belongs packaged and shipped on Sunfreeware. Any logical group of scripts (eg I/O diagnosis) would make a handy package for Sunfreeware too... ... PS. Phwoarrr, 70 mirror sites? That's probably spread on more webservers than Sun owns worldwide! Brendan On Tue, 21 Dec 2004, Steven M. Christensen wrote: > Yes, I am on this list and happy to offer space for dtrace > downloads. Everything I put on the site is mirrored to currently > 70 mirror sites. I presume this would be OK. > > Steve C. > > > > > Hosting is not an issue. I can do hosting if needed. Rather, I would > > like this > > to be in one easy to find place. Preferably a sun site or even better, > > with the system (/usr/demo/dtrace/contrib). Perhaps sunfreeware.com > > (is Steven Christensen on this mail list)??? > > > > I find it easiest to learn (and teach) dtrace by looking at and hacking > > existing scripts. When I need a detail, I can look it up in the 400+ page > > dtrace guide. > > > > max > > [...] From mws at eng.sun.com Wed Jan 5 11:05:17 2005 From: mws at eng.sun.com (Michael Shapiro) Date: Wed, 5 Jan 2005 11:05:17 -0800 (PST) Subject: [DTrace] Userland providers In-Reply-To: <8B5A5401A2DE59489F1438BBC587A586042C8BE9@ukwdc-mail3.royalblue.com> from Philip Beevers at "Jan 5, 2005 07:32:35 am" Message-ID: <200501051905.j05J5H3u108759@poptart.sfbay.sun.com> > Hi Mike, > > Thanks for that. > > > This is the correct procedure, and this does work properly on > > the latest bits, > > unless I'm missing some aspect of your configuration. But > > Adam and I made a > > bunch of changes in this area in the late builds of S10, so > > perhaps there is a > > discrepancy there. What architecture and build of S10 are > > you on? Are you > > using the Sun compiler or gcc? > > My environment's not all that clean - it's build 71 on Opteron with the Sun > compiler, but with a few tweaks here and there to solve some issues. I'll > clean it up and see if things improve. If you're trying to create and use userland SDT probes for a 64-bit Opteron program, you need to be using s10_73 or later. -Mike -- Mike Shapiro, Solaris Kernel Development. From brendan.gregg at tpg.com.au Thu Jan 6 02:58:52 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Thu, 6 Jan 2005 21:58:52 +1100 (EST) Subject: [DTrace] new iosnoop Message-ID: G'Day Folks, I've just rewritten iosnoop so that uses command line arguments (I caved in and used a little Bourne shell for this - hence it's lost it's ".d" extension). It's much more useful now. # iosnoop -h USAGE: iosnoop [-a|-A|-Degstv] [-d device] [-f filename] [-m mount_point] iosnoop # default output -a # print all data -A # dump all data, space delimited -D # include time delta, us -e # include device name -g # include command arguments -s # include start time, us -t # include completion time, us -v # include completion time, string -d device # instance name to snoop -f filename # snoop this file only -m mount_point # this FS only It also has it's very own website of examples, http://www.brendangregg.com/DTrace/iosnoop_example.html which is linked from the main one anyway. cheers, Brendan [Sydney, Australia] From brendan.gregg at tpg.com.au Mon Jan 17 02:29:27 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Mon, 17 Jan 2005 21:29:27 +1100 (EST) Subject: [DTrace] DTrace troubleshooting Message-ID: G'Day Folks, A few of us were discussing today why SMC took so long to boot the first time, so I pulled out DTrace and started looking. Pretty quickly we had a good idea why (eg, several million 1 byte read()s), and as it made for a nice demo of DTrace I just wrote a website for it. http://www.brendangregg.com/DTrace/smc.html I was careful not to suggest that these reads were actually a bug or a problem; I haven't read the source to SMC, I can only assume there is a good reason for this behaviour. Why I'm writing is if anyone thinks I should be more careful about what I say about SMC, please let me know. It's a good demo of DTrace, I wouldn't want people to misinterpret it as highlighting an SMC problem that may well not be a problem... thanks, Brendan [Sydney, Australia] From sommerfeld at sun.com Mon Jan 17 12:24:31 2005 From: sommerfeld at sun.com (Bill Sommerfeld) Date: Mon, 17 Jan 2005 15:24:31 -0500 Subject: [DTrace] DTrace troubleshooting In-Reply-To: References: Message-ID: <1105993470.2130.50.camel@unknown.hamachi.org> On Mon, 2005-01-17 at 05:29, Brendan Gregg wrote: > I was careful not to suggest that these reads were actually a bug or a > problem; I haven't read the source to SMC, I can only assume there is a > good reason for this behaviour. since lseek() isn't turning up in your profile, i have to assume that these are sequential 1-byte reads. sequential 1-byte reads on plain files are.. evidence of lack of tuning. - Bill From dp at eng.sun.com Mon Jan 17 18:03:26 2005 From: dp at eng.sun.com (Dan Price) Date: Mon, 17 Jan 2005 18:03:26 -0800 Subject: [DTrace] DTrace troubleshooting In-Reply-To: References: Message-ID: <20050118020326.GH1258@eng.sun.com> On Mon 17 Jan 2005 at 09:29PM, Brendan Gregg wrote: > G'Day Folks, > > A few of us were discussing today why SMC took so long to boot the first > time, so I pulled out DTrace and started looking. Pretty quickly we had a > good idea why (eg, several million 1 byte read()s), and as it made for a > nice demo of DTrace I just wrote a website for it. > > http://www.brendangregg.com/DTrace/smc.html > > I was careful not to suggest that these reads were actually a bug or a > problem; I haven't read the source to SMC, I can only assume there is a > good reason for this behaviour. > > Why I'm writing is if anyone thinks I should be more careful about what I > say about SMC, please let me know. It's a good demo of DTrace, I wouldn't > want people to misinterpret it as highlighting an SMC problem that may > well not be a problem... Brendan, I'd call it a (performance) bug, and you should feel free to do so, as far as I'm concerned. You're very generous to "assume there is a good reason". Remember that a lot of this code was written before we had DTrace; and also, it could be an interaction bug which is new. It would really help if you could enhance your analysis to show how much time these reads are actually chewing up (ex. aggregate on syscall by vtime spent in the syscall)-- that would give us some understanding of how much we're paying. Could you also please use the jstack() action to figure out the java stack trace which is inducing these reads? That will strengthen the analysis as well. I did a little digging in the admininstall source base, and found that registry.ser is a file which contains a set of serialized java objects. I filed 6218290 smc seems to pound registry.ser on startup, wasting cycles On your behalf, and basically just pointed at your webpage (so if you could leave it up, that would be helpful!) -dp -- Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp From brendan.gregg at tpg.com.au Tue Jan 18 07:40:06 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Wed, 19 Jan 2005 02:40:06 +1100 (EST) Subject: [DTrace] DTrace troubleshooting In-Reply-To: <1105993470.2130.50.camel@unknown.hamachi.org> Message-ID: G'Day Bill, On Mon, 17 Jan 2005, Bill Sommerfeld wrote: > On Mon, 2005-01-17 at 05:29, Brendan Gregg wrote: > > > I was careful not to suggest that these reads were actually a bug or a > > problem; I haven't read the source to SMC, I can only assume there is a > > good reason for this behaviour. > > since lseek() isn't turning up in your profile, i have to assume that these > are sequential 1-byte reads. sequential 1-byte reads on plain files are.. > evidence of lack of tuning. Surely 7 million 1 byte sequential reads can't be wrong? ;-) Brendan From brendan.gregg at tpg.com.au Tue Jan 18 07:58:13 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Wed, 19 Jan 2005 02:58:13 +1100 (EST) Subject: [DTrace] DTrace troubleshooting In-Reply-To: <20050118020326.GH1258@eng.sun.com> Message-ID: G'Day Dan, On Mon, 17 Jan 2005, Dan Price wrote: > On Mon 17 Jan 2005 at 09:29PM, Brendan Gregg wrote: > > G'Day Folks, > > > > A few of us were discussing today why SMC took so long to boot the first > > time, so I pulled out DTrace and started looking. Pretty quickly we had a > > good idea why (eg, several million 1 byte read()s), and as it made for a > > nice demo of DTrace I just wrote a website for it. > > > > http://www.brendangregg.com/DTrace/smc.html > > > > I was careful not to suggest that these reads were actually a bug or a > > problem; I haven't read the source to SMC, I can only assume there is a > > good reason for this behaviour. [...] > Brendan, > > I'd call it a (performance) bug, and you should feel free to do so, as > far as I'm concerned. You're very generous to "assume there is a good > reason". Remember that a lot of this code was written before we had > DTrace; and also, it could be an interaction bug which is new. Well, it's a public website so I ought to assume the best. Anyone who knows performance will make up their own mind anyway. I imagine DTrace will indeed shine light on many new things. (I ought to find a database server to pick on instead). > It would really help if you could enhance your analysis to show how > much time these reads are actually chewing up (ex. aggregate on syscall > by vtime spent in the syscall)-- that would give us some understanding > of how much we're paying. I ran into an ugly stumbling block before realising what you meant by vtime. I'm now using vtimestamps to measure syscall time on CPUs - it really helps! This is now done and on the site. Those reads are using 78% of the total syscall time (or 94 seconds). > Could you also please use the jstack() action to figure out the java > stack trace which is inducing these reads? That will strengthen the > analysis as well. Good idea, done. > I did a little digging in the admininstall source base, and found > that registry.ser is a file which contains a set of serialized java objects. > I filed > > 6218290 smc seems to pound registry.ser on startup, wasting cycles > > On your behalf, and basically just pointed at your webpage (so if you > could leave it up, that would be helpful!) No worries, Thanks Dan! :) Brendan From jamesd.wi at gmail.com Thu Jan 20 11:49:59 2005 From: jamesd.wi at gmail.com (James Dickens) Date: Thu, 20 Jan 2005 13:49:59 -0600 Subject: [DTrace] how to get socket read size Message-ID: Hi i'm trying to write my first dtrace script apparently i bit off a bit more than i can chew, i want to track io over sockets, i found your socketsize.d that gave me how to track writes, but i'm at a loss how to track reads, frankly i don't see how your write tracker works because it uses a probe in a function that only takes two arguments but you grab size of write via arg3? i would appreciate any hints on how to accomplish this task, and an explanation of how the args work if i'm missing something Thanks James Dickens information relied on: from socketsize.d /* ** Process Socket Write */ fbt:sockfs:so_update_attrs:entry /arg1 == 1/ { /* fetch details */ this->size = arg3; cmd = (string)curpsinfo->pr_psargs; /* store details */ @Size[pid,cmd] = quantize(this->size); } from src/src/uts/common/fs/sockfs/socksubr.c void so_update_attrs(struct sonode *so, int flag) From max at bruningsystems.com Thu Jan 20 13:18:08 2005 From: max at bruningsystems.com (Max Bruning) Date: Thu, 20 Jan 2005 13:18:08 -0800 Subject: [DTrace] how to get socket read size In-Reply-To: References: Message-ID: <1106255888.5852.76.camel@maxpc> Hi. It looks to me like socketsnoop.d (also on Brendan's page) will do most of what you want. Specifically, the socktpi_read trace. If you want to track the data itself, you'll need to trace calls to strread and kstrgetmsg. I have a dtrace script that tracks strrput calls and prints out the message data, but it is cumbersome to use (you need to give it a vnode address for the stream head you want to snoop). strrput is called by the downstream module (tcp/ip in the case of sockets) to enqueue messages at the stream head. strread or kstrgetmsg is used to retrieve enqueued messages. Here is a dtrace script that traces (and prints) M_DATA messages arriving on STREAMS heads. You give it a vnode address as an argument. Try to use this with data you expect to be ascii. Currently, there is no good way to print binary data in dtrace. -------------------CUT HERE---------------------------- #!/usr/sbin/dtrace -qs #pragma D option strsize=8192 BEGIN { cvp = (struct vnode *)$1; /* you tell dtrace which vnode */ } strrput:entry / args[0]->q_stream->sd_vnode == cvp && args[1]->b_datap->db_type == 0x00 / /* only M_DATA messages */ { self->mp = args[1]; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strrput:entry / args[1]->b_datap->db_type == 0x00 && self->mp != NULL / { printf("%*.*s", args[1]->b_wptr-args[1]->b_rptr, args[1]->b_wptr-args[1]->b_rptr, (string)args[1]->b_rptr); self->mp = self->mp->b_cont; } strclose:entry / args[0] == cvp / { exit(0); } ------------------CUT HERE------------------- To use this, you need 2 windows. In one window, start the program you want to trace. For instance, telnet: # telnet 192.168.2.102 (or whatever host...) ... In the other window (on local machine): # pfiles `pgrep telnet` <-- get fd for the socket you want to snoop 6444: /usr/bin/telnet Current rlimit: 256 file descriptors 0: S_IFCHR mode:0620 dev:270,0 ino:12582942 uid:100 gid:7 rdev:24,13 O_RDWR|O_NDELAY /devices/pseudo/pts at 0:13 1: S_IFCHR mode:0620 dev:270,0 ino:12582942 uid:100 gid:7 rdev:24,13 O_RDWR|O_NDELAY /devices/pseudo/pts at 0:13 2: S_IFCHR mode:0620 dev:270,0 ino:12582942 uid:100 gid:7 rdev:24,13 O_RDWR|O_NDELAY /devices/pseudo/pts at 0:13 3: S_IFSOCK mode:0666 dev:276,0 ino:32025 uid:0 gid:0 size:0 O_RDWR|O_NDELAY SOCK_STREAM SO_OOBINLINE,SO_SNDBUF(49152),SO_RCVBUF(49640) sockname: AF_INET 192.168.2.100 port: 40037 peername: AF_INET 192.168.2.102 port: 23 # mdb -k mdb: no terminal data available for TERM=emacs mdb: term init failed: command-line editing and prompt will not be available ::ps!grep telnet R 6444 6031 6444 6444 100 0x42004000 d352aa40 telnet d352aa40::walk file <-- walk open files for this proc d5597018 d5597018 d5597018 d4fbd508 <-- from pfiles output, this is the file we want d4fbd508::print file_t f_vnode f_vnode = 0xd6a4ff00 <-- and here's the vnode for the socket $q # ./strrput3.d 0xd6a4ff00 <-- now run the script Everything displayed on the screen in the telnet window should show up in the dtrace output. max On Thu, 2005-01-20 at 11:49, James Dickens wrote: > Hi > > i'm trying to write my first dtrace script apparently i bit off a bit > more than i can chew, i want to track io over sockets, i found your > socketsize.d that gave me how to track writes, but i'm at a loss how > to track reads, frankly i don't see how your write tracker works > because it uses a probe in a function that only takes two arguments > but you grab size of write via arg3? > > i would appreciate any hints on how to accomplish this task, and an > explanation of how the args work if i'm missing something > > > Thanks > > James Dickens > > > information relied on: > > from socketsize.d > /* > ** Process Socket Write > */ > fbt:sockfs:so_update_attrs:entry > /arg1 == 1/ > { > /* fetch details */ > this->size = arg3; > cmd = (string)curpsinfo->pr_psargs; > > /* store details */ > @Size[pid,cmd] = quantize(this->size); > } > > from src/src/uts/common/fs/sockfs/socksubr.c > > void > so_update_attrs(struct sonode *so, int flag) > _______________________________________________ > DTrace mailing list > DTrace at opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtrace > From brendan.gregg at tpg.com.au Thu Jan 20 15:35:56 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Fri, 21 Jan 2005 10:35:56 +1100 (EST) Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: G'Day James, On Thu, 20 Jan 2005, James Dickens wrote: > i'm trying to write my first dtrace script apparently i bit off a bit > more than i can chew, i want to track io over sockets, i found your I've been working on this for some time (networking by process) - so far I can only do it by turning on far too many probes, I'm trying to nail an elegant solution (almost there). I wouldn't pick a networking topic as your first DTrace script!! > socketsize.d that gave me how to track writes, but i'm at a loss how > to track reads, frankly i don't see how your write tracker works > because it uses a probe in a function that only takes two arguments > but you grab size of write via arg3? Yes, I'd like to know how I used arg3 as well. I often write scripts like this, #pragma D option flowindent fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } fbt::: /self->ok/ { printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", execname,probeprov,probemod,probefunc,probename, arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); } So I watch the 8 args from all functionss and spot things of interest, then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I spotted arg3 had the byte count. Great. Later I read the prototype, socketvar.h:extern void so_update_attrs(struct sonode *, int); Hmm. Not sure why DTrace is pulling meaningful information for args that shouldn't exist - but it was working reliably so I've left that mystery to investigate at a later date. At some point I'll get stuck into the Solaris source and may find extra prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that already!) Or perhaps DTrace is reading memory addresses from a previous function that it didn't clear. Whatever the reason, I imagine in the long term this will be done by a network provider and not fbt, so these scripts are rather temporary. no worries, Brendan > James Dickens [...] > > from src/src/uts/common/fs/sockfs/socksubr.c > > void > so_update_attrs(struct sonode *so, int flag) > > > > From max at bruningsystems.com Thu Jan 20 16:39:22 2005 From: max at bruningsystems.com (Max Bruning) Date: Thu, 20 Jan 2005 16:39:22 -0800 Subject: [DTrace] Re: how to get socket read size In-Reply-To: References: Message-ID: <1106267962.5852.82.camel@maxpc> Maybe arg3 is left over from a previously called function. Have you tried the script on both sparc and x86 to see if it works? max > > Yes, I'd like to know how I used arg3 as well. > > I often write scripts like this, > > #pragma D option flowindent > fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } > fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } > fbt::: > /self->ok/ > { > printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", > execname,probeprov,probemod,probefunc,probename, > arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); > } > > So I watch the 8 args from all functionss and spot things of interest, > then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I > spotted arg3 had the byte count. Great. Later I read the prototype, > > socketvar.h:extern void so_update_attrs(struct sonode *, int); > > Hmm. Not sure why DTrace is pulling meaningful information for args that > shouldn't exist - but it was working reliably so I've left that mystery to > investigate at a later date. > > At some point I'll get stuck into the Solaris source and may find extra > prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that > already!) Or perhaps DTrace is reading memory addresses from a previous > function that it didn't clear. > > Whatever the reason, I imagine in the long term this will be done by a > network provider and not fbt, so these scripts are rather temporary. > > no worries, > > Brendan > > > > James Dickens > [...] > > > > from src/src/uts/common/fs/sockfs/socksubr.c > > > > void > > so_update_attrs(struct sonode *so, int flag) > > > > > > > > > > _______________________________________________ > DTrace mailing list > DTrace at opensolaris.org > https://www.opensolaris.org/mailman/listinfo/dtrace > From jamesd.wi at gmail.com Thu Jan 20 20:04:40 2005 From: jamesd.wi at gmail.com (James Dickens) Date: Thu, 20 Jan 2005 22:04:40 -0600 Subject: [DTrace] Re: how to get socket read size In-Reply-To: <1106267962.5852.82.camel@maxpc> References: <1106267962.5852.82.camel@maxpc> Message-ID: Hi Okay i solved the problem it wasn't as easy as it looks, just damm glad i had the source, and the help of a couple solaris kernel team members, thanks guys.. there names are in the file... James Dickens see atached file for how it was done On Thu, 20 Jan 2005 16:39:22 -0800, Max Bruning wrote: > > Maybe arg3 is left over from a previously called function. > Have you tried the script on both sparc and x86 to see > if it works? > > max > > > > > Yes, I'd like to know how I used arg3 as well. > > > > I often write scripts like this, > > > > #pragma D option flowindent > > fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } > > fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } > > fbt::: > > /self->ok/ > > { > > printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", > > execname,probeprov,probemod,probefunc,probename, > > arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); > > } > > > > So I watch the 8 args from all functionss and spot things of interest, > > then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I > > spotted arg3 had the byte count. Great. Later I read the prototype, > > > > socketvar.h:extern void so_update_attrs(struct sonode *, int); > > > > Hmm. Not sure why DTrace is pulling meaningful information for args that > > shouldn't exist - but it was working reliably so I've left that mystery to > > investigate at a later date. > > > > At some point I'll get stuck into the Solaris source and may find extra > > prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that > > already!) Or perhaps DTrace is reading memory addresses from a previous > > function that it didn't clear. > > > > Whatever the reason, I imagine in the long term this will be done by a > > network provider and not fbt, so these scripts are rather temporary. > > > > no worries, > > > > Brendan > > > > > > > James Dickens > > [...] > > > > > > from src/src/uts/common/fs/sockfs/socksubr.c > > > > > > void > > > so_update_attrs(struct sonode *so, int flag) > > > > > > > > > > > > > > > > _______________________________________________ > > DTrace mailing list > > DTrace at opensolaris.org > > https://www.opensolaris.org/mailman/listinfo/dtrace > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: socket_io Type: application/octet-stream Size: 949 bytes Desc: not available URL: From brendan.gregg at tpg.com.au Thu Jan 20 20:38:44 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Fri, 21 Jan 2005 15:38:44 +1100 (EST) Subject: [DTrace] Re: how to get socket read size In-Reply-To: <1106267962.5852.82.camel@maxpc> Message-ID: G'Day Max, Ok, that confirms that it is unreliable - I just tried it on sparc and it dosen't work. Hmm. Either I forgot to check it on sparc before, or something changed in the builds. (I've always been suspicious about those socket*.d scripts, hence a version number of 0.50 - low confidence). I just rewrote it using the mib provider. It works fine on sparc now. http://www.brendangregg.com/DTrace/socketsize.d Brendan On Thu, 20 Jan 2005, Max Bruning wrote: > > Maybe arg3 is left over from a previously called function. > Have you tried the script on both sparc and x86 to see > if it works? > > max > From max at garlic.com Thu Jan 20 21:10:37 2005 From: max at garlic.com (Max Bruning) Date: Thu, 20 Jan 2005 21:10:37 -0800 Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: I've had a similar problem tracing segmap_getmapflt(). The 2nd (or 3rd) arg is the size. On Sparc, I use arg[2] and arg[3] for the next argument. On x86, I find I need to use (long long)arg[2] and arg[4] for the next argument. (When I am finished with the script, I'll post it...). max On Thursday, January 20, 2005, at 08:38 PM, Brendan Gregg wrote: > G'Day Max, > > Ok, that confirms that it is unreliable - I just tried it on sparc and > it > dosen't work. Hmm. Either I forgot to check it on sparc before, or > something changed in the builds. (I've always been suspicious about > those > socket*.d scripts, hence a version number of 0.50 - low confidence). > > > I just rewrote it using the mib provider. It works fine on sparc now. > > http://www.brendangregg.com/DTrace/socketsize.d > > Brendan > > > > On Thu, 20 Jan 2005, Max Bruning wrote: > >> >> Maybe arg3 is left over from a previously called function. >> Have you tried the script on both sparc and x86 to see >> if it works? >> >> max >> > > > > > From max at garlic.com Thu Jan 20 21:21:51 2005 From: max at garlic.com (Max Bruning) Date: Thu, 20 Jan 2005 21:21:51 -0800 Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: <5AF020DA-6B6C-11D9-BD84-000A95721C40@garlic.com> Hi, OK. So additional args to entry probes are documented by... reading the source!!!? and asking some kernel guys??? OK. I can do that... I just looked at the docs, and I don't see it documented... Also, I don't see any names in the file. max On Thursday, January 20, 2005, at 08:04 PM, James Dickens wrote: > Hi > > Okay i solved the problem it wasn't as easy as it looks, just damm > glad i had the source, and the help of a couple solaris kernel team > members, thanks guys.. there names are in the file... > > > James Dickens > > see atached file for how it was done > > > > > On Thu, 20 Jan 2005 16:39:22 -0800, Max Bruning > wrote: >> >> Maybe arg3 is left over from a previously called function. >> Have you tried the script on both sparc and x86 to see >> if it works? >> >> max >> >>> >>> Yes, I'd like to know how I used arg3 as well. >>> >>> I often write scripts like this, >>> >>> #pragma D option flowindent >>> fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } >>> fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } >>> fbt::: >>> /self->ok/ >>> { >>> printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", >>> execname,probeprov,probemod,probefunc,probename, >>> arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); >>> } >>> >>> So I watch the 8 args from all functionss and spot things of >>> interest, >>> then match versus the prototype. With >>> fbt:sockfs:so_update_attrs:entry I >>> spotted arg3 had the byte count. Great. Later I read the prototype, >>> >>> socketvar.h:extern void so_update_attrs(struct sonode *, int); >>> >>> Hmm. Not sure why DTrace is pulling meaningful information for args >>> that >>> shouldn't exist - but it was working reliably so I've left that >>> mystery to >>> investigate at a later date. >>> >>> At some point I'll get stuck into the Solaris source and may find >>> extra >>> prototypes that aren't in /usr/include/sys. (Oh! looks like you >>> tried that >>> already!) Or perhaps DTrace is reading memory addresses from a >>> previous >>> function that it didn't clear. >>> >>> Whatever the reason, I imagine in the long term this will be done by >>> a >>> network provider and not fbt, so these scripts are rather temporary. >>> >>> no worries, >>> >>> Brendan >>> >>> >>>> James Dickens >>> [...] >>>> >>>> from src/src/uts/common/fs/sockfs/socksubr.c >>>> >>>> void >>>> so_update_attrs(struct sonode *so, int flag) >>>> >>>> >>>> >>>> >>> >>> _______________________________________________ >>> DTrace mailing list >>> DTrace at opensolaris.org >>> https://www.opensolaris.org/mailman/listinfo/dtrace >>> >> > From jamesd.wi at gmail.com Thu Jan 20 21:22:33 2005 From: jamesd.wi at gmail.com (James Dickens) Date: Thu, 20 Jan 2005 23:22:33 -0600 Subject: [DTrace] Re: how to get socket read size In-Reply-To: References: <1106267962.5852.82.camel@maxpc> Message-ID: sorry bone head mistake.... wrong file got attached this one works On Thu, 20 Jan 2005 22:04:40 -0600, James Dickens wrote: > Hi > > Okay i solved the problem it wasn't as easy as it looks, just damm > glad i had the source, and the help of a couple solaris kernel team > members, thanks guys.. there names are in the file... > > James Dickens > > see atached file for how it was done > > > On Thu, 20 Jan 2005 16:39:22 -0800, Max Bruning wrote: > > > > Maybe arg3 is left over from a previously called function. > > Have you tried the script on both sparc and x86 to see > > if it works? > > > > max > > > > > > > > Yes, I'd like to know how I used arg3 as well. > > > > > > I often write scripts like this, > > > > > > #pragma D option flowindent > > > fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } > > > fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } > > > fbt::: > > > /self->ok/ > > > { > > > printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", > > > execname,probeprov,probemod,probefunc,probename, > > > arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); > > > } > > > > > > So I watch the 8 args from all functionss and spot things of interest, > > > then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I > > > spotted arg3 had the byte count. Great. Later I read the prototype, > > > > > > socketvar.h:extern void so_update_attrs(struct sonode *, int); > > > > > > Hmm. Not sure why DTrace is pulling meaningful information for args that > > > shouldn't exist - but it was working reliably so I've left that mystery to > > > investigate at a later date. > > > > > > At some point I'll get stuck into the Solaris source and may find extra > > > prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that > > > already!) Or perhaps DTrace is reading memory addresses from a previous > > > function that it didn't clear. > > > > > > Whatever the reason, I imagine in the long term this will be done by a > > > network provider and not fbt, so these scripts are rather temporary. > > > > > > no worries, > > > > > > Brendan > > > > > > > > > > James Dickens > > > [...] > > > > > > > > from src/src/uts/common/fs/sockfs/socksubr.c > > > > > > > > void > > > > so_update_attrs(struct sonode *so, int flag) > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > DTrace mailing list > > > DTrace at opensolaris.org > > > https://www.opensolaris.org/mailman/listinfo/dtrace > > > > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: socket_io.d Type: application/octet-stream Size: 1400 bytes Desc: not available URL: From max at garlic.com Thu Jan 20 21:24:02 2005 From: max at garlic.com (Max Bruning) Date: Thu, 20 Jan 2005 21:24:02 -0800 Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: Probably the mib or io provider is the right thing to use for most of the io based tracing. Its just more interesting to trace the calls that are doing the work;-) max On Thursday, January 20, 2005, at 09:10 PM, Max Bruning wrote: > > I've had a similar problem tracing segmap_getmapflt(). > The 2nd (or 3rd) arg is the size. On Sparc, I use arg[2] and arg[3] > for the next argument. On x86, I find I need to use (long long)arg[2] > and arg[4] for the next argument. (When I am finished with the script, > I'll post it...). > > max > > On Thursday, January 20, 2005, at 08:38 PM, Brendan Gregg wrote: > >> G'Day Max, >> >> Ok, that confirms that it is unreliable - I just tried it on sparc >> and it >> dosen't work. Hmm. Either I forgot to check it on sparc before, or >> something changed in the builds. (I've always been suspicious about >> those >> socket*.d scripts, hence a version number of 0.50 - low confidence). >> >> >> I just rewrote it using the mib provider. It works fine on sparc now. >> >> http://www.brendangregg.com/DTrace/socketsize.d >> >> Brendan >> >> >> >> On Thu, 20 Jan 2005, Max Bruning wrote: >> >>> >>> Maybe arg3 is left over from a previously called function. >>> Have you tried the script on both sparc and x86 to see >>> if it works? >>> >>> max >>> >> >> >> >> >> > > From jamesd.wi at gmail.com Thu Jan 20 21:31:03 2005 From: jamesd.wi at gmail.com (James Dickens) Date: Thu, 20 Jan 2005 23:31:03 -0600 Subject: [DTrace] Re: how to get socket read size In-Reply-To: References: <1106267962.5852.82.camel@maxpc> Message-ID: okay finnal fix of the night.... i'm tired... fixed a typo,. a cleaned up memory... like a good little programmer good night James On Thu, 20 Jan 2005 23:22:33 -0600, James Dickens wrote: > sorry bone head mistake.... wrong file got attached this one works > > > On Thu, 20 Jan 2005 22:04:40 -0600, James Dickens wrote: > > Hi > > > > Okay i solved the problem it wasn't as easy as it looks, just damm > > glad i had the source, and the help of a couple solaris kernel team > > members, thanks guys.. there names are in the file... > > > > James Dickens > > > > see atached file for how it was done > > > > > > On Thu, 20 Jan 2005 16:39:22 -0800, Max Bruning wrote: > > > > > > Maybe arg3 is left over from a previously called function. > > > Have you tried the script on both sparc and x86 to see > > > if it works? > > > > > > max > > > > > > > > > > > Yes, I'd like to know how I used arg3 as well. > > > > > > > > I often write scripts like this, > > > > > > > > #pragma D option flowindent > > > > fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } > > > > fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } > > > > fbt::: > > > > /self->ok/ > > > > { > > > > printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", > > > > execname,probeprov,probemod,probefunc,probename, > > > > arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); > > > > } > > > > > > > > So I watch the 8 args from all functionss and spot things of interest, > > > > then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I > > > > spotted arg3 had the byte count. Great. Later I read the prototype, > > > > > > > > socketvar.h:extern void so_update_attrs(struct sonode *, int); > > > > > > > > Hmm. Not sure why DTrace is pulling meaningful information for args that > > > > shouldn't exist - but it was working reliably so I've left that mystery to > > > > investigate at a later date. > > > > > > > > At some point I'll get stuck into the Solaris source and may find extra > > > > prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that > > > > already!) Or perhaps DTrace is reading memory addresses from a previous > > > > function that it didn't clear. > > > > > > > > Whatever the reason, I imagine in the long term this will be done by a > > > > network provider and not fbt, so these scripts are rather temporary. > > > > > > > > no worries, > > > > > > > > Brendan > > > > > > > > > > > > > James Dickens > > > > [...] > > > > > > > > > > from src/src/uts/common/fs/sockfs/socksubr.c > > > > > > > > > > void > > > > > so_update_attrs(struct sonode *so, int flag) > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > DTrace mailing list > > > > DTrace at opensolaris.org > > > > https://www.opensolaris.org/mailman/listinfo/dtrace > > > > > > > > > > > > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: socket_io.d Type: application/octet-stream Size: 1402 bytes Desc: not available URL: From brendan.gregg at tpg.com.au Thu Jan 20 22:21:34 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Fri, 21 Jan 2005 17:21:34 +1100 (EST) Subject: [DTrace] Re: how to get socket read size In-Reply-To: <5AF020DA-6B6C-11D9-BD84-000A95721C40@garlic.com> Message-ID: G'Day, On Thu, 20 Jan 2005, Max Bruning wrote: > Hi, > OK. So additional args to entry probes are documented by... > reading the source!!!? and asking some kernel guys??? ... So far I haven't read the source as I'm writing DTrace scripts for a public website, and I don't want to publish info I ought not to. When OpenSolaris really is open life will be a little easier. :) > OK. I can do that... > I just looked at the docs, and I don't see it documented... On the public forum you can read the "DTrace watches ssh keystrokes" thread, where I explain determining the arguments by reading hex dumps for several hours. Jonathan Adams did a follow up post and explained how to do it in mdb. In one line! So... you can read hex dumps for several hours, or, you can use "funcion_name::nm -f ctype" from mdb. # mdb -k Loading modules: [ unix krtld genunix specfs dtrace ufs ip sctp usba fctl nca nfs audiosup random sppp sd crypto ptm ipc logindmux ] > socktpi_read::nm -f ctype C Type int (*)(struct vnode *, struct uio *, int, struct cred *, struct caller_context *) not to worry, I know more of mdb's secrets now ;) Brendan From brendan.gregg at tpg.com.au Thu Jan 20 22:29:12 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Fri, 21 Jan 2005 17:29:12 +1100 (EST) Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: On Thu, 20 Jan 2005, Max Bruning wrote: > Probably the mib or io provider is the right thing to use for most of > the io based tracing. Its just more interesting to trace the calls > that are > doing the work;-) I like your attitude. It's even more interesting to read probe hex dumps. honest! ;) Brendan From brendan.gregg at tpg.com.au Thu Jan 20 23:04:42 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Fri, 21 Jan 2005 18:04:42 +1100 (EST) Subject: [DTrace] Re: how to get socket read size In-Reply-To: Message-ID: G'Day James, Walking through those structs look quite nicely done. :) I'm a little cautious about this though, fbt:sockfs:socktpi_read:entry /uid == 1000 / (I assume 1000 is the UID for the user you'd like to trace). This probe will only fire if the proc on the CPU has a UID of 1000, ie the process you are interested in tracing. So, the process that is doing a socket read needs to be on the CPU during this socktpi_read:entry call. Here I trace the execname for the same call, while I send telnet packets, # dtrace -n 'fbt:sockfs:socktpi_read:entry { trace(execname); }' dtrace: description 'fbt:sockfs:socktpi_read:entry ' matched 1 probe CPU ID FUNCTION:NAME 0 19191 socktpi_read:entry Xsun 0 19191 socktpi_read:entry Xsun 0 19191 socktpi_read:entry Xsun 0 19191 socktpi_read:entry Xsun 0 19191 socktpi_read:entry Xsun 0 19191 socktpi_read:entry Xsun [...] However telnet is not caught. What I think is happening is that the packet arrives asynchronously, so we can't guarantee that the correct process is on the CPU (here it is incorrectly identifying Xsun). Sometimes it does work, sometimes not, depending on scheduling. (seems ssh is picked up a lot more correctly than telnet). Either that or inetd/in.telnetd does something different strange after connection to avoid using socktpi_read:entry.... It's the asynchronous problem that's had me working on this for some time now. I'm writing scripts to associate the asynchronous read with the proc completing the syscall read.. Any insight on what happens is appreciated! I'm slowly figuring this out with loads of DTrace. cheers, Brendan On Thu, 20 Jan 2005, James Dickens wrote: > okay finnal fix of the night.... i'm tired... > > fixed a typo,. > a cleaned up memory... like a good little programmer > > > good night > > James > > > > On Thu, 20 Jan 2005 23:22:33 -0600, James Dickens wrote: > > sorry bone head mistake.... wrong file got attached this one works > > > > > > On Thu, 20 Jan 2005 22:04:40 -0600, James Dickens wrote: > > > Hi > > > > > > Okay i solved the problem it wasn't as easy as it looks, just damm > > > glad i had the source, and the help of a couple solaris kernel team > > > members, thanks guys.. there names are in the file... > > > > > > James Dickens > > > > > > see atached file for how it was done > > > > > > > > > On Thu, 20 Jan 2005 16:39:22 -0800, Max Bruning wrote: > > > > > > > > Maybe arg3 is left over from a previously called function. > > > > Have you tried the script on both sparc and x86 to see > > > > if it works? > > > > > > > > max > > > > > > > > > > > > > > Yes, I'd like to know how I used arg3 as well. > > > > > > > > > > I often write scripts like this, > > > > > > > > > > #pragma D option flowindent > > > > > fbt:sockfs:sotpi_recvmsg:entry { self->ok=1; } > > > > > fbt:sockfs:sotpi_recvmsg:return { self->ok=0; } > > > > > fbt::: > > > > > /self->ok/ > > > > > { > > > > > printf("%12s %s:%s:%s:%s %5x %5x %5x %5x %5x %5x %5x %5x\n", > > > > > execname,probeprov,probemod,probefunc,probename, > > > > > arg0,arg1,arg2,arg3,arg4,arg5,arg6,arg7); > > > > > } > > > > > > > > > > So I watch the 8 args from all functionss and spot things of interest, > > > > > then match versus the prototype. With fbt:sockfs:so_update_attrs:entry I > > > > > spotted arg3 had the byte count. Great. Later I read the prototype, > > > > > > > > > > socketvar.h:extern void so_update_attrs(struct sonode *, int); > > > > > > > > > > Hmm. Not sure why DTrace is pulling meaningful information for args that > > > > > shouldn't exist - but it was working reliably so I've left that mystery to > > > > > investigate at a later date. > > > > > > > > > > At some point I'll get stuck into the Solaris source and may find extra > > > > > prototypes that aren't in /usr/include/sys. (Oh! looks like you tried that > > > > > already!) Or perhaps DTrace is reading memory addresses from a previous > > > > > function that it didn't clear. > > > > > > > > > > Whatever the reason, I imagine in the long term this will be done by a > > > > > network provider and not fbt, so these scripts are rather temporary. > > > > > > > > > > no worries, > > > > > > > > > > Brendan > > > > > > > > > > > > > > > > James Dickens > > > > > [...] > > > > > > > > > > > > from src/src/uts/common/fs/sockfs/socksubr.c > > > > > > > > > > > > void > > > > > > so_update_attrs(struct sonode *so, int flag) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > DTrace mailing list > > > > > DTrace at opensolaris.org > > > > > https://www.opensolaris.org/mailman/listinfo/dtrace > > > > > > > > > > > > > > > > > > > > > > > > > From brendan.gregg at tpg.com.au Fri Jan 21 10:44:33 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Sat, 22 Jan 2005 05:44:33 +1100 (EST) Subject: [DTrace] Loose ends Message-ID: G'Day Folks, I've rewritten execsnoop, opensnoop and shellsnoop so that they are wrapped in the Bourne shell to provide command line options (they have all lost their ".d" extensions, but the old versions are still online). They are rather more meaningful tools now. (Eg, I can run shellsnoop with "-qp PID" with the PID of a shell, and see only the characters echo'd to their screen, in their original formatting). I common switch I'm adding is -v for time, # ./execsnoop -v STRTIME UID PID PPID ARGS 2005 Jan 22 00:07:22 0 23053 20933 date 2005 Jan 22 00:07:24 0 23054 20933 uname -a 2005 Jan 22 00:07:25 0 23055 20933 ls -latr 2005 Jan 22 00:07:27 0 23056 20933 df -k 2005 Jan 22 00:07:29 0 23057 20933 ps -ef 2005 Jan 22 00:07:34 0 23058 20933 uptime Which has been quite helpful so far. ... Tomorrow I'll be heading overseas for 4 weeks, and I'm not sure I'll have access to this email address or be able to update my website. So I'm doing what I can beforehand. During the next few weeks I'll be easier to catch from bdgregg at yahoo.com.au, and I'll be somewhere in California, I think. Oh, and it looks like I'll miss all the fun, http://www.opensource.org/licenses/cddl1.php !!! cheers, Brendan Gregg [Sydney, Australia] From brendan.gregg at tpg.com.au Fri Jan 21 19:37:22 2005 From: brendan.gregg at tpg.com.au (Brendan Gregg) Date: Sat, 22 Jan 2005 14:37:22 +1100 (EST) Subject: [DTrace] DTrace troubleshooting In-Reply-To: Message-ID: G'Day Folks, I've just added to the troubleshooting examples with a DTrace look at catman, http://www.brendangregg.com/DTrace/lostcpu.html I'm making a list of these at the bottom of my main dtrace website. I picked catman as it creates loads of short lived processes that are difficult to troubleshoot using traditional tools, yet it can still flog the CPUs. (Imagine trying to spot it using ps -ef. :) I hope to add a handful more examples in about a month. cheers, Brendan [Sydney, Australia] On Mon, 17 Jan 2005, Brendan Gregg wrote: > G'Day Folks, > > A few of us were discussing today why SMC took so long to boot the first > time, so I pulled out DTrace and started looking. Pretty quickly we had a > good idea why (eg, several million 1 byte read()s), and as it made for a > nice demo of DTrace I just wrote a website for it. > > http://www.brendangregg.com/DTrace/smc.html > [...] From jamesd.wi at gmail.com Sun Jan 23 05:02:45 2005 From: jamesd.wi at gmail.com (James Dickens) Date: Sun, 23 Jan 2005 07:02:45 -0600 Subject: [DTrace] sock_top.d Message-ID: i finished my second script, it tracks socket data on a per UID/PID basis you can get the script and see sample output on my blog http://uadmin.blogspot.com/ feel free to reply with comments James Dickens -------------- next part -------------- A non-text attachment was scrubbed... Name: sock_top.d Type: application/octet-stream Size: 2173 bytes Desc: not available URL: