From Peter.Harvey at Sun.COM Fri Apr 25 16:52:25 2008 From: Peter.Harvey at Sun.COM (Peter Harvey) Date: Sat, 26 Apr 2008 00:52:25 +0100 Subject: [libumem-discuss] ::findleaks reports a zero-depth stack for a subset of leaked buffers Message-ID: <48126EB9.9050209@Sun.COM> Any ideas on why ::findleaks would report zero-depth stacks for some leaks and not others? > ::findleaks CACHE LEAKED BUFCTL CALLER 000000010121c868 1 000000010124c470 libnsl.so.1`xdr_bytes+0xb4 000000010121c2e8 1 00000001013a30c0 0 000000010121d628 1 0000000101263c30 libnsl.so.1`xdr_reference+0x50 000000010121cde8 25953 00000001012503a0 0 000000010121c868 1 000000010124c390 libnsl.so.1`xdr_reference+0x50 000000010121cb28 1 000000010125ce10 libnsl.so.1`xdr_reference+0x50 000000010121c5a8 1 000000010126e720 libnsl.so.1`xdr_string+0x11c 000000010121c868 1 000000010124c550 libnsl.so.1`xdr_string+0x11c 000000010121c868 1 000000010124c2b0 libnsl.so.1`xdr_string+0x11c 000000010121c5a8 1 000000010126e640 libnsl.so.1`xdr_string+0x11c 000000010121c5a8 1 000000010126e560 libnsl.so.1`xdr_string+0x11c ---------------------------------------------------------------------- Total 25963 buffers, 1661336 bytes > I've found one other OpenSolaris thread on this[1] but it didn't get me very far. I've been back and forth through the libumem code and as far as I can see the only thing that might do this would be getpcstack() returning zero, eg due to a bogus frame pointer. A few hours of code inspection later I'm still none the wiser. I could DTrace this but that's not easy as I'm working on a core file that's been sent to me. Access to the leaking application isn't readily available. For bonus marks, could someone point me to a reasonably concise explanation about how ::findleaks does its work? Again, reading the code has got me quite a long way but there are areas I'm still unclear about. In particular, does it only look for references to the base address or does it recognise offsets within the buffer? -- Peter [1] -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3249 bytes Desc: S/MIME Cryptographic Signature Url : http://mail.opensolaris.org/pipermail/libumem-discuss/attachments/20080426/0e4476e8/attachment.bin From jonathan.adams at sun.com Fri Apr 25 17:52:15 2008 From: jonathan.adams at sun.com (Jonathan Adams) Date: Fri, 25 Apr 2008 17:52:15 -0700 Subject: [libumem-discuss] ::findleaks reports a zero-depth stack for a subset of leaked buffers In-Reply-To: <48126EB9.9050209@Sun.COM> References: <48126EB9.9050209@Sun.COM> Message-ID: <20080426005215.GB20027@eng.sun.com> On Sat, Apr 26, 2008 at 12:52:25AM +0100, Peter Harvey wrote: > Any ideas on why ::findleaks would report zero-depth stacks for some > leaks and not others? > > > ::findleaks > CACHE LEAKED BUFCTL CALLER > 000000010121c868 1 000000010124c470 libnsl.so.1`xdr_bytes+0xb4 > 000000010121c2e8 1 00000001013a30c0 0 > 000000010121d628 1 0000000101263c30 libnsl.so.1`xdr_reference+0x50 > 000000010121cde8 25953 00000001012503a0 0 > 000000010121c868 1 000000010124c390 libnsl.so.1`xdr_reference+0x50 > 000000010121cb28 1 000000010125ce10 libnsl.so.1`xdr_reference+0x50 > 000000010121c5a8 1 000000010126e720 libnsl.so.1`xdr_string+0x11c > 000000010121c868 1 000000010124c550 libnsl.so.1`xdr_string+0x11c > 000000010121c868 1 000000010124c2b0 libnsl.so.1`xdr_string+0x11c > 000000010121c5a8 1 000000010126e640 libnsl.so.1`xdr_string+0x11c > 000000010121c5a8 1 000000010126e560 libnsl.so.1`xdr_string+0x11c > ---------------------------------------------------------------------- > Total 25963 buffers, 1661336 bytes > > > > I've found one other OpenSolaris thread on this[1] but it didn't get me > very far. I've been back and forth through the libumem code and as far > as I can see the only thing that might do this would be getpcstack() > returning zero, eg due to a bogus frame pointer. A few hours of code > inspection later I'm still none the wiser. What is the output of ::findleaks -d for the two "0" callers? This could happen if the code was compiled in a way that omits frame pointers; do the other stack traces terminate with library functions? (is this sparc or Intel?) > I could DTrace this but that's not easy as I'm working on a core file > that's been sent to me. Access to the leaking application isn't readily > available. > > For bonus marks, could someone point me to a reasonably concise > explanation about how ::findleaks does its work? Again, reading the code > has got me quite a long way but there are areas I'm still unclear about. findleaks() essentially runs: leaky_subr_estimate() // get upper-bound on # buffers leaky_subr_fill() // get actual buffer list qsort(array of buffers by beginning address) leaky_subr_run() // invoke leaky_grep*() on the roots // of the address space for each leaked buffer leaky_subr_add_leak(buffer); and then various stuff to dump the results out. leaky_grep() is the heart of ::findleaks. It takes an [addr,len) region, scans every pointer in it, to see if it is inside of a buffer on the buffer list. Any found buffer is marked, and (conceptually) recursively processed. The actual algorithm avoids recursion, and uses a stack and a buffer to limit memory usage. > In particular, does it only look for references to the base address or > does it recognise offsets within the buffer? It looks for a pointer in the allocated span of the buffer; any offset inside the allocated space is used. Cheers, - jonathan