Dereferencing user space pointers in DTrace(1M) – a worked example

Introduction

Every time I write a DTrace script I learn something new. This time is was all about exploring a user process problem and using copyin().

Take a look at bug 6523693 (mountd core dump in nfsauth_prog on S10). I could have added some debugging to the mountd source or tinkered about with truss(1) but it was (a) Friday afternoon, and (b) it’s not the Solaris 10 way šŸ™‚

What I was trying to find …

Having explored the problem and the core dump a little I found that check_client() was being called with a NULL clnames pointer.

in_access_list(ffbffc60, 0, ffa33, 1, 2a7d0, 19400)
check_client_old(0, ffbffc60, 0, 1, ffa30, ffa33))
check_client()
nfsauth_access_svc(ffbffcd4, ffbffcd0, 2d330, ff32f928, 4b59c, cfb40)
nfsauth_prog+0x98(fdb38, fdb38, 2d330, 0, ff0d2000, 1)
libnsl.so.1`_svc_prog_dispatch+0x184(fdb38, fdc18, 2d330, 1, 162a0, 1)
libnsl.so.1`_svc_run_mt+0x640(fdb38, ff32ce34, 0, ff322f58, ffffff39, ff327054)
libnsl.so.1`svc_run+0x94(193dc, ff327018, 2a008, ff322f58, 4f788, 0)
main+0x754(19000, 19000, 12c00, 18400, 3, 19000)
_start+0x108(0, 0, 0, 0, 0, 0)

The OpenSolaris version of nfsauth_access_svc looks like this:

52 static void
53 nfsauth_access(auth_req *argp, auth_res *result)
54 {
55 	struct netconfig *nconf;
56 	struct nd_hostservlist *clnames = NULL;
57 	struct netbuf nbuf;
58 	struct share *sh;
59 	char tmp[MAXIPADDRLEN];
60 	char *host = NULL;
61
62 	result->auth_perm = NFSAUTH_DENIED;
63
64 	/*
65 	 * Convert the client's address to a hostname
66 	 */
67 	nconf = getnetconfigent(argp->req_netid);
68 	if (nconf == NULL) {
69 		syslog(LOG_ERR, "No netconfig entry for %s", argp->req_netid);
70 		return;
71 	}
72
73 	nbuf.len = argp->req_client.n_len;
74 	nbuf.buf = argp->req_client.n_bytes;
75
76 	if (netdir_getbyaddr(nconf, &clnames, &nbuf)) {
77 		host = &tmp[0];
78 		if (strcmp(nconf->nc_protofmly, NC_INET) == 0) {
79 			struct sockaddr_in *sa;
80
81 			/* LINTED pointer alignment */
82 			sa = (struct sockaddr_in *)nbuf.buf;
83 			(void) inet_ntoa_r(sa->sin_addr, tmp);
84 		} else if (strcmp(nconf->nc_protofmly, NC_INET6) == 0) {
85 			struct sockaddr_in6 *sa;
86 			/* LINTED pointer */
87 			sa = (struct sockaddr_in6 *)nbuf.buf;
88 			(void) inet_ntop(AF_INET6, sa->sin6_addr.s6_addr,
89 				    tmp, INET6_ADDRSTRLEN);
90 		}
91 		clnames = anon_client(host);
92 	}
93
94 	/*
95 	 * Now find the export
96 	 */
97 	sh = findentry(argp->req_path);
98 	if (sh == NULL) {
99 		syslog(LOG_ERR, "%s not exported", argp->req_path);
100 		goto done;
101 	}
102
103 	result->auth_perm = check_client(sh, &nbuf, clnames, argp->req_flavor);
104

The combination of static function names and pointers to pointers made this somewhat hard to debug using the pid provider. In particular, I wanted to decode struct nd_hostservlist *clnames.

Dereferencing a user address pointer

We first decode the call to netdir_getbyaddr(nconf, &clnames, &nbuf) which graciously provides us with the address of the pointer:

pid$1::netdir_getbyaddr:entry
/self->nfsauth = 1/
{
self->arg1 = arg1;
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p", self->clnames);
}

We save the value of arg1 (&clnames) as a thread-local variable so that we can use it later to monitor what happens to that pointer. As it it’s a user address pointer we can’t look at it directly, we have to copy it into the kernel.

copyin() copies from the user address to DTrace scratch space. It returns a DTrace pointer to the result. At this point we have to assume that we’re examining a 32-bit process – pointers are 32-bit – so the copyin() copies 4 bytes. To use this further we need to cast it to what it represents … in this case a pointer to a 32-bit number. To examine that value we dereference the DTrace pointer. The result is what clnames is pointing to – which should be NULL at this point.

On the return from that function we can check what clnames is set to, and decode it if possible. As DTrace doesn’t have conditionals we have to use predicates:

pid$1::netdir_getbyaddr:return
/self->nfsauth = 1 && *(uint32_t *)copyin(self->arg1, 4)/
{
printf("returned %d, ", arg1);
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p\n", self->clnames);
this->h_cnt = (int *)copyin(self->clnames, 4);
this->h_hostservs = (uint32_t *)copyin(self->clnames + 4, 4);
printf("h_cnt %d, ", *this->h_cnt);
printf("h_hostservs %#p\n", *this->h_hostservs);
this->h_hostp = (uint32_t *)copyin(*this->h_hostservs, 4);
this->h_host = copyinstr(*this->h_hostp);
printf("h_host %s", this->h_host);
}
pid$1::netdir_getbyaddr:return
/self->nfsauth = 1 && ! *(uint32_t *)copyin(self->arg1, 4)/
{
printf("returned %d, ", arg1);
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p", self->clnames);
}

The predicate dereferences the clnames pointer and only decodes the result if it’s non-NULL.

The probe actions further demonstrate dereferencing user pointers. The structure we’re picking apart looks like this:

struct nd_hostservlist {
int			h_cnt;		/* number of nd_hostservs */
struct nd_hostserv	*h_hostservs;	/* the entries */
};

As self->clnames is a DTrace pointer to a 32-bit user address we can use copyin() again to get what it’s pointing to … in this case a structure. The first member of which is an int …

this->h_cnt = (int *)copyin(self->clnames, 4);

The second member of which is another pointer to another structure:

this->h_hostservs = (uint32_t *)copyin(self->clnames + 4, 4);

The latter can be dereferenced again using copyin() to decode:

struct nd_hostserv {
char		*h_host;	/* the host name */
char		*h_serv;	/* the service name */
};

This is the complete script

#!/usr/sbin/dtrace -s
/*
* This DTrace script traces calls to netdir_byaddr() et al. in
* nfsauth_access_svc(). It's somewhat complicated by some of the
* functions being static and the general complexity of manipulating
* user pointers in DTrace's kernel space. It handles both resolved
* and unresolved names.
*/
/*
* This was written to debug 6523693 (mountd core dump in nfsauth_prog
* on S10) where clnames was NULL on calling client_check(). It should
* be started:
*
*	# ./mountd-auth-debug.d $(pgrep -x mountd)
*
*/
pid$1::nfsauth_prog:entry
{
self->nfsauth = 1;
}
pid$1::nfsauth_prog:return
/self->nfsauth = 1/
{
self->nfsauth = 0;
}
pid$1::netdir_getbyaddr:entry
/self->nfsauth = 1/
{
self->arg1 = arg1;
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p", self->clnames);
}
pid$1::netdir_getbyaddr:return
/self->nfsauth = 1 && *(uint32_t *)copyin(self->arg1, 4)/
{
printf("returned %d, ", arg1);
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p\n", self->clnames);
this->h_cnt = (int *)copyin(self->clnames, 4);
this->h_hostservs = (uint32_t *)copyin(self->clnames + 4, 4);
printf("h_cnt %d, ", *this->h_cnt);
printf("h_hostservs %#p\n", *this->h_hostservs);
this->h_hostp = (uint32_t *)copyin(*this->h_hostservs, 4);
this->h_host = copyinstr(*this->h_hostp);
printf("h_host %s", this->h_host);
}
pid$1::netdir_getbyaddr:return
/self->nfsauth = 1 && ! *(uint32_t *)copyin(self->arg1, 4)/
{
printf("returned %d, ", arg1);
self->clnames = *(uint32_t *)copyin(self->arg1, 4);
printf("clnames %#p", self->clnames);
}
pid$1::anon_client:entry
/self->nfsauth = 1/
{
printf("host '%s'", copyinstr(arg0));
}
pid$1::anon_client:return
/self->nfsauth = 1/
{
printf("returned %#p\n", arg1);
this->h_cnt = (int *)copyin(arg1, 4);
this->h_hostservs = (uint32_t *)copyin(arg1 + 4, 4);
printf("h_cnt %d, ", *this->h_cnt);
printf("h_hostservs %#p\n", *this->h_hostservs);
this->h_hostp = (uint32_t *)copyin(*this->h_hostservs, 4);
this->h_host = copyinstr(*this->h_hostp);
printf("h_host %s", this->h_host);
}
pid$1::check_client:entry
/self->nfsauth = 1/
{
printf("clnames %#p\n", arg2);
this->h_cnt = (int *)copyin(arg2, 4);
this->h_hostservs = (uint32_t *)copyin(arg2 + 4, 4);
printf("h_cnt %d, ", *this->h_cnt);
printf("h_hostservs %#p\n", *this->h_hostservs);
this->h_hostp = (uint32_t *)copyin(*this->h_hostservs, 4);
this->h_host = copyinstr(*this->h_hostp);
printf("h_host %s", this->h_host);
}

Conclusion

As we say in Perl TIMTOWTDI and I did this as much as an exercise as anything else but, think about this:

I can send this script to any customer running Solaris 10 and decode their mountd behaviour associated with the bug. No debug binaries, no intrusive and wasteful trussing or complicated debugger procedures.

As any DTrace person will tell you, most DTrace scripts are evolved as they explore the problem. DTrace is a superb observability tool but like all observations it matters where you look and having found one clue you follow the trail to the next.

Think: Treasure trail … enjoy šŸ™‚

PS Bonus points for anyone who can spot the obvious lack of error checking in that code

Advertisements
Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

  • Sporadic tweets from me

    Error: Twitter did not respond. Please wait a few minutes and refresh this page.

  • Tags of interest

  • Categories

%d bloggers like this: