ASSERT (with clone) DynamoRIO debug check failure: .../dispatch.c:144 dcontext == get_thread_private_dcontext() || (dcontext->whereami == DR_WHERE_APP && dcontext->last_exit == get_syscall_linkstub())

[egrimley-arm]

A simple example to demonstrate the problem (if it is a problem):

#define _GNU_SOURCE
#include <sched.h>

int main()
{
    unsigned long flags = CLONE_VFORK | CLONE_VM;
    asm volatile("mov x8, #220\n\t" // SYS_clone
                 "mov x0, %0\n\t"
                 "mov x1, #0\n\t"
                 "mov x2, #0\n\t"
                 "mov x3, #0\n\t"
                 "mov x4, #0\n\t"
                 "svc #0\n\t"
                 "cbnz x0, 1f\n\t"
                 "mov x8, #93\n\t" // SYS_exit
                 "mov x0, #0\n\t"
                 "svc #0\n\t"
                 ".inst 0\n\t"
                 "1:"
                 : : "r" (flags) : "x0", "x1", "x2", "x3", "x4", "x8");
    return 0;
}

This calls clone with the stated flags and all other arguments zero, and the child calls exit.

Natively this works as expected with this output under strace:

...
2993559 brk(0x17ebb78)                  = 0x17ebb78
2993559 brk(0x17ec000)                  = 0x17ec000
2993559 mprotect(0x48d000, 16384, PROT_READ) = 0
2993559 clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK <unfinished ...>
2993560 exit(0)                         = ?
2993559 <... clone resumed>)            = 2993560
2993560 +++ exited with 0 +++
2993559 exit_group(0)                   = ?
2993559 +++ exited with 0 +++

Under DynamoRIO (a98db8f) I get:

<Application /tmp/a.out (2993714).  Internal Error: DynamoRIO debug check failure: .../dispatch.c:144 dcontext == get_thread_private_dcontext() || (dcontext->whereami == DR_WHERE_APP && dcontext->last_exit == get_syscall_linkstub())
(Error occurred @705 frags in tid 2993714)

In fact, whereami seems to be DR_WHERE_FCACHE.

It's an unusual app, of course (it's a simplified case of something I was doing to investigate a perhaps unrelated DynarmoRIO bug), but it's not immediately obvious to me what aspect of this program might be causing DynamoRIO to get confused. If I knew that I might be able to work around this (suspected) bug and get back to investigating the first (suspected) bug.

The non-debug build gives me:

<Application /tmp/a.out (2993970). Cannot correctly handle received signal 11 in thread 2993970: multiple native handlers.>

[egrimley-arm]

The CLONE_VFORK seems to be significant: with flags = CLONE_VM and wait(NULL) before the return 0 everything seems to work as expected.