tracking dirty pages without patching the kernel?

The main idea is to replace all writable mappings (both anonymous and vm_file) with a VM_SHARED pseudo file mapping, which will imitate the right behavior of the vm_area.
Each vm_area gets a separate pseudo file/address space which stores the original mapping's properties.

Initially all pages (also the anonymous ones that got COW-ed from a private mapping) are converted to shared mappings of this pseudo file. Reverse mappings and page cache need to be updated consistently (i.e. each page has to be linked to the pseudo address space and the vm_area has to be included into the address space's priority tree.)

Each iteration of the incremental update clears the write bit of all PTEs (belonging to the dirty pages).

The benefit of the pseudo file mapping is the address space callbacks, page_mkwrite() in do_wp_page() and fault() in __do_fault().
We will be always notified when a page gets written to first, while we don't utilize the dirty bit (swapping can still work) and don't miss page writes in case of an mprotect() call after the write (since the write itself had been logged before mprotect()).


Considering the different cases of the original mappings:
(anon vs. file backed / private vs. shared, bold text means the original)

vm_file && VM_SHARED:

init: we need to replace the address space operations, all pages are mapped as shared anyway, no COW necessary.
(the pseudo file's write_page will call the original write_page, ensuring that we actually modify the original file)

fault(): load the page through the original fault().

page_mkwrite(): calls the original and logs the event.

vm_file && !VM_SHARED:

init: we need to iterate the page table and find the pages present, the ones that are writable are now anonymous, so we have to convert them to shared mappings of the pseudo file mapping, most importantly the reverse mappings have to be taken care. the usage counter of these pages will be 1, so we will never copy them in subsequent writes.

(the pseudo file's write_page will not call the original write_page, ensuring that we only work on our private copy in the memory)

fault(): load the page through the original read

page_mkwrite(): here we have to make a copy of the original page because normally, COW would ensure that we get our own private copy. However instead of mapping it as anonymous page, we will map it as a shared page of the pseudo file so that consequent write faults will not find the page as anonymous and page_mkwrite() will be called again.
(we either make a new copy of all written pages in each iteration or keep track the ones which have been actually copied by us, it is not necessary to copy those again)

!vm_file && VM_SHARED:

init: (this is the case of shared memory) update the vma that it is a file mapping now, and see which other vmas map these pages (update those as well?-> we can find all mappings through the reverse priority tree)

BUT: are we supposed to migrate a process that shares memory with someone else??
(if we ensure that all the processes that share the memory are our targets, it could be done..)

fault(): creates a new mapping and zeros the page

page_mkwrite(): logs the write


!vm_file && !VM_SHARED:

init: our private malloc()ed memory and memory that we inherited from fork() (if no exec() occurred after)
(this is a problem if our process was fork()ed from a big address space and it is actually not using the most of it, but is that really a relevant case?)

1.) update the vma that it is a file mapping now with the pseudo file
2.) iterate PTEs and check the ones which are present:
- if writable, we already COW-ed, map as file shared of the pseudo file (this is the most common case, malloc()ed memory)
- if not yet writable, make a copy (as COW) and map as file shared of the pseudo file, this can be very expensive, if the address space is big... *

(* for avoiding the immediate copy, we could create a fake vma, link the page to it anonymously, store the address that it belonged to that page and mark the real pte as non-present. In this case during fault() we could check this fake vma and see if the given address was already present originally, if so we make a copy and actually map it in.)

fault(): creates a new mapping and zeros the page (or see *)

page_mkwrite(): we have our own copy, just log the write

git: pulling into a dirty tree

When you are in the middle of something, you learn that there are upstream changes that are possibly relevant to what you are doing. If your local changes conflict with the upstream changes, git pull refuses to overwrite your changes. In such a case, you can stash your changes away, perform a pull, and then unstash, like this:

$ git pull  

... 

file foobar not up to date, cannot merge.

$ git stash 

$ git pull 

$ git stash apply

IPMI, remote servers

ipmitool -I lan -H sun20-sp -U root chassis power on
ipmitool -I lan -H sun20-sp -U root chassis power forceoff
ipmitool -I lan -H sun20-sp -U root chassis power reset
ipmitool -I lan -H sun20-sp -U root chassis power status

ssh root@sun20-sp

start /SP/AgentInfo/console

spin lock variations...

spin_lock_irqsave(spinlock_t *lock, unsigned long flags);

Disables interrupts on the local processor and stores the current interrupt state in flags. Note that all of the spinlock primitives are defined as macros, and that the flags argument is passed directly, not as a pointer.

spin_lock_irq(spinlock_t *lock);

Acts like spin_lock_irqsave, except that it does not save the current interrupt state. This version is slightly more efficient than spin_lock_irqsave, but it should only be used in situations in which you know that interrupts will not have already been disabled.

spin_lock_bh(spinlock_t *lock);

Obtains the given lock and prevents the execution of bottom halves.

spin_unlock(spinlock_t *lock);

spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags);

spin_unlock_irq(spinlock_t *lock);

spin_unlock_bh(spinlock_t *lock);

These functions are the counterparts of the various locking primitives described previously. spin_unlock unlocks the given lock and nothing else. spin_unlock_irqrestore possibly enables interrupts, depending on the flags value (which should have come from spin_lock_irqsave). spin_unlock_irq enables interrupts unconditionally, and spin_unlock_bh reenables bottom-half processing. In each case, your function should be in possession of the lock before calling one of the unlocking primitives, or serious disorder will result.

spin_is_locked(spinlock_t *lock);

spin_trylock(spinlock_t *lock)

spin_unlock_wait(spinlock_t *lock);

spin_is_locked queries the state of a spinlock without changing it. It returns nonzero if the lock is currently busy. To attempt to acquire a lock without waiting, use spin_trylock, which returns nonzero if the operation failed (the lock was busy). spin_unlock_wait waits until the lock becomes free, but does not take possession of it.

A table for locking among different contexts:

http://www.kernel.org/pub/linux/kernel/people/rusty/kernel-locking/c214.html

tcp_sock.ucopy always clean in BLCR checkpoint

extremely fortunate: while BLCR's checkpoint signal handler is executed tcp_sock's fasthpath is surely not in action and the ucopy field is clear. tcp_recvmsg() breaks the main loop (with -ERESTARTSYS) in case of signal_pending(), but ucopy is cleaned up first. As a consqeuence, during cr_restart we can simply reinject packages by calling ip_rcv_finish() on each, tcp_recvmsg() will be reexecuted and it will process the queues properly.

on netfilter's NF_QUEUE verdict..

Installing a netfilter hook that returns NF_QUEUE on certain packets will cause the kernel to find a nf_queue_handler and call it with the given packet. If no handler is installed tha packet is discarded. A handler can be registered with the nf_register_queue_handler(). (ip_queue module uses this for expose the packets to userspace.) After your queue handler is done, you are supposed to insert it back to the network stack by calling nf_reinject().

TCP send in Linux 2.6

tcp_sendmsg() copies the data from userspace by buiding socket buffers and calling skb_entail() on each packet. skb_entail() calls tcp_add_write_queue_tail() for adding the buffer to the sk_write_queue of the socket and setting sk_send_head to the packet if it's not yet set.
Note the difference between sk_write_queue and sk_send_head, send_head denotes the first package which has not been requested for transmitting through the lower layer (IP) while all that the packages remain on the write_queue (until they are acked, check tcp_ack() during receiving)
In case the last packet added is the only one waiting for transmission (i.e. skb == skb_send_head) tcp_push_one() is called in order to advance sk_send_head and to call tcp_transmit_skb() on the package, otherwise it is simply enqueued.
Before returning the number of bytes copied from userspace tcp_sendmsg() calls tcp_push(), which calls __tcp_push_pending_frames(), which calls tcp_write_xmit(), the general function for iterating packets from sk_send_head and calling tcp_transmit_skb() for each of them.
Both tcp_push_one() and tcp_write_xmit() call tcp_transmit_skb() for the actuall transmission of a package through the socket's icsk_af_ops->queue_xmit() function.
Both tcp_push_one() and tcp_write_xmit() call tcp_event_new_data(), which advances sk_send_head and sets up the TCP_TIME_RETRANS timer of the socket if it's not set yet.