spin lock variations...

spin_lock_irqsave(spinlock_t *lock, unsigned long flags);

Disables interrupts on the local processor and stores the current interrupt state in flags. Note that all of the spinlock primitives are defined as macros, and that the flags argument is passed directly, not as a pointer.

spin_lock_irq(spinlock_t *lock);

Acts like spin_lock_irqsave, except that it does not save the current interrupt state. This version is slightly more efficient than spin_lock_irqsave, but it should only be used in situations in which you know that interrupts will not have already been disabled.

spin_lock_bh(spinlock_t *lock);

Obtains the given lock and prevents the execution of bottom halves.

spin_unlock(spinlock_t *lock);

spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags);

spin_unlock_irq(spinlock_t *lock);

spin_unlock_bh(spinlock_t *lock);

These functions are the counterparts of the various locking primitives described previously. spin_unlock unlocks the given lock and nothing else. spin_unlock_irqrestore possibly enables interrupts, depending on the flags value (which should have come from spin_lock_irqsave). spin_unlock_irq enables interrupts unconditionally, and spin_unlock_bh reenables bottom-half processing. In each case, your function should be in possession of the lock before calling one of the unlocking primitives, or serious disorder will result.

spin_is_locked(spinlock_t *lock);

spin_trylock(spinlock_t *lock)

spin_unlock_wait(spinlock_t *lock);

spin_is_locked queries the state of a spinlock without changing it. It returns nonzero if the lock is currently busy. To attempt to acquire a lock without waiting, use spin_trylock, which returns nonzero if the operation failed (the lock was busy). spin_unlock_wait waits until the lock becomes free, but does not take possession of it.

A table for locking among different contexts:

http://www.kernel.org/pub/linux/kernel/people/rusty/kernel-locking/c214.html

tcp_sock.ucopy always clean in BLCR checkpoint

extremely fortunate: while BLCR's checkpoint signal handler is executed tcp_sock's fasthpath is surely not in action and the ucopy field is clear. tcp_recvmsg() breaks the main loop (with -ERESTARTSYS) in case of signal_pending(), but ucopy is cleaned up first. As a consqeuence, during cr_restart we can simply reinject packages by calling ip_rcv_finish() on each, tcp_recvmsg() will be reexecuted and it will process the queues properly.

on netfilter's NF_QUEUE verdict..

Installing a netfilter hook that returns NF_QUEUE on certain packets will cause the kernel to find a nf_queue_handler and call it with the given packet. If no handler is installed tha packet is discarded. A handler can be registered with the nf_register_queue_handler(). (ip_queue module uses this for expose the packets to userspace.) After your queue handler is done, you are supposed to insert it back to the network stack by calling nf_reinject().

TCP send in Linux 2.6

tcp_sendmsg() copies the data from userspace by buiding socket buffers and calling skb_entail() on each packet. skb_entail() calls tcp_add_write_queue_tail() for adding the buffer to the sk_write_queue of the socket and setting sk_send_head to the packet if it's not yet set.
Note the difference between sk_write_queue and sk_send_head, send_head denotes the first package which has not been requested for transmitting through the lower layer (IP) while all that the packages remain on the write_queue (until they are acked, check tcp_ack() during receiving)
In case the last packet added is the only one waiting for transmission (i.e. skb == skb_send_head) tcp_push_one() is called in order to advance sk_send_head and to call tcp_transmit_skb() on the package, otherwise it is simply enqueued.
Before returning the number of bytes copied from userspace tcp_sendmsg() calls tcp_push(), which calls __tcp_push_pending_frames(), which calls tcp_write_xmit(), the general function for iterating packets from sk_send_head and calling tcp_transmit_skb() for each of them.
Both tcp_push_one() and tcp_write_xmit() call tcp_transmit_skb() for the actuall transmission of a package through the socket's icsk_af_ops->queue_xmit() function.
Both tcp_push_one() and tcp_write_xmit() call tcp_event_new_data(), which advances sk_send_head and sets up the TCP_TIME_RETRANS timer of the socket if it's not set yet.

TCP receive in Linux 2.6

There are two ways of receiving packets from the NIC in the kernel, either through interrupts or by polling the interface.
During receiving through interrupts the netif_rx() function is called with the skb sk_buff, which enques the packet on the softnet_data's input_pkt_queue and schedules the NET_RX_SOFTIRQ softirq. (For the case when network cards would generate too many interrupts, the driver can register a poll function and switch off interrupts, by using the NAPI interface.)
The NET_RX_SOFTIRQ's func, net_rx_action() iterates the per processor softnet_data's poll queue - on which the backlog pseudo device is present as well - and calls each poll function. The backlog device's poll function, process_backlog(), is actually the one which processes the softnet_data's input_pkt_queue and pushes the sk_buff packets to the upper layers by calling netif_receive_skb() on each packet.
netif_receive_skb() will iterate the network packet handlers which match the protocol type and calls deliver_skb() function which in turn calls the protocol's "func" function, IP's packet handler function is ip_rcv().
ip_rcv() eventually calls ip_rcv_finish() which calls ip_route_info() on the skb in order to fill in the dst field. The dst field is a "rtable" struct which embeds a "dst_entry" struct. The dst_entry's ipunt field is a function pointer which is set to ip_local_deliver() in case of a packet that has to be delivered locally.
There is a global inet_protos array (hash) of "net_protocol" structs which represents the registered transport layer protocols, TCP's net_protocol is called tcp_protocol. ip_local_deliver() dereferences the protocol array and calls its handler function, TCP's handler is tcp_v4_rcv().
There are three receive queues of a socket, the sk_backlog, the ucopy.prequeue and the main sk_receive_queue.
The three queues have the following purposes: prequeue is responsible for putting off in order data processing to process context in case a process is waiting on the socket (called fastpath), receive queue is the standard way of receiving packets if no reader process is waiting when the package was received. Backlog is used to temporarily store received packages in case a reader is processing the receive queue.
tcp_v4_rcv() calls tcp_rcv_established() if the connection is established and puts a socket buffer on the preqeue if the socket is not locked but there is a user process waiting for reading (ucopy.task is set) and the socket buffer contains in order data according to the expected sequence number.
The main purpose of the prequeue mechanism is to allow processing of socket buffers in process context and therefore decreasing the amount of time spent in bottom half (softirq) processing.
The process is awaken after putting the buffer on the queue. If there is no user process waiting tcp_v4_do_rcv() is called directly in the softirq context and the skb is placed on the main receive queue. If the socket is locked, the packet goes to the backlog queue. tcp_recvmsg() is the function called from process context in order to copy segments to user space. It processes the receive queue and/or installs itself as a waiter for data in case the queue is empty or the requested amount of bytes were not available yet. Doing so it calls tcp_data_wait() which will wait until tcp_v4_rcv() receives packages from the IP layer.
Backlog queue is also processed in tcp_recvmsg() right before the socket is released after a read operation.
Note that tcp_v4_do_rcv() can be called either in softirq or process context. Process context is responsible for data transfer to userspace, while softirq will place the buffer on the receive queue. tcp_v4_do_rcv() is called in process context through the sk_backlog_rcv field of the socket when the prequeue is iterated in tcp_prequeue_process().
Note that tcp_rcv_established() calls tcp_ack() as well, which cleans the socket's write_queue (i.e. retransmit queue) according to the ACK received in the package (check send side).

WinNT page replacement

Page replacement call stack: MmAccessFault -> MiDispatchFault -> MiAllocateWsle -> MiDoReplacement -> MiTrimWorkingSet -> decides which pages should be removed from the working set and adjusts firstFree and NextSlot accordingly. (the page is included to FirstFree position unless there is no free position left, in which case NextSlot is the one to be replaced.) besides direct page fault handling the KeBalanceSetManager thread -> MmWorkingSetManager -> if (MiComputeSystemTrimCriteria) -> MiProcessWorkingSets -> MiTrimWorkingSet and MiAgeWorkingSet: Aging pages by turning off access bits and incrementing age counts for pages which haven't been accessed.

example call stacks:

Prototype PTE:
f3dffb48 80837fc0 nt!MiFreeWsle+0x67 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 1360]
f3dffb80 808380fc nt!MiReplaceWorkingSetEntry+0xcc [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 730]
f3dffbb4 808381e3 nt!MiDoReplacement+0xee [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 390]
f3dffbd0 80848ec9 nt!MiAllocateWsle+0x4b [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 189]
f3dffbe8 8084a60a nt!MiAddValidPageToWorkingSet+0x6d [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 6359]
f3dffc24 8084b622 nt!MiCompleteProtoPteFault+0x274 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 3660]
f3dffc58 8084c05a nt!MiResolveProtoPteFault+0x4c [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 3103]
f3dffcf0 80857954 nt!MiDispatchFault+0x832 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 1040]
f3dffd4c 80886278 nt!MmAccessFault+0x9c0 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\mmfault.c @ 2001]
f3dffd4c 7c819e44 nt!_KiTrap0E+0xdc [C:\Users\bgerofi\code\WinNTKernel\base\ntos\ke\i386\trap.asm @ 5527]

Transition PTE:
f3dffb98 80837fc0 nt!MiFreeWsle+0x59 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 1356]
f3dffbd0 808380fc nt!MiReplaceWorkingSetEntry+0xcc [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 730]
f3dffc04 808381e3 nt!MiDoReplacement+0xee [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 390]
f3dffc20 80848ec9 nt!MiAllocateWsle+0x4b [c:\users\bgerofi\code\winntkernel\base\ntos\mm\wslist.c @ 189]
f3dffc38 8084b5a7 nt!MiAddValidPageToWorkingSet+0x6d [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 6359]
f3dffc6c 8084c225 nt!MiResolveTransitionFault+0x539 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 2484]
f3dffcf0 80857954 nt!MiDispatchFault+0x9fd [c:\users\bgerofi\code\winntkernel\base\ntos\mm\pagfault.c @ 1072]
f3dffd4c 80886278 nt!MmAccessFault+0x9c0 [c:\users\bgerofi\code\winntkernel\base\ntos\mm\mmfault.c @ 2001]
f3dffd4c 77eaae5f nt!_KiTrap0E+0xdc [C:\Users\bgerofi\code\WinNTKernel\base\ntos\ke\i386\trap.asm @ 5527]

in this case MiReplaceWorkingSetEntry() is where the actual decision is made regarding which page to replace, the search starts from NextSlot (which is determined/maintained by MiTrimWorkingSet()).
The "WsleAllocationAny" flag value determines that any of the entries which fulfills the aging requirement is okay, while "WsleAllocationReplace" attempts to replace the first that is possible to replace. (Note that the entire working set could be locked in which case the replacement fails) Replacing an entry in fact means removing one and therefore setting FirstFree which is in trun used in MiAllocateWsle().

WinNT working set issue..

When u debug the working set, strange issue:

!process 0 0
check the name and the address

dt _EPROCESS address
dt _MMSUPPORT address+0x1e8
dt PMMWSL address+0x1e8+0x028

you will get c0502000 all the time, this is because the pointer stores a virtualaddress in the process address space but WinDBG will show you the structure in the active process address space.
(seems like WinDBG translates the virtual addresses according to the current rocess page table?)
You have to achieve to be in the debugger while the current process is the one u are investigating, for instance by adding a system call and "Run To Cursor": in the debugger to your syscall..