So, I decided to leave Fabrix and as of last week I am software engineer and senior consultant at Dell IDC (Israel Developer Center).

Related posts:

1. Resume
2. I am looking for a new job
3. What it takes to be a QA engineer

As surprising as it is, when you write some information to the disk, it doesn’t get there immediately. In Linux especially, the kernel tries to cache write requests in memory as much as it can. This means that in addition to writing the data to the disk, kernel keeps it in memory. Consecutive read request to the same place on the disk will be much faster because there’s no need to read the information from slow disk – it is already in memory. On the other hand, the information goes to the hard-disk only after some period of time (short though) or when the system runs out of memory. In the meantime, Linux reports that data has been written, despite it is not yet on the disk.

This causes several interesting phenomena that you may have noticed. For example Linux machines with little memory, even though not all memory is in use, do use disk much more than machines with more memory. There are several reasons for it. One of them is because kernel doesn’t have enough memory to cache information from the disks and as a result read requests rarely hit the cache and more often go to the disk.

Want to see exactly how Linux reports successful write requests before data actually lands on the disk? Try writing some information to floppy disk and see how fast it is in Linux. The truth is that it’s still frustratingly slow. Linux just makes it look like a fast device.

Sheer thought that Linux doesn’t write the data to the disk, despite it says it did, may be pretty scary. But it shouldn’t really. First of all, if its not very busy it does write the data to the disk as soon as possible. Second, Linux does excellent job avoiding various problems and even if something bad happens, it is pretty good at recovering lost data. The way Linux works is excellent for 99% of users. This approach improves performance in various ways and makes the system more healthy and stable.

However, some folks out there are not happy with this situation. Some software systems cannot work the way Linux works. One of the examples are so called clustered file-systems – file-systems that are spread among multiple servers for redundancy purposes. Such systems need a way to know that data has been written to the disk for real, not just being cached. Also, such systems want to make sure that reads hit disk and not OS cache.

This is where direct I/O becomes handy. Direct I/O is a way to avoid entire caching layer in the kernel and send the I/O directly to the disk. Overall, this makes I/O slower and does not let Linux do various optimizations that it usually does. Also, it introduces some constraints on memory buffer that being used for the I/O. Yet sometimes it is inevitable.

Want to try it yourself? dd, I/O Swiss army knife, has an option called direct. It tells dd to do direct I/O instead of regular I/O. Another option for doing direct I/O is writing your own program that opens files with O_DIRECT flag (see open(2) for details).

Update Feb. 12: Thanks to Ivan for noting the difference between synchronous I/O and direct I/O. I updated the post to reflect the difference.

Related posts:

1. Swap vs. no swap
2. SMP affinity and proper interrupt handling in Linux
3. Aligned vs. unaligned memory access

One problem that you should take care of when tackling this problem, is that symbolic link can point to a symbolic link. Then symbolic link should also point to symbolic link. And once again, and again and again…

So what can we do about it? First, lets start with lstat() system all. It will tell us is the file we’re interested in, is actually a symbolic link. To be more precise, st_mode field in struct stat will have flag S_IFLNK if specified file is a symbolic link. Note that it should be lstat() and not stat() – the later will return information about file the link points to and not about the link.

Next step is to call readlink(). This system call returns name of the file pointed to by specified symbolic link.

Finally, you should repeat the process recursively, for the pointed to file. Here is the code that does it:

int recursive_deleter(const char* filename)
{
  struct stat st;
  char buffer[1024];

  if (lstat(filename, &st)) {
    perror("stat");
    return -1;
  }     

  if (st.st_mode & S_IFLNK == S_IFLNK) {
    memset(buffer, 0, sizeof(buffer));
    if (readlink(filename, buffer, sizeof(buffer)) < 0) {
      perror("readlink");
      return -1;
    }           

    printf("File %s is a symbolic link to %s\n", filename, buffer);

    if (recursive_deleter(buffer))
      return 0;
  }     

  printf("Deleting %s\n", filename);

  if (unlink(filename)) {
    perror("unlink");
    return -1;
  }     

  return 0;
}

Have fun!

Related posts:

1. Hex dump functions
2. Direct IO in Python

When its not enough, you can always upgrade to open() and close() from os module. Opening man page on open() (the system call – open(2)) reveals all those O_something options that you can pass to os.open(). But not all of them can be used in Python. For example, if you open a file with O_DIRECT and then try to write to it, you will end up with some strange error message.

>>> import os
>>> f = os.open('file', os.O_CREAT | os.O_TRUNC | os.O_DIRECT | os.O_RDWR)
>>> s = ' ' * 1024
>>> os.write(f, s)
Traceback (most recent call last):
  File "", line 1, in
OSError: [Errno 22] Invalid argument
>>>

Invalid argument?. What invalid argument? Hey there’s nothing wrong with those arguments…

Reading open(2) man page further reveals that working with O_DIRECT requires that all buffers used for I/O should be aligned to 512 byte boundary. But how can you have a memory buffer aligned to 512 bytes in Python?

Apparently, there’s a way. Python comes with a module called mmap. mmap() is a system call that allows one to map portion of file into memory. All writes to memory mapped file, go directly to file despite it looks like you’re working with plain memory buffer. Same with reads.

There’s one interesting thing about mmap. It works with granularity of one memory page – 4kb that is. So every memory mapped buffer is naturally memory aligned to 4kb, thus to 512 byte boundary too. But hey, shouldn’t mmap map files?

Well, apparently mmap can be used for memory allocations. I.e. specifying -1 as file descriptor does just that – allocates RAM, as much as you tell it. So, this is what we do:

>>> import os
>>> import mmap
>>>
>>> f = os.open('file', os.O_CREAT | os.O_DIRECT | os.O_TRUNC | os.O_RDWR)
>>> m = mmap.mmap(-1, 1024 * 1024)
>>> s = ' ' * 1024 * 1024
>>>
>>> m.write(s)
>>> os.write(f, m)
1048576
>>> os.close(f)
>>>

Note that mmap memory buffer object behaves like a file. I.e. you can write into the buffer and read from it – like I do in line 8. More on it in official documentation.

Have fun!

Related posts:

1. What’s is direct I/O anyway?
2. Recursively deleting symbolic link and what it points to
3. How to send SMS message with Python

I am joining as a chief video content consumer senior software engineer.

Related posts:

1. Resume

Anyway, here’s something else that I learned during one of the interviews.

I was asked to implement insertion into a linked list in C. Each list node has a number (num). Numbers in the list should be sorted in ascending order – node with lowest number should be first. Here is my naive implementation.

void list_insert(int num)
{
        list_node_t *n, *tmp, *prev;

        n = allocate_node();
        n->num = num;

        if (!head) {
                head = n;
                n->next = NULL;
        } else {
                tmp = head;
                prev = NULL;
                while (tmp) {
                        if (tmp->num >= n->num) {
                                if (prev == NULL) {
                                        head->next = n;
                                        n->next = NULL;
                                } else {
                                        n->next = tmp;
                                        prev->next = n;
                                }

                                break;
                        }

                        prev = tmp;
                        tmp = tmp->next;
                }

                if (!tmp) {
                        n->next = NULL;
                        prev->next = n;
                }
        }
}

Yeah, well it is pretty long, but that’s how you do it, right? You have to make sure that the list is not empty (line 8 ) – if it is empty, then new node should be placed at the head of the list. Then we have to find right place for the new node and insert it into its position (lines 14-29). Finally, if we didn’t find a good position for the new node, we should put the node at the end of the list (lines 31-34).

When I showed this to the examiner he asked me to rethink the routine. In particular he didn’t like number of if statements. After an hour of discussing various optimizations, this is what we came up with.

void list_insert(int num)
{
        list_node_t **tmp, *n;

        n = allocate_node();
        n->num = num;

        tmp = &head;
        while (*tmp) {
                if ((*tmp)->num >= n->num)
                        break;
                tmp = &(*tmp)->next;
        }

/*
        (*tmp) = n;
       n->next = (*tmp)->next;
*/

/*
        The code in the comment above is broken. Thanks to jagan
        for pointing this out. Here's the right code.
*/
        n->next = *tmp;
        *tmp = n;
}

There are several things that make this version better.

First of all, instead of having a pointer to current node (tmp) and previous node (prev), we have one pointer that corresponds to pointer to previous node. This way we can always get to the current node via next field in the previous node. Thus we don’t need two pointers anymore.

Also, this version uses pointer to pointer to node to traverse through the list. We don’t insert the node in the while loop anymore. Purpose of the while loop is to find right place for the new node. This spares if statement that tests if the list is empty and if statement in the loop.

Finally, once tmp points to the right place, we insert the node.

Second version runs faster too. I did a small program that tests both versions and it appears that the later version is something like 5% faster.

Related posts:

1. pthread mutex vs pthread spinlock
2. Todo list

This means that I am looking for a new job. The good thing is that now your or your friend’s company has  a chance to hire a programmer with ten years of experience in writing application for Linux and Linux kernel. So, if you can help, please pass my resume. You can find it here.

Thanks.

Alex.

Related posts:

1. I got a new job, take 3
2. Resume
3. New LKML daily podcast

Exanet is developing storage solutions for large organisations. ExaStore, main product of the company, is a clustered NAS gateway solution providing highly available and distributed data storage.

Related posts:

1. I am looking for a new job
2. A new kind of virtualization

Interrupt affinity, which I discussed here and here, has a fundamental problem. That is inevitable CPU cache misses. To emphasise this, think about what happens when your computer receives a packet from the network. Packet belongs to some connection. With interrupt affinity the packet would land on core X, while the chances are that previous packet on the same TCP connection has landed on core Y (X ≠ Y).

Handing the packet would require kernel to load TCP connection object into X’s cache. But, this is so ineffective. After all, the TCP connection object is already in Y’s cache. Wouldn’t it be better to handle second packet on core Y as well?

This is the  problem with interrupt affinity. From one point of view we want to spread interrupts to even the load on cores. From another point of view, doing simple round robin isn’t enough. The little fella that decides where each interrupt goes, should be able to look into the packet and depending on what TCP connection it belongs to, send the interrupt to core that handles all packets that belong to this connection.

Ideally, NICs should be able to:

1. Look into packets and identify connections.
2. Direct interrupt to core that handles the connection.

Apparently, this functionality already here. Devices that support MSI-X do exactly this.

Meet MSI-X

MSI-X is an extension to MSI. MSI replaces good old pin based interrupt delivery mechanism.

Each IO-APIC chip (x86 permits up to 5) has 24 legs, each connected to one or more devices. When IO-APIC receives an interrupt, it redirects the interrupt to one of the local-APICs. Each local-APIC connected to a core that receives an interrupt.

MSI provides a kind of protocol for interrupt delivery. Instead of raising signal on pins, PCI cards send a message over MSI and IO-APIC translates the message into right interrupt. Theoretically this means that each device can have number of interrupt vectors. In reality, plain MSI does not support this, but MSI-X does.

Modern high-end network cards that support MSI-X, implement multiple tx-rx queues. Each queue tied up to an interrupt vector and each NIC has plenty of them. I checked Intel’s 82575 chipset. With igb driver compiled properly, it has up to eight queues, four rx and four tx. Broadcom’s 5709 chipset provides eight queues (and eight interrupt vectors), each handling both rx and tx.

In kernel 2.6.24, kernel developers introduced new member of struct sk_buff called queue_mapping. This member tells incoming NIC driver what queue to use when transmitting the packet.

Before transmitting the packet, kernel decides what queue to use for this packet (net/core/dev.c:dev_queue_xmit()). It uses two techniques to do so. First, kernel can ask NIC driver to provide a queue number for the packet. This functionality, however, is optional in NIC drivers and at the moment both Intel and Broadcom drivers don’t provide it. Otherwise, kernel uses a simple hashing algorithm that produces 16 bit number from two ip addresses and (in case of TCP or UDP) two port numbers. All this happens in function named simple_tx_hash() in net/core/dev.c.

When receiving packets, things are even easier because NIC firmware and the driver decide what queue to use to introduce the packet to the kernel.

Using this simple technique kernel and modern NIC’s can verify that packets that belong to certain connection land on certain queue. Using interrupt affinity binding techniques you can bind certain interrupt vector to certain core (writing to smp_affinity, etc). Thus you can spread interrupts among multiple cores and yet make sure there are no cache misses.

Related posts:

1. Why interrupt affinity with multiple cores is not such a good thing
2. SMP affinity and proper interrupt handling in Linux
3. A new kind of virtualization

I’ve written an article explaining this mechanism in greater detail. Yet let me remind you how it works in two words.

Every x86 motherboard has a chip called IO-APIC. This is a device that controls interrupt delivery within your system. It knows how many CPUs are in your system and can direct various interrupts to various CPUs. It uses so called local APIC-ID as an identifier of the processor.

It has two modes of operation. In one mode it sends interrupts from certain device to single, predefined core. This mode of operation called fixed/physical mode. In another mode, it can deliver interrupts from certain device to multiple cores. The later mode called logical/low priority interrupt delivery mode.

When in logical mode, IO-APIC can deliver interrupts to up to eight cores. Source of this limitation is the size of the bitmask register that tells what CPUs should receive the interrupts. The bitmask is only eight bits long.

When considering the round robin-type of interrupt delivery mode (the logical mode), I cannot stop thinking about how it degrades performance.

You see, having burden of interrupt handling spread among multiple cores may solve some bottle-necks, but it creates a problem.

Consider network interface card for example. Lets say we have a TCP connection to some host out there. When packet arrives, the network card issues an interrupt and IO-APIC directs it to one of the cores. Next, the core handing the packet should fetch the TCP connection objects from the memory to its cache.

IO-APIC does not guarantee that next packet that belongs to the connection will be handled by the same core. So, it is likely that two cores will have to work with TCP connection object. Both of them will have to fetch its content into their cache. This will cause cache coherency problems (cache misses). And as you can learn from the article I’ve written on misaligned memory accesses, accessing memory that is not in cache can take up to 30 times more time than accessing cached RAM.

Moreover, assuming TCP connection object is properly protected using synchronization techniques, one of the cores will inevitably have to wait for the other, adding unnecessary delay to packet processing.

My point is that round-robin style interrupt delivery can be quiet nasty on performance. It is much better to deliver interrupts from certain device to given core.

Luckily, smp_affinity interface, that I mentioned in my old article, allows you to bind interrupts from certain device to certain (single) core.

On some computers IO-APIC does not support logical delivery mode. This can be because of buggy BIOS or too many CPUs. On such computers physical interrupt delivery mode is the only thing that works, so binding single interrupt to single core is the only choice and the only thing you can do is switch the core from one to another.

My point is that round-robin style interrupt delivery can:

1. Malfunction
2. Cause performance degradation.

So, when it still might become useful, you may ask? It depends on what you do with your computer. Usually, you don’t need round-robin style interrupt delivery. You only need it if you know that your computer receives lots of interrupts and you have real-time applications.

In this case, scheduler (which has no idea about interrupts) can schedule thread that requires lots of CPU time to run on core that serves interrupts. Since interrupts has higher priority, the thread will receive less CPU time. In case of real-time application it may result in reduced responsiveness.

Even so, you can still assign all interrupts to one core and use thread affinity techniques to make sure that your application doesn’t use that core.

Related posts:

1. MSI-X – the right way to spread interrupt load
2. SMP affinity and proper interrupt handling in Linux
3. A new kind of virtualization