Today I’d like to talk about one nice trick that I learned few days ago.

When working with large software systems, memory management becomes an imperative. In C, you can easily allocate a large chunk of memory and allocate structure right on that buffer. This is by far more difficult in C++, because compiler has to call consturctor.

Apparently, you can, in a way, directly call object’s constructor . I.e. you can allocate an object, on specified memory region, without actually allocating this region.

This is how you do it.

char* s = new char[1024];

SomeClass* p = new (s) SomeClass;

First new operator just allocates 1024 bytes. This is good old allocation as we know it. Note the special syntax of the second new operator. It allocates the new object on memory specified in brackets. Basically, this calls SomeClass’s constructor using s as storage.

One thing that I don’t know how to do is how to call destructor on the object – i.e. how to delete an object in place.

Related posts:

1. Direct IO in Python
2. One sane voice in the crowd
3. Recursively deleting symbolic link and what it points to

I think this is going to become my new icon. What do you think?

No related posts.

At first, I thought less reads entire input first. This would make standard input stream free to process key presses from the user. So I decided to check this out. I created a 1Gb long text file and ran less on it. I expected less to take some time to show file contents, but it showed first lines of the file instantly. Also, it didn’t consume 1Gb of RAM as I expected.

The conclusion is obvious. less does not read entire input buffer before letting user to interact with itself. Then how?

Here’s what less does. It separates input file stream from user input stream. Both inputs initially come via standard input stream, so less separates between them. First it duplicates the standard input stream. This allocates a new file descriptor. Then it closes old file descriptor, freeing file descriptor 0 for use. Then it opens /dev/tty. When opening a file, Linux uses next available file descriptor. File descriptor used as standard input is 0, so when less opens /dev/tty again, file descriptor of the newly opened file has value 0.

Eventually, it ends up with the new input coming via standard input stream file descriptor (0), and old input still available via file descriptor that it duplicated in the beginning. It reads the input file from the duplicated file descriptor, and uses curses on standard input stream.

You may be wondering what is /dev/tty and what it has to do with standard input streams. This is really fascinating stuff.

As you know, in Linux, everything is a file. So is terminal. Linux uses device files to represent various system devices. /dev/tty is a file that represents terminal of the current process. When process reads from /dev/tty it becomes its input. When program writes to /dev/tty it becomes its standard output or standard error stream.

So, when less reopens /dev/tty, it actually recreates standard input. Older descriptor, one that has been duplicated, can no longer accept new input, but less still can use it to read what it has been written into it already.

Related posts:

1. Creating new application on top of SSH
2. sed – the missing manual
3. Direct IO in Python

So, I decided to leave Fabrix and as of last week I am software engineer and senior consultant at Dell IDC (Israel Developer Center).

Related posts:

1. Resume
2. I am looking for a new job
3. What it takes to be a QA engineer

As surprising as it is, when you write some information to the disk, it doesn’t get there immediately. In Linux especially, the kernel tries to cache write requests in memory as much as it can. This means that in addition to writing the data to the disk, kernel keeps it in memory. Consecutive read request to the same place on the disk will be much faster because there’s no need to read the information from slow disk – it is already in memory. On the other hand, the information goes to the hard-disk only after some period of time (short though) or when the system runs out of memory. In the meantime, Linux reports that data has been written, despite it is not yet on the disk.

This causes several interesting phenomena that you may have noticed. For example Linux machines with little memory, even though not all memory is in use, do use disk much more than machines with more memory. There are several reasons for it. One of them is because kernel doesn’t have enough memory to cache information from the disks and as a result read requests rarely hit the cache and more often go to the disk.

Want to see exactly how Linux reports successful write requests before data actually lands on the disk? Try writing some information to floppy disk and see how fast it is in Linux. The truth is that it’s still frustratingly slow. Linux just makes it look like a fast device.

Sheer thought that Linux doesn’t write the data to the disk, despite it says it did, may be pretty scary. But it shouldn’t really. First of all, if its not very busy it does write the data to the disk as soon as possible. Second, Linux does excellent job avoiding various problems and even if something bad happens, it is pretty good at recovering lost data. The way Linux works is excellent for 99% of users. This approach improves performance in various ways and makes the system more healthy and stable.

However, some folks out there are not happy with this situation. Some software systems cannot work the way Linux works. One of the examples are so called clustered file-systems – file-systems that are spread among multiple servers for redundancy purposes. Such systems need a way to know that data has been written to the disk for real, not just being cached. Also, such systems want to make sure that reads hit disk and not OS cache.

This is where direct I/O becomes handy. Direct I/O is a way to avoid entire caching layer in the kernel and send the I/O directly to the disk. Overall, this makes I/O slower and does not let Linux do various optimizations that it usually does. Also, it introduces some constraints on memory buffer that being used for the I/O. Yet sometimes it is inevitable.

Want to try it yourself? dd, I/O Swiss army knife, has an option called direct. It tells dd to do direct I/O instead of regular I/O. Another option for doing direct I/O is writing your own program that opens files with O_DIRECT flag (see open(2) for details).

Update Feb. 12: Thanks to Ivan for noting the difference between synchronous I/O and direct I/O. I updated the post to reflect the difference.

Related posts:

1. Swap vs. no swap
2. SMP affinity and proper interrupt handling in Linux
3. Aligned vs. unaligned memory access

I am joining as a chief video content consumer senior software engineer.

Related posts:

1. Resume
2. About

Anyway, here’s something else that I learned during one of the interviews.

I was asked to implement insertion into a linked list in C. Each list node has a number (num). Numbers in the list should be sorted in ascending order – node with lowest number should be first. Here is my naive implementation.

void list_insert(int num)
{
        list_node_t *n, *tmp, *prev;

        n = allocate_node();
        n->num = num;

        if (!head) {
                head = n;
                n->next = NULL;
        } else {
                tmp = head;
                prev = NULL;
                while (tmp) {
                        if (tmp->num >= n->num) {
                                if (prev == NULL) {
                                        head->next = n;
                                        n->next = NULL;
                                } else {
                                        n->next = tmp;
                                        prev->next = n;
                                }

                                break;
                        }

                        prev = tmp;
                        tmp = tmp->next;
                }

                if (!tmp) {
                        n->next = NULL;
                        prev->next = n;
                }
        }
}

Yeah, well it is pretty long, but that’s how you do it, right? You have to make sure that the list is not empty (line 8 ) – if it is empty, then new node should be placed at the head of the list. Then we have to find right place for the new node and insert it into its position (lines 14-29). Finally, if we didn’t find a good position for the new node, we should put the node at the end of the list (lines 31-34).

When I showed this to the examiner he asked me to rethink the routine. In particular he didn’t like number of if statements. After an hour of discussing various optimizations, this is what we came up with.

void list_insert(int num)
{
        list_node_t **tmp, *n;

        n = allocate_node();
        n->num = num;

        tmp = &head;
        while (*tmp) {
                if ((*tmp)->num >= n->num)
                        break;
                tmp = &(*tmp)->next;
        }

/*
        (*tmp) = n;
       n->next = (*tmp)->next;
*/

/*
        The code in the comment above is broken. Thanks to jagan
        for pointing this out. Here's the right code.
*/
        n->next = *tmp;
        *tmp = n;
}

There are several things that make this version better.

First of all, instead of having a pointer to current node (tmp) and previous node (prev), we have one pointer that corresponds to pointer to previous node. This way we can always get to the current node via next field in the previous node. Thus we don’t need two pointers anymore.

Also, this version uses pointer to pointer to node to traverse through the list. We don’t insert the node in the while loop anymore. Purpose of the while loop is to find right place for the new node. This spares if statement that tests if the list is empty and if statement in the loop.

Finally, once tmp points to the right place, we insert the node.

Second version runs faster too. I did a small program that tests both versions and it appears that the later version is something like 5% faster.

Related posts:

1. pthread mutex vs pthread spinlock
2. Todo list

This means that I am looking for a new job. The good thing is that now your or your friend’s company has  a chance to hire a programmer with ten years of experience in writing application for Linux and Linux kernel. So, if you can help, please pass my resume. You can find it here.

Thanks.

Alex.

Related posts:

1. About
2. I got a new job, take 3
3. Resume

Exanet is developing storage solutions for large organisations. ExaStore, main product of the company, is a clustered NAS gateway solution providing highly available and distributed data storage.

Related posts:

1. I am looking for a new job
2. A new kind of virtualization

Interrupt affinity, which I discussed here and here, has a fundamental problem. That is inevitable CPU cache misses. To emphasise this, think about what happens when your computer receives a packet from the network. Packet belongs to some connection. With interrupt affinity the packet would land on core X, while the chances are that previous packet on the same TCP connection has landed on core Y (X ≠ Y).

Handing the packet would require kernel to load TCP connection object into X’s cache. But, this is so ineffective. After all, the TCP connection object is already in Y’s cache. Wouldn’t it be better to handle second packet on core Y as well?

This is the  problem with interrupt affinity. From one point of view we want to spread interrupts to even the load on cores. From another point of view, doing simple round robin isn’t enough. The little fella that decides where each interrupt goes, should be able to look into the packet and depending on what TCP connection it belongs to, send the interrupt to core that handles all packets that belong to this connection.

Ideally, NICs should be able to:

1. Look into packets and identify connections.
2. Direct interrupt to core that handles the connection.

Apparently, this functionality already here. Devices that support MSI-X do exactly this.

Meet MSI-X

MSI-X is an extension to MSI. MSI replaces good old pin based interrupt delivery mechanism.

Each IO-APIC chip (x86 permits up to 5) has 24 legs, each connected to one or more devices. When IO-APIC receives an interrupt, it redirects the interrupt to one of the local-APICs. Each local-APIC connected to a core that receives an interrupt.

MSI provides a kind of protocol for interrupt delivery. Instead of raising signal on pins, PCI cards send a message over MSI and IO-APIC translates the message into right interrupt. Theoretically this means that each device can have number of interrupt vectors. In reality, plain MSI does not support this, but MSI-X does.

Modern high-end network cards that support MSI-X, implement multiple tx-rx queues. Each queue tied up to an interrupt vector and each NIC has plenty of them. I checked Intel’s 82575 chipset. With igb driver compiled properly, it has up to eight queues, four rx and four tx. Broadcom’s 5709 chipset provides eight queues (and eight interrupt vectors), each handling both rx and tx.

In kernel 2.6.24, kernel developers introduced new member of struct sk_buff called queue_mapping. This member tells incoming NIC driver what queue to use when transmitting the packet.

Before transmitting the packet, kernel decides what queue to use for this packet (net/core/dev.c:dev_queue_xmit()). It uses two techniques to do so. First, kernel can ask NIC driver to provide a queue number for the packet. This functionality, however, is optional in NIC drivers and at the moment both Intel and Broadcom drivers don’t provide it. Otherwise, kernel uses a simple hashing algorithm that produces 16 bit number from two ip addresses and (in case of TCP or UDP) two port numbers. All this happens in function named simple_tx_hash() in net/core/dev.c.

When receiving packets, things are even easier because NIC firmware and the driver decide what queue to use to introduce the packet to the kernel.

Using this simple technique kernel and modern NIC’s can verify that packets that belong to certain connection land on certain queue. Using interrupt affinity binding techniques you can bind certain interrupt vector to certain core (writing to smp_affinity, etc). Thus you can spread interrupts among multiple cores and yet make sure there are no cache misses.

Related posts:

1. Why interrupt affinity with multiple cores is not such a good thing
2. SMP affinity and proper interrupt handling in Linux
3. What’s is direct I/O anyway?