Version control that I did try is git. This is a very popular version control system and for a good reason. Comparing git and subversion brought me to a conclusion that git is really a very consequence of how things work in the world. Thing about git is that it is a distributed version control while subversion is not. You can make a distributed version of subversion, but it won’t be subversion anymore.

Distributivity is one thing, but there are more. Take the standard trunk/branches/tags layout that you have to create in subversion – version control could do it for you, as git/bazaar/mercurial do. At first, after working with CVS for some time, having an option to have non-standard layout seemed cool for some time. But then it appeared completely useless.

This brings a notion of evolution into version control systems and I bet there’s similar process with programming languages. I think we can already starting drawing a programming language that will replace Python.

Yep, Python isn’t perfect. I thought I’d compile a list of things that shall be different in ought to be programming language that comes after Python.

1. Passing self to methods is counter-productive. Using self to access members is super counter productive. Yet this is how Python works. Without this, it won’t be Python anymore.
2. Same thing with global. Every program has this one global variable that every other class and function use. So, every entity that uses that variable has to declare it with global keyword. Utterly counter-productive.
3. Starting every private method and field with __ (double under-score) is ugly. The way it changes method name prepending it with a class name is even uglier. It is a really hackish way to handle this problem. I understand that this is the Python way of doing things, but it feels wrong.
4. Another hackish thing are decorators. Make an option to have static methods, class methods and properties and decorators are gone. No one would need them anymore, and it will be one bit easier to master Python.
5. You name it.

Disclaimer (can’t get along without one this time ): This post may seem like one criticizing Python and it does in some ways, but I doubt these can be fixed in something called Python (unless Guido van Rossum brews it and calls it Python 4). In the meantime I love Python and use it on a daily basis.

Related posts:

1. Distributed vs. centralized version control systems
2. Bazaar for subversion users, part 1 – the basics
3. Todo list

Today I’d like to talk about one nice trick that I learned few days ago.

When working with large software systems, memory management becomes an imperative. In C, you can easily allocate a large chunk of memory and allocate structure right on that buffer. This is by far more difficult in C++, because compiler has to call consturctor.

Apparently, you can, in a way, directly call object’s constructor . I.e. you can allocate an object, on specified memory region, without actually allocating this region.

This is how you do it.

char* s = new char[1024];

SomeClass* p = new (s) SomeClass;

First new operator just allocates 1024 bytes. This is good old allocation as we know it. Note the special syntax of the second new operator. It allocates the new object on memory specified in brackets. Basically, this calls SomeClass’s constructor using s as storage.

One thing that I don’t know how to do is how to call destructor on the object – i.e. how to delete an object in place.

Related posts:

1. Direct IO in Python
2. One sane voice in the crowd
3. Recursively deleting symbolic link and what it points to

I think this is going to become my new icon. What do you think?

No related posts.

At first, I thought less reads entire input first. This would make standard input stream free to process key presses from the user. So I decided to check this out. I created a 1Gb long text file and ran less on it. I expected less to take some time to show file contents, but it showed first lines of the file instantly. Also, it didn’t consume 1Gb of RAM as I expected.

The conclusion is obvious. less does not read entire input buffer before letting user to interact with itself. Then how?

Here’s what less does. It separates input file stream from user input stream. Both inputs initially come via standard input stream, so less separates between them. First it duplicates the standard input stream. This allocates a new file descriptor. Then it closes old file descriptor, freeing file descriptor 0 for use. Then it opens /dev/tty. When opening a file, Linux uses next available file descriptor. File descriptor used as standard input is 0, so when less opens /dev/tty again, file descriptor of the newly opened file has value 0.

Eventually, it ends up with the new input coming via standard input stream file descriptor (0), and old input still available via file descriptor that it duplicated in the beginning. It reads the input file from the duplicated file descriptor, and uses curses on standard input stream.

You may be wondering what is /dev/tty and what it has to do with standard input streams. This is really fascinating stuff.

As you know, in Linux, everything is a file. So is terminal. Linux uses device files to represent various system devices. /dev/tty is a file that represents terminal of the current process. When process reads from /dev/tty it becomes its input. When program writes to /dev/tty it becomes its standard output or standard error stream.

So, when less reopens /dev/tty, it actually recreates standard input. Older descriptor, one that has been duplicated, can no longer accept new input, but less still can use it to read what it has been written into it already.

Related posts:

1. Creating new application on top of SSH
2. sed – the missing manual
3. Direct IO in Python

So, I decided to leave Fabrix and as of last week I am software engineer and senior consultant at Dell IDC (Israel Developer Center).

Related posts:

1. Resume
2. I am looking for a new job
3. What it takes to be a QA engineer

As surprising as it is, when you write some information to the disk, it doesn’t get there immediately. In Linux especially, the kernel tries to cache write requests in memory as much as it can. This means that in addition to writing the data to the disk, kernel keeps it in memory. Consecutive read request to the same place on the disk will be much faster because there’s no need to read the information from slow disk – it is already in memory. On the other hand, the information goes to the hard-disk only after some period of time (short though) or when the system runs out of memory. In the meantime, Linux reports that data has been written, despite it is not yet on the disk.

This causes several interesting phenomena that you may have noticed. For example Linux machines with little memory, even though not all memory is in use, do use disk much more than machines with more memory. There are several reasons for it. One of them is because kernel doesn’t have enough memory to cache information from the disks and as a result read requests rarely hit the cache and more often go to the disk.

Want to see exactly how Linux reports successful write requests before data actually lands on the disk? Try writing some information to floppy disk and see how fast it is in Linux. The truth is that it’s still frustratingly slow. Linux just makes it look like a fast device.

Sheer thought that Linux doesn’t write the data to the disk, despite it says it did, may be pretty scary. But it shouldn’t really. First of all, if its not very busy it does write the data to the disk as soon as possible. Second, Linux does excellent job avoiding various problems and even if something bad happens, it is pretty good at recovering lost data. The way Linux works is excellent for 99% of users. This approach improves performance in various ways and makes the system more healthy and stable.

However, some folks out there are not happy with this situation. Some software systems cannot work the way Linux works. One of the examples are so called clustered file-systems – file-systems that are spread among multiple servers for redundancy purposes. Such systems need a way to know that data has been written to the disk for real, not just being cached. Also, such systems want to make sure that reads hit disk and not OS cache.

This is where direct I/O becomes handy. Direct I/O is a way to avoid entire caching layer in the kernel and send the I/O directly to the disk. Overall, this makes I/O slower and does not let Linux do various optimizations that it usually does. Also, it introduces some constraints on memory buffer that being used for the I/O. Yet sometimes it is inevitable.

Want to try it yourself? dd, I/O Swiss army knife, has an option called direct. It tells dd to do direct I/O instead of regular I/O. Another option for doing direct I/O is writing your own program that opens files with O_DIRECT flag (see open(2) for details).

Update Feb. 12: Thanks to Ivan for noting the difference between synchronous I/O and direct I/O. I updated the post to reflect the difference.

Related posts:

1. Swap vs. no swap
2. SMP affinity and proper interrupt handling in Linux
3. Aligned vs. unaligned memory access

One problem that you should take care of when tackling this problem, is that symbolic link can point to a symbolic link. Then symbolic link should also point to symbolic link. And once again, and again and again…

So what can we do about it? First, lets start with lstat() system all. It will tell us is the file we’re interested in, is actually a symbolic link. To be more precise, st_mode field in struct stat will have flag S_IFLNK if specified file is a symbolic link. Note that it should be lstat() and not stat() – the later will return information about file the link points to and not about the link.

Next step is to call readlink(). This system call returns name of the file pointed to by specified symbolic link.

Finally, you should repeat the process recursively, for the pointed to file. Here is the code that does it:

int recursive_deleter(const char* filename)
{
  struct stat st;
  char buffer[1024];

  if (lstat(filename, &st)) {
    perror("stat");
    return -1;
  }     

  if (st.st_mode & S_IFLNK == S_IFLNK) {
    memset(buffer, 0, sizeof(buffer));
    if (readlink(filename, buffer, sizeof(buffer)) < 0) {
      perror("readlink");
      return -1;
    }           

    printf("File %s is a symbolic link to %s\n", filename, buffer);

    if (recursive_deleter(buffer))
      return 0;
  }     

  printf("Deleting %s\n", filename);

  if (unlink(filename)) {
    perror("unlink");
    return -1;
  }     

  return 0;
}

Have fun!

Related posts:

1. Hex dump functions
2. Direct IO in Python
3. How less processes its input

When its not enough, you can always upgrade to open() and close() from os module. Opening man page on open() (the system call – open(2)) reveals all those O_something options that you can pass to os.open(). But not all of them can be used in Python. For example, if you open a file with O_DIRECT and then try to write to it, you will end up with some strange error message.

>>> import os
>>> f = os.open('file', os.O_CREAT | os.O_TRUNC | os.O_DIRECT | os.O_RDWR)
>>> s = ' ' * 1024
>>> os.write(f, s)
Traceback (most recent call last):
  File "", line 1, in
OSError: [Errno 22] Invalid argument
>>>

Invalid argument?. What invalid argument? Hey there’s nothing wrong with those arguments…

Reading open(2) man page further reveals that working with O_DIRECT requires that all buffers used for I/O should be aligned to 512 byte boundary. But how can you have a memory buffer aligned to 512 bytes in Python?

Apparently, there’s a way. Python comes with a module called mmap. mmap() is a system call that allows one to map portion of file into memory. All writes to memory mapped file, go directly to file despite it looks like you’re working with plain memory buffer. Same with reads.

There’s one interesting thing about mmap. It works with granularity of one memory page – 4kb that is. So every memory mapped buffer is naturally memory aligned to 4kb, thus to 512 byte boundary too. But hey, shouldn’t mmap map files?

Well, apparently mmap can be used for memory allocations. I.e. specifying -1 as file descriptor does just that – allocates RAM, as much as you tell it. So, this is what we do:

>>> import os
>>> import mmap
>>>
>>> f = os.open('file', os.O_CREAT | os.O_DIRECT | os.O_TRUNC | os.O_RDWR)
>>> m = mmap.mmap(-1, 1024 * 1024)
>>> s = ' ' * 1024 * 1024
>>>
>>> m.write(s)
>>> os.write(f, m)
1048576
>>> os.close(f)
>>>

Note that mmap memory buffer object behaves like a file. I.e. you can write into the buffer and read from it – like I do in line 8. More on it in official documentation.

Have fun!

Related posts:

1. What is direct I/O anyway?
2. Call a constructor or allocate an object in-place
3. Recursively deleting symbolic link and what it points to

I am joining as a chief video content consumer senior software engineer.

Related posts:

1. Resume
2. About

Anyway, here’s something else that I learned during one of the interviews.

I was asked to implement insertion into a linked list in C. Each list node has a number (num). Numbers in the list should be sorted in ascending order – node with lowest number should be first. Here is my naive implementation.

void list_insert(int num)
{
        list_node_t *n, *tmp, *prev;

        n = allocate_node();
        n->num = num;

        if (!head) {
                head = n;
                n->next = NULL;
        } else {
                tmp = head;
                prev = NULL;
                while (tmp) {
                        if (tmp->num >= n->num) {
                                if (prev == NULL) {
                                        head->next = n;
                                        n->next = NULL;
                                } else {
                                        n->next = tmp;
                                        prev->next = n;
                                }

                                break;
                        }

                        prev = tmp;
                        tmp = tmp->next;
                }

                if (!tmp) {
                        n->next = NULL;
                        prev->next = n;
                }
        }
}

Yeah, well it is pretty long, but that’s how you do it, right? You have to make sure that the list is not empty (line 8 ) – if it is empty, then new node should be placed at the head of the list. Then we have to find right place for the new node and insert it into its position (lines 14-29). Finally, if we didn’t find a good position for the new node, we should put the node at the end of the list (lines 31-34).

When I showed this to the examiner he asked me to rethink the routine. In particular he didn’t like number of if statements. After an hour of discussing various optimizations, this is what we came up with.

void list_insert(int num)
{
        list_node_t **tmp, *n;

        n = allocate_node();
        n->num = num;

        tmp = &head;
        while (*tmp) {
                if ((*tmp)->num >= n->num)
                        break;
                tmp = &(*tmp)->next;
        }

/*
        (*tmp) = n;
       n->next = (*tmp)->next;
*/

/*
        The code in the comment above is broken. Thanks to jagan
        for pointing this out. Here's the right code.
*/
        n->next = *tmp;
        *tmp = n;
}

There are several things that make this version better.

First of all, instead of having a pointer to current node (tmp) and previous node (prev), we have one pointer that corresponds to pointer to previous node. This way we can always get to the current node via next field in the previous node. Thus we don’t need two pointers anymore.

Also, this version uses pointer to pointer to node to traverse through the list. We don’t insert the node in the while loop anymore. Purpose of the while loop is to find right place for the new node. This spares if statement that tests if the list is empty and if statement in the loop.

Finally, once tmp points to the right place, we insert the node.

Second version runs faster too. I did a small program that tests both versions and it appears that the later version is something like 5% faster.

Related posts:

1. pthread mutex vs pthread spinlock
2. Todo list