Comments on: Direct IO in Python http://www.alexonlinux.com/direct-io-in-python Fri, 30 Jul 2010 05:08:02 +0000 hourly 1 http://wordpress.org/?v=abc By: Alexander Sandler http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23832 Alexander Sandler Mon, 12 Apr 2010 08:14:58 +0000 http://www.alexonlinux.com/?p=1684#comment-23832 <a href='#comment-23822' rel="nofollow">@eswierk</a> I am not sure os.read does memory allocations. In any case, direct I/O requires aligned memory buffer. Otherwise, any buffer will do.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23832','Alexander Sandler'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23832','Alexander Sandler','<a href=\'#comment-23822\' rel=\"nofollow\">@eswierk<\/a>\r\nI am not sure os.read does memory allocations. In any case, direct I\/O requires aligned memory buffer. Otherwise, any buffer will do.'); return false;">Quote</a></div> @eswierk
I am not sure os.read does memory allocations. In any case, direct I/O requires aligned memory buffer. Otherwise, any buffer will do.
Reply – Quote
]]> By: Kepler Kramer http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23824 Kepler Kramer Mon, 05 Apr 2010 16:44:29 +0000 http://www.alexonlinux.com/?p=1684#comment-23824 I too would like to use Python for O_DIRECT reads and could really use an example similar to your excellent write snippet.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23824','Kepler Kramer'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23824','Kepler Kramer','I too would like to use Python for O_DIRECT reads and could really use an example similar to your excellent write snippet.'); return false;">Quote</a></div> I too would like to use Python for O_DIRECT reads and could really use an example similar to your excellent write snippet.
Reply – Quote
]]> By: eswierk http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23822 eswierk Sat, 03 Apr 2010 04:36:40 +0000 http://www.alexonlinux.com/?p=1684#comment-23822 <a href='#comment-23818' rel="nofollow">@Alexander Sandler</a> - OK, I give up. How exactly does allocating an aligned buffer help you if you're calling os.read, which does its own memory allocation?<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23822','eswierk'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23822','eswierk','<a href=\'#comment-23818\' rel=\"nofollow\">@Alexander Sandler<\/a> - OK, I give up. How exactly does allocating an aligned buffer help you if you\'re calling os.read, which does its own memory allocation?'); return false;">Quote</a></div> @Alexander Sandler – OK, I give up. How exactly does allocating an aligned buffer help you if you’re calling os.read, which does its own memory allocation?
Reply – Quote
]]> By: Alexander Sandler http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23818 Alexander Sandler Wed, 31 Mar 2010 06:36:10 +0000 http://www.alexonlinux.com/?p=1684#comment-23818 <a href='#comment-23816' rel="nofollow">@zls</a> This is exactly the problem this post addresses. Please read it more carefully.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23818','Alexander Sandler'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23818','Alexander Sandler','<a href=\'#comment-23816\' rel=\"nofollow\">@zls<\/a>\r\nThis is exactly the problem this post addresses. Please read it more carefully.'); return false;">Quote</a></div> @zls
This is exactly the problem this post addresses. Please read it more carefully.
Reply – Quote
]]> By: zls http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23816 zls Tue, 30 Mar 2010 15:04:40 +0000 http://www.alexonlinux.com/?p=1684#comment-23816 But there's nothing we can do for os.read where file is opened with flag O_DIRECT. Because system call "read" also needs memory alignment for the read buffer. But the memory is allocated in python without any memory alignment.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23816','zls'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23816','zls','But there\'s nothing we can do for os.read where file is opened with flag O_DIRECT. Because system call \"read\" also needs memory alignment for the read buffer. But the memory is allocated in python without any memory alignment.'); return false;">Quote</a></div> But there’s nothing we can do for os.read where file is opened with flag O_DIRECT. Because system call “read” also needs memory alignment for the read buffer. But the memory is allocated in python without any memory alignment.
Reply – Quote
]]> By: What’s is direct I/O anyway? - Alex on Linux http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23738 What’s is direct I/O anyway? - Alex on Linux Mon, 15 Feb 2010 10:04:03 +0000 http://www.alexonlinux.com/?p=1684#comment-23738 [...] days ago I’ve written a post explaining how to do a direct I/O in Python. But then I thought that it might be a good idea to explain what direct I/O is. So, here we [...]<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23738','What’s is direct I\/O anyway? - Alex on Linux'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23738','What’s is direct I\/O anyway? - Alex on Linux','[...] days ago I’ve written a post explaining how to do a direct I\/O in Python. But then I thought that it might be a good idea to explain what direct I\/O is. So, here we [...]'); return false;">Quote</a></div> [...] days ago I’ve written a post explaining how to do a direct I/O in Python. But then I thought that it might be a good idea to explain what direct I/O is. So, here we [...]
Reply – Quote
]]> By: Alexander Sandler http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23668 Alexander Sandler Sat, 16 Jan 2010 21:14:58 +0000 http://www.alexonlinux.com/?p=1684#comment-23668 <a href='#comment-23646' rel="nofollow">@Ivan Novick</a> Well, first of all, fsync() isn't a solution for the problem because it is not atomic, expensive and unnecessary - operating system should give you a method to disable caching for portion of disk, instead of you trying to simulate it with fsync(). It's simple. Imagine you have a cluster with two machines that processes financial transactions. Imagine that cluster being asked to place 10000$ on your account. Server 1 handles the request and crashes right after it. Moment later, cluster receives a request to transfer 5000$ from your account to some other bank account. The question is how much money remains on your account. This particular application would do it's own caching, as you suggested, but so would many other applications. Don't get me wrong. I am in for caching in OS with all of mine limbs. However, there's must be a way to control it, because there are many applications that need something more delicate then OS caching.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23668','Alexander Sandler'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23668','Alexander Sandler','<a href=\'#comment-23646\' rel=\"nofollow\">@Ivan Novick<\/a>\r\nWell, first of all, fsync() isn\'t a solution for the problem because it is not atomic, expensive and unnecessary - operating system should give you a method to disable caching for portion of disk, instead of you trying to simulate it with fsync().\r\n\r\nIt\'s simple. Imagine you have a cluster with two machines that processes financial transactions. Imagine that cluster being asked to place 10000$ on your account. Server 1 handles the request and crashes right after it. Moment later, cluster receives a request to transfer 5000$ from your account to some other bank account. The question is how much money remains on your account.\r\n\r\nThis particular application would do it\'s own caching, as you suggested, but so would many other applications.\r\n\r\nDon\'t get me wrong. I am in for caching in OS with all of mine limbs. However, there\'s must be a way to control it, because there are many applications that need something more delicate then OS caching.'); return false;">Quote</a></div> @Ivan Novick
Well, first of all, fsync() isn’t a solution for the problem because it is not atomic, expensive and unnecessary – operating system should give you a method to disable caching for portion of disk, instead of you trying to simulate it with fsync().

It’s simple. Imagine you have a cluster with two machines that processes financial transactions. Imagine that cluster being asked to place 10000$ on your account. Server 1 handles the request and crashes right after it. Moment later, cluster receives a request to transfer 5000$ from your account to some other bank account. The question is how much money remains on your account.

This particular application would do it’s own caching, as you suggested, but so would many other applications.

Don’t get me wrong. I am in for caching in OS with all of mine limbs. However, there’s must be a way to control it, because there are many applications that need something more delicate then OS caching.

Reply – Quote
]]> By: Ivan Novick http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23646 Ivan Novick Wed, 13 Jan 2010 15:41:12 +0000 http://www.alexonlinux.com/?p=1684#comment-23646 Making sure your data is on disk is a separate issue from bypassing the filesystem cache. To make sure your data is on disk the system call is fsync. If you don't call fysnc the OS will not harden the data to disk immediately for performance reasons. When you need to know the data is on disk then you make the fsync call. O_DIRECT bypasses the filesystem cache and writes "directly" to disk. This will be a lot slower than writing to filesystem cache so you don't want to use O_DIRECT unless your application is caching data itself. Linus's argument is someone has to cache the data so why not let the OS do it. This is exactly what postgresql does, they rely on using the OS filesystem cache and dont use O_DIRECT.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23646','Ivan Novick'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23646','Ivan Novick','Making sure your data is on disk is a separate issue from bypassing the filesystem cache.\r\n\r\nTo make sure your data is on disk the system call is fsync. If you don\'t call fysnc the OS will not harden the data to disk immediately for performance reasons. When you need to know the data is on disk then you make the fsync call.\r\n\r\nO_DIRECT bypasses the filesystem cache and writes \"directly\" to disk. This will be a lot slower than writing to filesystem cache so you don\'t want to use O_DIRECT unless your application is caching data itself. \r\n\r\nLinus\'s argument is someone has to cache the data so why not let the OS do it. This is exactly what postgresql does, they rely on using the OS filesystem cache and dont use O_DIRECT.'); return false;">Quote</a></div> Making sure your data is on disk is a separate issue from bypassing the filesystem cache.

To make sure your data is on disk the system call is fsync. If you don’t call fysnc the OS will not harden the data to disk immediately for performance reasons. When you need to know the data is on disk then you make the fsync call.

O_DIRECT bypasses the filesystem cache and writes “directly” to disk. This will be a lot slower than writing to filesystem cache so you don’t want to use O_DIRECT unless your application is caching data itself.

Linus’s argument is someone has to cache the data so why not let the OS do it. This is exactly what postgresql does, they rely on using the OS filesystem cache and dont use O_DIRECT.

Reply – Quote
]]> By: Alexander Sandler http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23639 Alexander Sandler Wed, 13 Jan 2010 06:25:29 +0000 http://www.alexonlinux.com/?p=1684#comment-23639 <a href='#comment-23637' rel="nofollow">@Ivan Novick</a> In Linux, when write() finishes, this does not necessarily mean that the data is on media. If you want an example of this behavior, try dding 1MB to floppy disk and see what happens. This is natural behavior considering page cache. On the other hand this is a nightmare for everyone concerned with high availability of data. Why? Because you can't build highly available system knowing that although you think your data is on disk, it may not be there if something bad happens. So, O_DIRECT being used naturally in databases, clustered file-systems, and every other highly available system. As for what Linus said about it... Well, Linus and other kernel developers define what Linux will look like in the future. What Linus say today will become reality in 5-10 years. But it is not a reality yet. In reality, the thread you linked to is three years old and yet POSIX_FADV_NOREUSE is still a no-op in Linux.<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23639','Alexander Sandler'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23639','Alexander Sandler','<a href=\'#comment-23637\' rel=\"nofollow\">@Ivan Novick<\/a>\r\nIn Linux, when write() finishes, this does not necessarily mean that the data is on media. If you want an example of this behavior, try dding 1MB to floppy disk and see what happens.\r\nThis is natural behavior considering page cache. On the other hand this is a nightmare for everyone concerned with high availability of data. Why? Because you can\'t build highly available system knowing that although you think your data is on disk, it may not be there if something bad happens.\r\nSo, O_DIRECT being used naturally in databases, clustered file-systems, and every other highly available system.\r\n\r\nAs for what Linus said about it... Well, Linus and other kernel developers define what Linux will look like in the future. What Linus say today will become reality in 5-10 years. But it is not a reality yet. In reality, the thread you linked to is three years old and yet POSIX_FADV_NOREUSE is still a no-op in Linux.'); return false;">Quote</a></div> @Ivan Novick
In Linux, when write() finishes, this does not necessarily mean that the data is on media. If you want an example of this behavior, try dding 1MB to floppy disk and see what happens.
This is natural behavior considering page cache. On the other hand this is a nightmare for everyone concerned with high availability of data. Why? Because you can’t build highly available system knowing that although you think your data is on disk, it may not be there if something bad happens.
So, O_DIRECT being used naturally in databases, clustered file-systems, and every other highly available system.

As for what Linus said about it… Well, Linus and other kernel developers define what Linux will look like in the future. What Linus say today will become reality in 5-10 years. But it is not a reality yet. In reality, the thread you linked to is three years old and yet POSIX_FADV_NOREUSE is still a no-op in Linux.

Reply – Quote
]]> By: Ivan Novick http://www.alexonlinux.com/direct-io-in-python/comment-page-1#comment-23637 Ivan Novick Tue, 12 Jan 2010 13:36:17 +0000 http://www.alexonlinux.com/?p=1684#comment-23637 The real question is why would you want to do this? I would like to hear a use case for using direct IO in python. Databases are the best use case for direct IO and even some of them don't use it (see postgresql). You may also find this posting by Linus regarding direct io useful. http://lkml.org/lkml/2007/1/10/233 Cheers, Ivan<div class="comment-remix-meta"><a href="#" class="replyto" onclick="replyto('23637','Ivan Novick'); return false;">Reply</a> - <a href="#" class="quote" onclick="quote('23637','Ivan Novick','The real question is why would you want to do this? I would like to hear a use case for using direct IO in python.\r\n\r\nDatabases are the best use case for direct IO and even some of them don\'t use it (see postgresql).\r\n\r\nYou may also find this posting by Linus regarding direct io useful.\r\n\r\nhttp:\/\/lkml.org\/lkml\/2007\/1\/10\/233\r\n\r\nCheers,\r\nIvan'); return false;">Quote</a></div> The real question is why would you want to do this? I would like to hear a use case for using direct IO in python.

Databases are the best use case for direct IO and even some of them don’t use it (see postgresql).

You may also find this posting by Linus regarding direct io useful.

http://lkml.org/lkml/2007/1/10/233

Cheers,
Ivan

Reply – Quote
]]>