Filesystems: Difference between revisions

From CCP4 wiki
Jump to navigation Jump to search
(New page: Actually, ZFS is available on Linux too as a user space filesystem, and Sun are considering a kernel port: http://www.wizy.org/wiki/ZFS_on_FUSE However, I'm inclined to wait for btrfs (bu...)
 
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
Actually, ZFS is available on Linux too as a user space filesystem, and Sun are considering a kernel port:
This page serves to give some information on possible filesystems for Linux machines, in particular those that may be used for very large filesystems (many TB). The list is sorted by age!
http://www.wizy.org/wiki/ZFS_on_FUSE


However, I'm inclined to wait for btrfs (butter-fs). Here's a review of btrfs from an ex-ZFS engineer:
== XFS ==
http://lwn.net/Articles/342892/
* SGI's filesystem (1994) that was ported to Linux around 2001
* no 16 TB limitation; max size is 8 Eib (1 exbibyte = 2^60 bytes)
* no data compression and checksumming (data-integrity) features. It does have metadata checksums.
* the default for RHEL starting at version 7


Oracle are working on a new generation NFS replacement designed specifically to benefit from some of the btrfs features:
== ZFS ==
http://oss.oracle.com/projects/crfs/
* Sun's filesystem which is natively available on Solaris (since 2005)) and OpenSolaris (both of which are freely available)
* no 16 TB limitation; max size is 2^128 bytes
* (optional) transparent (gzip) compression (LZO compression available as patch)
* atomic updates - means that the on-disk state is consistent at all times, there's no need to perform a lengthy filesystem check after forced reboots/power failures
* background checksumming and self-healing for data integrity
* Built-in stripes (RAID-0), mirrors (RAID-1) and RAID-Z (it's like software RAID-5, but more efficient due to ZFS's copy-on-write transactional model)
* triple parity raid (raidz3) has been added to ZFS [http://blogs.sun.com/ahl/entry/triple_parity_raid_z]
* there is a port of ZFS to FreeBSD
* the source code license of ZFS is incompatible with the GPL of the Linux kernel, therefore not in the latter
* install from http://zfsonlinux.org/


Here's an article one of the truely astonishing btrfs features: You can upgrade an existing linux file system to btrfs without destroying the existing fs or duplicating the data!
== Ext4 ==
http://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 well, what are the alternatives?
* ext4 is based on ext3 (same developers)
* default of some Linux distros
* no 16 TB limitation; max size is 1 EiB (1 exbibyte = 2^60 bytes)
* no compression and checksumming (data-integrity) features


- ext3 does not support more than 16GB,
== btrfs ==
- ext4 does but is too young and lacks the features of zfs and btrfs
* latest addition to native Linux filesystems; considered stable since 2.6.31 kernel (2009; see http://www.h-online.com/open/Kernel-Log-2-6-31-Tracking--/features/113671)
- xfs also does but lacks the features of zfs and btrfs, and requires at least a good UPS. Bad XFS crashes are not unheard of.
* no 16 TB limitation; max size is 16 Eib (1 exbibyte = 2^60 bytes)
- don't know about jfs
* (optional) transparent (zlib) compression
 
* checksumming for data integrity is implemented
All four do not support compression.
* a review of btrfs from an ex-ZFS engineer: http://lwn.net/Articles/342892/
 
* you can upgrade an existing linux file system to btrfs without destroying the existing fs or duplicating the data: http://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3
It should be clear that a 100 TB filesystem requires some kind of backup. And tape backup itself is tricky and costly to get right.
* http://www.h-online.com/open/The-Btrfs-file-system--/features/113738
 
Currently, I would setup a Solaris fileserver with ZFS because ZFS is mature and the filesystem can be read by FreeBSD and Linux. A few months ago I tried an OpenSolaris installation CD and it worked like a charm.
 
One possibility in Linux is ZFS over FUSE; this has a large number of advantages over other filesystems (except Btrfs) - see http://www.linux-magazine.com/w3/issue/103/ZFS.pdf . The article explains installation for Ubuntu. I must admit that I did not try it so far.
 
The alternative would be Btrfs, see http://www.h-online.com/open/The-Btrfs-file-system--/features/113738 . This is available for latest Fedora and Ubuntu, is part of the recently released 2.6.31 kernel (http://www.h-online.com/open/Kernel-Log-2-6-31-Tracking--/features/113671), and will therefore in the future be available in all distros.

Latest revision as of 22:23, 28 November 2016

This page serves to give some information on possible filesystems for Linux machines, in particular those that may be used for very large filesystems (many TB). The list is sorted by age!

XFS[edit | edit source]

  • SGI's filesystem (1994) that was ported to Linux around 2001
  • no 16 TB limitation; max size is 8 Eib (1 exbibyte = 2^60 bytes)
  • no data compression and checksumming (data-integrity) features. It does have metadata checksums.
  • the default for RHEL starting at version 7

ZFS[edit | edit source]

  • Sun's filesystem which is natively available on Solaris (since 2005)) and OpenSolaris (both of which are freely available)
  • no 16 TB limitation; max size is 2^128 bytes
  • (optional) transparent (gzip) compression (LZO compression available as patch)
  • atomic updates - means that the on-disk state is consistent at all times, there's no need to perform a lengthy filesystem check after forced reboots/power failures
  • background checksumming and self-healing for data integrity
  • Built-in stripes (RAID-0), mirrors (RAID-1) and RAID-Z (it's like software RAID-5, but more efficient due to ZFS's copy-on-write transactional model)
  • triple parity raid (raidz3) has been added to ZFS [1]
  • there is a port of ZFS to FreeBSD
  • the source code license of ZFS is incompatible with the GPL of the Linux kernel, therefore not in the latter
  • install from http://zfsonlinux.org/

Ext4[edit | edit source]

  • ext4 is based on ext3 (same developers)
  • default of some Linux distros
  • no 16 TB limitation; max size is 1 EiB (1 exbibyte = 2^60 bytes)
  • no compression and checksumming (data-integrity) features

btrfs[edit | edit source]

Retrieved from "https://wiki.uni-konstanz.de/ccp4/index.php?title=Filesystems&oldid=2428"