Skip to content

largefile is not just for large files

December 19, 2006

Way back when Solaris 2.6, was released we got “large file” support for 32 bit applications. Where to quote the largefile(5) manual page large file is:

A large file is a regular file whose size is greater than or equal to 2 Gbyte ( 2^31 bytes).

You had to recompile them and could optionally modify the code to use the xxx64 calls (open64, stat64 etc, see lf64(5) & lfcompile64(5)) or just compile with the flags that were returned by “getconf LFS_CFLAGS” eg:

: principia IA 41 $; getconf LFS_CFLAGS -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 : principia IA 42 $; 

Since the number of applications that need to support large files is small once they were converted interest was limited.

However last week a customer stumbled on another case where large file support is required. The clue is buried in the stat(2) manual page:

EOVERFLOW The file size in bytes or the number of blocks allocated to the file or the file serial number cannot be represented correctly in the struct pointed to by buf.

In the old stat buffer the st_ino element, which contains the file serial number was 32 bit, in the stat64 structure it is 64 bit.

The customer was seeing problems with a application that could not use files which were served by a particular vendors NAS box. The reason was that that NAS box decided, perfectly legally, that it would have 64bit serial numbers. Since UFS only has a 32 bit inode and therefore serial number it never delivered a serial number that could not fit in the old stat(2) buf so we rarely, if ever, saw the problem.

With the advent of ZFS which most certainly can have 64 bit serial numbers the possibility of having files with 64bit serial numbers is closer to prime time. However since ZFS does not use large serial numbers until it has to and the serial numbers are per file system not per pool it should not be hard to avoid. However if your NAS appliance is not so friendly compiling with large file support will aid application portability.

So if you have an application that you wish to work no matter what the constraints of the file system on which it is stored make sure you are building it either 64 bit or with largefile support.

It is tempting to see what would break if you home directory was on a system that was only handing out files with large serial numbers. Need to get a bigger file server for that though.

Tags: topic:[ZFS] topic:[filesystems] topic:[solaris] topic:[largefile]

From → Solaris

One Comment
  1. It is even worse than you say. Although the stat man page doesn’t mention it, this also applies to the st_rdev field. This is really bad because the overflow can be induced in *other* processes.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: