Follow Techotopia on Twitter

On-line Guides
All Guides
eBook Store
iOS / Android
Linux for Beginners
Office Productivity
Linux Installation
Linux Security
Linux Utilities
Linux Virtualization
Linux Kernel
System/Network Admin
Programming
Scripting Languages
Development Tools
Web Development
GUI Toolkits/Desktop
Databases
Mail Systems
openSolaris
Eclipse Documentation
Techotopia.com
Virtuatopia.com
Answertopia.com

How To Guides
Virtualization
General System Admin
Linux Security
Linux Filesystems
Web Servers
Graphics & Desktop
PC Hardware
Windows
Problem Solutions
Privacy Policy

  




 

 

Samba HowTo Guide
Prev Home Next

Chapter32.Handling Large Directories

Jeremy Allison

Samba Team

John H. Terpstra

Samba Team

March 5, 2005

Samba-3.0.12 and later implements a solution for sites that have experienced performance degradation due to the problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory.

The key was fixing the directory handling to read only the current list requested instead of the old (up to samba-3.0.11) behavior of reading the entire directory into memory before doling out names. Normally this would have broken OS/2 applications, which have very strange delete semantics, but by stealing logic from Samba4 (thanks, Tridge), the current code in 3.0.12 handles this correctly.

To set up an application that needs large numbers of files per directory in a way that does not damage performance unduly, follow these steps:

First, you need to canonicalize all the files in the directory to have one case, upper or lower take your pick (I chose upper because all my files were already uppercase names). Then set up a new custom share for the application as follows:

[bigshare]
path = /data/manyfilesdir
read only = no
case sensitive = True
default case = upper
preserve case = no
short preserve case = no

Of course, use your own path and settings, but set the case options to match the case of all the files in your directory. The path should point at the large directory needed for the application any new files created in there and in any paths under it will be forced by smbd into uppercase, but smbd will no longer have to scan the directory for names: it knows that if a file does not exist in uppercase, then it doesn't exist at all.

The secret to this is really in the case sensitive = True line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file called FOO, and it cannot be found by a simple stat call, then smbd will return file not found immediately without scanning the containing directory for a version of a different case. The other xxx case xxx lines make this work by forcing a consistent case on all files created by smbd.

Remember, all files and directories under the path directory must be in uppercase with this smb.conf stanza because smbd will not be able to find lowercase filenames with these settings. Also note that this is done on a per-share basis, allowing this parameter to be set only for a share servicing an application with this problematic behavior (using large numbers of entries in a directory) the rest of your smbd shares don't need to be affected.

This makes smbd much faster when dealing with large directories. My test case has over 100,000 files, and smbd now deals with this very efficiently.

Samba HowTo Guide
Prev Home Next

 
 
  Published under the terms fo the GNU General Public License Design by Interspire