Sometimes what a person really wants is a LACK of size

by Michael S. Kaplan, published on 2006/12/13 04:46 -08:00, original URI: http://blogs.msdn.com/michkap/archive/2006/12/13/1275292.aspx

Over the time I have had this blog, I have often had occasion to say nice things about work that the Shell team does. One of the main reasons for this is a particular characteristic that its members tend to have, one where if there is not an easy way to do something, they don't just complain about the lack. They write the code to get the job done.

Now sometimes this doesn't work out as well, like in the tons of shlwapi functions that do string comparison, which just leads to confusion as to which function to use. But that's not what am I going to talk about today; this is a "glass half full" post. :-)

For many years, the MAX_PATH constant of 260 characters has been getting to be progressively more annoying. As hard drives get bigger and therefore can contain more and more files, it has become easier and easier to run into real problems where you can create files in adirectory that can no longer be easily accessed due to a path that is to put it simply too freaking long (where too freaking long is defined as 260 characters!).

This is a problem that can get bad faster in international environments due to path names being even longer in languages that tend to have longer words. But in truth no one is immune to the problem.

Historically, no one has been willing to do anything about it since obviously solving the problem in one component can simply expose it in another, and in the end no one actually gets a fix to the problem. The sheer scope of it keeps people away.

Now first they have some obvious fixes, like using shorter special directory names. I mean \Documents and Settings becoming \Users and getting rid of the zillion My prefixes snd so on was a positive step. A small one, but how many times has the problem been trying to access paths starting with \Documents and Settings\All Users\Application Data\Microsoft when \Users\All Users\Microsoft would have done?

Ok, that is nice but really that is just them trying to fix a problem that they did most of the work in initially causing. So kudos for the reversal, but why was I claiming this was going to be a "glass half full" post?

Well, it is actually one of the efforts that took place in Vista, a feature whose official name I am not entirely sure about but which can be best described as "auto path shrinking."

The idea is simple enough -- when a path turns out to be too long, start shrinking individual elements in the path one by one until it is short enough to fix within MAX_PATH.

Now this is obviously cooler and much more usable than a call to GetShortPathName, which will just shrink the whole thing and make the filename less usable. By keeping as much of the path long as they can (especially the file name) that big problem with nested paths with too many long directories is easily solved (it does not take too many subdirectories with GUID names to hit a MAX_PATH problem!).

I have never seen anyone else suggest this as a possible mitigation for the whole class of MAX_PATH problems that it helps to solve, and it is work that they put into the bulk of the Shell functions dealing with paths. While obviously not solving every instance of the problem, it at least proves that there are ways to try to make things work.

Anyway, just thought I'd take a moment to say kudos to the Shell team for being a group to actually take the first positive steps I have heard about in a long time to try to help out with the MAX_PATH problem. Thanks!

To take advantage of auto path shrinking, you do need to not turn off short name generation. So start spreading the word that NtfsDisable8dot3NameCreation may not always be in the best interests of people, no matter what you may hear elsewhere. Just something to consider....

There are many applications (both from MS and from third parties) that can and in some cases will break if the limit were arbitrarily raised, duet o the longstanding asumption. It is not simply updating a constant; it is updating a piece of the industry....

Sorry I wasn't clear, I understand why giving a filename to an application, like via WIN32_FIND_DATA, would need to still be MAX_PATH (unless you go through all the fuss of making a new structure and function calls for "long" filenames--is this what FINDEX_INFO_LEVELS is supposed to be for? (I only show one possible value)).

I was talking about when you pass a string to windows. The Docs for CreateFile, for example, state that the A version can have no more than MAX_PATH, and the W version can have up to 37 thousand if it starts with "\\?\". Why does the API care?

Then the question I have is: Why can't they make it dynamic? How would this impact applications? Isn't the static buffer on the DLL side, where it theoretically should be inaccessible by applications? Is it just because the A version is no longer "supported"?

It is because we are talking about thousands of functions and an implied chain of behavior stretching over a decade, behavior that there is a dependency on.

And external code often has the same kind of dependency created as well.

Now this is just the tip of the iceberg, but there is no way that is has ever been considered to be a simple issue -- a global change along the lines you are thinking of here would take man years of effort and would essentially break a ton of applications and code, not to mention leading to countless security holes with people overrunning buffers that are suddenly made too small in every shipping version of Windows....

I'm fairly sure there's a part of MSDN Library which mentions that it's 3 for the drive specification (e.g. "C:\"), 256 for the path, and 1 for the terminating NULL = 260 total.

A slight difference with the \\?\ syntax is that these can only be absolute paths, not relative ones. Regular paths (not starting with \\?\) are relative on desktop Windows; coming from Windows CE I'm used to a lack of support for relative paths (all paths are absolute on CE, so if you say CreateFile( L"Blah.txt" ) your file ends up in the root - there is no SetCurrentDirectory API).

Really \\?\ is just an escape to the Win32 API to tell it to pass the path unchanged to the underlying NT native API (e.g. NtCreateFile), rather than prepend the supplied path with the current directory (the native API only works with absolute paths, the current directory is a Win32 concept).

Clearly the absolute path to a file can already exceed MAX_PATH characters, because of these two features (you can pass a MAX_PATH relative filename to CreateFile when the current directory is not a root path). The question is how to work around the problem. It was done in the move from Win16 to Win32, but of course people had to recompile their programs for that, and the backcompat scheme of generating the 8.3 name was retained.

I say break all those apps (how do we know it'll break a lot of apps, isn't that an assumption in itself?). Seriously. I'm so tired of this limitation, it's breaking stuff RIGHT NOW. It's really inexcusable. Plenty of other OSes do not have problems with this. Make a break for it!

Or at the very least introduce an undocumented option for experts!

Will we be sitting here in 5 years' time still agonizing about the 260 char path limit? 10 years? 20 years?

Gah.

Microsoft has AppCompat labs containing many thousands of apps written by Microsoft and other companies, so we do not ever have to guess that things might break in situation -- we know when they might do so.

If a new version "fixed" the limitation and those apps would fail or introduce security holes by overrunning buffers, it is pretty certain that Microsoft would be blamed for boith the breaks and the security holes.

Working and safe is a more palatable option for all concerned....