No upproblems with $UpCase when you do a upVista upgrade

by Michael S. Kaplan, published on 2007/12/27 10:16 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2007/12/27/6875879.aspx

One of the interesting consequences I have noted when it comes to dealing with having a Blog with >2200 blogs in it, such as what happens when somebody comes across the Blog as it is and start randomly reading posts from it.

When combined with the fact that I no longer tend to shut down comments in damn near any of them, another interesting consequence emerges: some people will start randomly commenting in posts, treating the ones from three years ago no differently than the ones written three days ago. :-)

One such person is Tanveer Badar, who has been commenting in many different posts recently....

Though one fact became obvious -- no response was usually expected, since when a response was desired, the question shows up in the Suggestion Box:

It is something I covered a little bit recently in In Case you have problems that you might think are ǸȦȘȚȲ, where admittedly the issue is smaller since a jump drive is a bit easier to deal with than a full partition, especially the main partition of the Windows install potentially being upgraded. :-)

Basically, since case is preserved but not changed, the definition of case changing is not an issue as long as you always ask for items using the proper case, or if nothing else you are willing to live with the file you get when you have more than one available and you don't ask for the right case potentially getting the "wrong" item back....

Now with that said, there is an interesting side effect that is covered in the following article from the MS knowledge base:

Basically, if you dual boot between an XP and a Vista partition you will occasionally see chkdsk being run at boot time (you can also request this be done if you request that chkdsk be run and it cannot when you ask due to the drive being locked).

During that run, you will get the message Correcting errors in the uppercase file when that run happens on XP at boot time, even if you do not pass the /F flag to "fix" errors. If you do pass the /F flag then the drive will be "fixed" by which I mean the UpCase table in the drive will be updated (or in this case downdated) to the XP table, though there is no problem with files on the drive violating those rules as long as functions dealing with the files preserve case (and you don't try to move or copy files that would conflict on the tables update/downdated!)....

The error does not happen when you run in Vista since the updated chkdsk does not treat the situation as an error and thus does not report it as such. There was originally going to be an update to the older versions of chkdsk,exe but that update was not triaged as being important enough (since other than the possibly mildly unclear error message there is really no error here).

So in the end whether to do the update is up to you and there is no need to reformat or even change anything....

Hi, perhaps I should prefix my name with "Not the random

reader”. : -)

The reason you are seeing comments on old posts is because I always read a blog back to front when I find one that is interesting. I must say that I am having trouble keeping up with the backlog of all those posts.

Hopefully, I will catch up with your (exceedingly high) posting rate in the next month and you won’t see any more “some people will start randomly commenting in posts, treating the ones from three years ago no differently than the ones written three days ago".

There is only one me!

I've always thought the NTFS $UpCase mechanism was a very smart bit of forethought in the design. Since it is essentially a database, and there is a specific set of conversion semantics that are required for operational consistency, it makes sense to store those semantics in the database itself. Protection from external changes.

I don't understand the chkdsk thing, though. What is modifying the $UpCase table in the first place, that requires chkdsk to fix it?

Preserving case in requests might actually be harder than it seems. You can pass FILE_FLAG_POSIX_SEMANTICS to things like CreateFile(), or use the Nt* functions directly without passing the case-insensitive flag, but newer versions of Windows have the object manager force case-insensitivity by default, and there's nothing you can do at an API level to override that. (Unless you're in-kernel, like a driver.)

If you end up with two directories having the same name that differs only in case, you'll have an amazingly hard time getting into the non-favored one via normal means, even if you preserve the case in your request. (Differs according to $UpCase, not NLS.)

It's difficult to see a situation where this would come up in normal usage, but just the idea that chkdsk and something else are making changes to $UpCase raises a red flag for me...

Yeah, I probably should have mentioned, it's a simple registry key to change that behavior. There's mention of it in http://support.microsoft.com/kb/929110

The problem here though is that it's an administrative setting, not a programmatic one.

I missed it last time around, but KB940830 says the chkdsk thing happens when you run an earlier chkdsk (XP/2003) on an NTFS volume formatted by a later OS (Vista). It doesn't mention whether Vista's chkdsk does the same thing though.

In any case, the problem scenario I was thinking of should only occur if you put conflicting files on the drive using XP rules, $UpCase is changed to use Vista rules, and then try to access them using the new rules -- similar to the post about the experiment with the jump drive. If Vista's chdksk doesn't screw with $UpCase the way XP's does, this would probably never happen.

I don't think we're talking about the same scenario. Consider a situation where XP says "Filename" and "filename" are different, because it does not know about case differences for the letter 'F'. So, under XP, you can create both "Filename" and "filename" in the same directory, perhaps by accident, with no problems.

Now Vista comes along and adds a table entry that says 'F' and 'f' are the same if you ignore case. Both files still exist in that directory, but given the default object manager settings, it is impossible for you to access one of those files via normal APIs.

Sample program with many assumptions and a horrid lack of error checking:

------------------------------------
#include <stdio.h>
#include <windows.h>
int main(void) {
HANDLE fh;
DWORD count;
char buffer[32] = {0};
fh = CreateFile("FILENAME.txt", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_FLAG_POSIX_SEMANTICS, NULL);
WriteFile(fh, "UPPERCASE FILE CONTENTS", 23, &count, NULL);
CloseHandle(fh);

fh = CreateFile("filename.txt", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_FLAG_POSIX_SEMANTICS, NULL);
WriteFile(fh, "lowercase file contents", 23, &count, NULL);
CloseHandle(fh);

printf("files created\n");
getchar();
printf("opening uppercase filename... ");

fh = CreateFile("FILENAME.txt", GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);
ReadFile(fh, buffer, sizeof(buffer), &count, NULL);
printf("got %s\n", buffer);
CloseHandle(fh);
printf("opening lowercase filename... ");
fh = CreateFile("filename.txt", GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);
ReadFile(fh, buffer, sizeof(buffer), &count, NULL);
printf("got %s\n", buffer);
CloseHandle(fh);
return 0;
}

------------------------------------

When I test that on a 2003 box with the registry key set appropriately per KB929110 so I can distinguish case, both files are successfully created. However, it only reads the contents of FILENAME.txt, despite the fact that I explicitly asked for the other file in the second request.

That's the problem I'm talking about. IOW, a lookup in case-insensitive mode finds first match, not best match.

(On the same box I could just use FILE_FLAG_POSIX_SEMANTICS to access the other file anyway. On a box without the object manager change, and without administrative access to change the setting, I'm SOL.)

Since it sounds like Vista and XP will keep changing $UpCase back and forth in a dual-boot situation, this kind of thing could happen by accident (although it still seems rather unlikely). The correct fix, AFAICT, would be to have chkdsk stop messing with the table. OTOH, if the table got corrupted, chkdsk wouldn't be able to help you. Such are the tradeoffs I guess.

It annoys me badly when discussion threads get closed because of age or anyone complains about resurrecting an old thread. I can see the logic in that if it's a personal discussion between two friends and it doesn't make sense to talk about it anymore, but in the case of a programming discussion people should continue to add to the thread for all eternity until the topic naturally passes out of relevance to the world. By adding to the thread we continue to impart more and more knowledge to the topic.

Regarding this discussion of the $UPCASE file on NTFS, the situation is this: Vista's format tool write the $UPCASE file differently than does the XP and earlier tools, I think Vista is writing it incorrectly, actually. So to the poster that asked "Who is changing the $UPCASE file," there's your answer, the Vista format tool that formatted the disk when Vista was installed. The Vista chkdsk program has been patched so that it takes no action on the problem, that's why running chkdsk in Vista neither reports nor fixes the problem.

The XP CHKDSK program, however, both reports and fixes the problem when run on a disk that contains Vista, another way the $UPCASE file could get changed. Apparently Vista runs equally well against the XP style $UPCASE file or the Vista style file, or maybe you guys were discussing the finer points of that above but I just tuned out because I'm not interested right now. If the file is wrong as I suspect, that Vista CHKDSK has been patched to ignore it is either a sign of negligence (M$ didn't bother to investigate the matter but instead just swept it under the rug by commenting out a check in CHKDSK), or malice (M$ intentionally planted this problem in Vista and then paid hush money to a CHKDSK programmer), or it could be good old plain incompetence of a coordinated nature: FORMAT & CHKDSK both incompetent.

However there is a problem that hasn't been mentioned, and that is that the incorrect Vista style $UPCASE file breaks PowerQuest's (or Symantec's) Partition Magic (PM) program. Since that program is useful that makes this $UPCASE issue a problem for me. PM will issue an error 4444 if it encounters this problem and will refuse to proceed with its work. The solution is to run the XP CHKDSK program on the Vista partition so that it will fix the $UPCASE file and then PM can proceed. I have to wonder if this bug was planted maliciously by Microsoft to kill this excellent program. They've done it plenty of times before.

I hope that no one here will object that the Vista disk management tool has obsoleted Partition Magic. People that say that obviously don't edit many partitions. To put it briefly, a full featured partition management tool offers many many more functions than does the anemic management tool provided by Vista. One quick example is the ability to grow a partition into free space to its left, a very useful function. Partition Magic does that easily in about 2 minutes, Vista can't do that at all.

One of the other reasons to ask for no more updates is when updates get into untruths, conspiracy theories, and other swill that the blog owner feels are beneath the dignity of the topic.

Consider yourself warned, Larry Brown!

The only change that was made was to the situation leading to an error msg claiming corruption in the case of a known change that chkdsk was unaware of. The Vista change was not wrong; neither is the Win7 change.

When one formats USB jump drives in XP and plugs them into Vista or Win7, one gets the warning to update for the same problem; it is not due to the casing table being wrong or hush money or negligence or any other crap like that....