Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

by Michael S. Kaplan, published on 2007/05/14 13:51 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2007/05/14/2629747.aspx

(apologies to George Orwell, of course!) 

Val asks:

Michael,

I've been reading your "Striping Diacritics" post, and it's been a great help. I've also been comparing it with another version I've seen. This other version is similar to yours, except that it breaks up the original string into characters, then normalizes the characters individually.

I threw a few languages at the two functions. I found yours handled Vietnamese while the other one did not.

However I have a problem where I don't know the code page of the string before I feed it to the function. It may be a far-east language, it may be an european language, or it may be a far-east + european language. Further more it may be a katakana or hiragana character set (i think I got those right)

Your function corrupted japanese, chinese, and especially korean. The other function did not. However I'm hesitant to use the other function because of the problems it has with vietnamese (which is a latin character set, so it worries me that it's hiding other issues with other languages)

Is there a way to modify your function to be friendlier to languages that don't have diacritics? Right now, I've modified it that if it finds a non-latin character (based on unicode value ranges), it aborts the whole processing and returns the original string, but this obviously can't handle strings that have multiple character sets.

Fyi: the problems I've seen with yours is that if I run JP through it, it strips out things that LOOK like accents but are not. For example, on of the JP characters looks like a reverse V with a small circle on top right. After your function, the small circl on top right is missing. With korean, it seemed like it inserted blanks between each character after we run it through Normalize() function.

It's funny, I never know when I get questions like this one if people fully understand what they are asking for (no offense to Val!).

And I am not helped in this case by the fact that Val is the name of both

• a male software consultant in Toronto dealing a great deal with FoxPro who ran a whole bunch of conferences I have spoken at over the last decade and
• a female college student in Columbus working part-time as an exotic dancer to cover her tuition I dated for several months over a decade and a half ago

The latter being a stripper is interesting given that this new Val is asking me about stripping, albeit an entirely different kind of stripping!). The fact that Valerie found things about programming to be interesting only further clouds the situation. 

I have literally no sense of the person asking the question and am somewhat certain that neither of my previous Val experiences will properly guide me in this respect. :-)

I'll start with the Japanese case, which really is not a bug, or a shortcoming in the code.

After all, to me the conversion from å to a is a normal "diacritic stripping" operation, while a Swedish user  would probably throw something heavy at my head were it not for the fact that all the Swedes I have met are so bleeding polite.

I don't find that Swede's reaction to be any more or less reasonable that the Japanese user's suggestion (mirroring Val's) that the change from ピ to ヒ is "corrupting Japanese text."

In both cases, you can see why it happens:

U+00e5 (å) decomposes to U+0061 U+030a, and U+030a is Mn (Mark, Nonspacing), which the code is designed to strip.

U+30d4 (ピ) decomposes to U+30d2 U+309a, and U+309a is  Mn (Mark, Nonspacing), which the code is designed to strip.

So in essence what Val is asking for here is a way to say get at the sentiment expressed in the title: that all Mn characters are non-spacing, but some are more non-spacing than others. Which really does not exist in any kind of automated sense.

Now the Korean problem is a bit easier to describe and deal with. After all U+d3fc (폼) decomposes to U+1111 U+1169 U+11b7, and although none of them are non-spacing, if you don't have a font that can handle the conjoining Jamo (e.g. Gulim Old Hangul, the font I have on this machine), it will look like three separate Jamo rather than one single modern Hangul syllable.

Of course, with just a minor change to the code I first presented in Stripping diacritics.... to take the decomposed text and recompose it by converting it to Unicode Normalization Form C. The new code would look something like this:

namespace Remove {
using System;
using System.Text;
using System.Globalization;
  class Remove {
    [STAThread]
    static void Main(string[] args) {
      foreach(string st in args) {
        Console.WriteLine(RemoveDiacritics(st));
      }
    }

    static string RemoveDiacritics(string stIn) {
      string stFormD = stIn.Normalize(NormalizationForm.FormD);
      StringBuilder sb = new StringBuilder();

      for(int ich = 0; ich < stFormD.Length; ich++) {
        UnicodeCategory uc = CharUnicodeInfo.GetUnicodeCategory(stFormD[ich]);
        if(uc != UnicodeCategory.NonSpacingMark) {
          sb.Append(stFormD[ich]);
        }
      }

      return(sb.ToString().Normalize(NormalizationForm.FormC));
    }
  }
}

I don't know what in particular in the Chinese text was going wrong, and without an example it is hard to say; perhaps the new code will resolve the problem.

But the first case (the one with Japanese) is obviously the more interesting one in terms of functionality -- the request being for a way to strip diacritics that are considered to be meaningless does demand that a bit of rigor be applied to the meaning of the term meaningless, which could be linguistically derived from the meaning of the term corrupted (by assuming that anything that is called corrupted would have been meaningful had corruption not taken place!).

This post brought to you by ピ (U+30d4, a.k.a. KATAKANA LETTER PI)

# Dean Harding on 14 May 2007 6:50 PM:

# Peter Karlsson on 8 Jun 2007 3:21 AM:

# Michael S. Kaplan on 8 Jun 2007 10:37 AM:

# Steven Sudit on 19 May 2008 11:34 AM:

# Jan Kučera on 28 Oct 2008 8:38 PM:

# mkukik on 31 Oct 2008 5:38 AM:

Amino on 17 Dec 2009 1:54 AM:

James White on 23 Jul 2010 8:16 AM:

Gökhan Berberoglu on 16 Mar 2011 6:42 AM:

referenced by

go to newer or older post, or back to index or month or day