Every character has a story #15: CAPITAL SHARP S (not encoded)

by Michael S. Kaplan, published on 2005/09/25 06:01 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2005/09/25/473632.aspx

Regular reader Maurits asked, in the Suggestion Box:

Can you comment on Andreas Stötzner's 2004 proposal for an upper-case ß code point, which was rejected by the Unicode consortium?

http://std.dkuug.dk/jtc1/sc2/wg2/docs/n2888.pdf

The proposal in question underwent a great deal of (not always entirely civil) conversation on the "member's only" list of Unicode....

I have also posted about the Sharp S before, on this blog, for example here.

The initial post about the proposal came from Markus Scherer of IBM:

Purely personal opinion:

I would have expected a proposal like this to see the light of day in a little less than 5 months...

Aside from few "discussions" and other curiosities, the majority of the document's samples shows clear _lowercase_ ß in otherwise uppercase text. Using normal ß in this way (like applying simple case mappings rather than full ones) is reasonably common. While German school children might at first scratch their heads about this irregularity, I am pretty sure that there is no pressure at all for introducing an uppercase variant - other than possibly by a local font vendor in search of a market.

It might be more likely for Germans to give up on ß than to add an uppercase version.

markus

http://www.daujones.com/comments_all.php?usrid=3504
http://faql.de/eszett.html
http://www.eibe-online.de/schulen/bfs_bensheim/darstellung_bensheim/FachbereichFarbtechnik.htm

Michael Everson then weighed in:

Look again. It shows capital sharp esses, though it does show small sharp esses in capital use because nothing else was available. The Duden evidence is not to be ignored.

People have been discussing this issue for a century. I think Stötzner has shown clear evidence for e capital sharp s.

Nobuyoshi Mori took a more technical approach to the analysis of the propoal:

My understanding is:
    1) Technically toupper( U+00DF ) should be defined.  It is currently defined as : toupper( U+00DF ) -->  U+00DF
    2) There are several ways to "display" an "uppercase ß" in German:
      2-1) "SS"    This is what German orthography says. It is also the most usual way to handle it.
      2-2) "ß"     This is used in exceptional cases when either there is no space for two characters, or for typographic reasons, or by ignorance of the correct orthography.
      2-3) "SZ"     This is an old variant of 2-1, only very rarely used.

The change of the current definition 1) breaks many existing Unicode implementations and data, and will cause compatibility issues.  The major issue is that the result of toupper( U+00DF ) becomes Unicode standard version dependent. 

\I know huge amount of Unicode implementations and Unicode customer data which will run into problems with the suggested Standard change.  Most of the database implementations, OS and PC products, Computer language implementations such as Java, C#, etc would be some of the examples. 

...

I therefore would like to request UTC to refuse the proposal.

Mark Davis agreed but had one small correction:

See http://www.unicode.org/versions/Unicode4.0.0/ch03.pdf#G33992

The Unicode data supports two types of case operations: full and simple. The simple operations are for restricted environments where the number of characters cannot be changed. For any other situations the full mappings should be used. And when a full mapping is used, toUppercase( U+00DF ) --> "SS"

Markus Scherer was also unconvinced by Michael Everson's response:

I did look at the whole document including at each and every sample. Most of them are clearly lowercase ß between uppercase letters.

"People" may have been talking about it depending who "people" is. I spent my first 27 years in Germany and have never heard of any serious discussion of an uppercase ß. (Not sure I even need to qualify this with "serious".) Unless there has since then been an outcry in the population that I missed while visiting about once a year or while talking with my relatives, I don't see that this is on anyone's mind.

Real issues in discussion included the spelling of Kaiser (Keiser?) and other beloved words when the spelling reform was published.

Michael Everson responded thusly:

Which is what you would expect to find, in the absence of a more widespread availability of fonts with capital sharp esses. The evidence, and the author's arguments, suggest that the sharp ess is certainly acquiring case, and indeed has done so, whether the use of it is widespread or not. In my view, the Universal Character Set should encode such entities where they exist. They are facts.

Ken Whistler had to respond to that argument, though (I was tempted to respond myself, but I am glad he did instead since he argued the case more convincingly:

You are getting caught up in your own rhetoric about the UCS. The Universal Character Set is *not* the Universal Encyclopedia of Writing Systems -- it is a practical attempt at an engineering solution that everyone can use for digital representation of text.

Introduction of a capital ess-tzet just because it "is a fact", and despite the manifest evidence of overwhelming German practice and implementation to the contrary, while utterly ignoring all the kinds  of implementation problems that would result -- just hinted at by Nobu -- is just foolish.

The problem you are trying to deal with, namely the appearance of an majuscule design in some fonts for an ess-tzet in an all-uppercase context, can be dealt with by other techniques, specific to fonts and to word-processing systems (if there even proves to be a demand for it, which I doubt, given Markus' testimony). It does not require a muley insistence that because somebody shows in some context that it *might* be treated as a distinct uppercase letter, that that resolves all issues and makes it obvious that separate encoding is required in the Unicode Standard for this "thing".

I am getting *really* impatient with the kind of rhetorical stance you have taken here. It is not your job nor the job of the UTC nor WG2 to reform the German writing system. And it certainly is not in the UTC's interest to introduce, on spec, a dubious new German character, without a demonstrated need, but with a horrendous downside potential for screwing up German casing implementations. My prediction is that the UTC is quite likely to turn this one down flat, without a single member in favor of it.

As I said, much more convincing.... :-)

But it looks like everyone dug their heels in; Michael responded to Ken:

As a student of the world's writing systems, I maintain that what I said is true. The evidence, and  the author's arguments, suggest that the sharp ess is certainly acquiring case, and indeed has done so, whether the use of it is widespread or not.

This may be an issue for some, or many, or most, current implementations. That's a concern for industry today. My work on the Universal Character Set, as you know, looks to tomorrow.

Not being a complete idiot, my response to Stötzner on this particular character is "Get Germany and Austria behind the proposal."

But facts are facts. You recently wrote a piece where you acknowledged that many of Unicode/10646's current "mistakes" will one day be purged. A hack for casing sharp-ess would seem to be one such. Stötzner's Weise, Weisse, Weiße casing to WEISE, WEISSE, WEISSE/WEIßE is a problem German implementations have to deal with now. I strongly suspect that the "solutions" are not AT ALL uniform or satisfactory to the Mr Whites out there. A capital scharp-ess would allow a consistent solution, and would, in my view, be superior to some sort of smart-font hack where a sharp-ess preceded by a capital letter would take on a different shape. That is not very portable, and, if I may remind you, from the 10646 side we are concerned with data preservation and transfer, not just implememtation by big companies.

Yes, these are philosophical differences in the two standards, but they are ther nonetheless.

>I am getting *really* impatient with the kind of  rhetorical stance you have taken here. It is not your job nor the job of the UTC nor WG2 to reform the German writing system.

No, the Germans have been looking at that themselves. In 1902 they did, and Stötzner is doing it again today. That's also a fact.

>And it certainly is not in the UTC's interest to introduce, on spec, a dubious new German character, without a demonstrated need, but with a horrendous downside potential for screwing up German casing implementations.

I wouldn't encode the character on foot of this one proposal either. But there is a case to be made for this character, and it would be wrong to reject the proposal out of hand.

Others such as Asmus Freytag and Benson Margulies also chimed in agreeing that an answer of 'proposal insufficient' seemed best at this point.

John Hudson then mentioned:

By the way, I met Andreas Stötzner at the recent ATypI conference in Prague, and am familiar with his journal. He is an intelligent and reasonable man, and I doubt if he would be insistant about the encoding of a Capital Double S if the text encoding and processing impact were explained to him. He has documented, in an admirably thorough way, a development in *some* German typography, which needs to be addressed at some level of text encoding or display. It is not obvious that the best way to do this is to encode a new character.

A bit more discussion but it kind of petered out without any real sense of consensus.

Shortly thereafter, at the November 2004 UTC meeting in Cupertino, CA, a bunch of discussion ensued, but in the end Conensus 22 happened:

[101-C22] Consensus: The UTC concurs with Stoetzner that Capital Double S is a typographical issue. Therefore the UTC believes it is inappropriate to encode it as a separate character.

and it was added to the Rejected Characters list with the following comment:

LATIN CAPITAL LETTER DOUBLE S (existence as character not demonstrated; would cause casing problems for legacy German data)

I probably would not have worded the consensus in just that way, but the end result would have been the same....

Three months later, a thread came up on the Unicode List about the Sharp S and uppercasing it, which mainly dwelt on issues other than adding the character. So I will spare everyone the further conversation. :-)

# Mathias Raacke on 25 Sep 2005 6:53 AM:

# CornedBee on 25 Sep 2005 9:49 AM:

# Chris Nahr on 26 Sep 2005 2:56 AM:

# Suzanne McCarthy on 26 Sep 2005 10:34 AM:

# Jonas Grumby on 26 Sep 2005 11:30 AM:

# Maurits [MSFT] on 26 Sep 2005 12:12 PM:

# Michael S. Kaplan on 26 Sep 2005 2:50 PM:

Scott on 11 Oct 2009 10:56 PM:

referenced by

go to newer or older post, or back to index or month or day