Thursday, July 18, 2013

The Mystery of the Missing Accents

Kim Elliott's description of an apparent bug in the Fldigi suite of applications closely parallels mine.

The issue is that, if Fldigi is left open for the entire program, the way anyone would do it, something changes the character set even though no defaults have been changed.  While the program is still in UTF-8 characters, which include all the international accented ones, subsequent copy omits them, printing place holders instead.

Kim's repeated runs seemed to indicate that the bug happens after Flmsg is used. My testing would seem to verify this.

In any event, the problem goes away if Fldigi is restarted before the next foreign language message containing the accented characters.

Here's what Kim has to say about this:

VOA Radiogram during the weekend on 13-14 July 2013 included this sequence:

(1) Greetings to the Mexico City meeting of shortwave listeners in MFSK32 plain-text Spanish

(2) Same greetings to the Mexico City meeting in MFSK32 formatted for Flmsg (a message handling program that works with Fldigi).

(3) MFSK32 image publicizing the Mexico City meeting

(4) VOA News story about Pluto’s moons in MFSK32 plain-text MFSK32 Spanish

When I decoded from recordings and from the actual broadcasts, the Spanish accents displayed correctly from (1) but never from (4), even though the character set was still UTF-8. The only way to restore the accents was to close Fldigi and and turn it back on during (4). Why was this happening?

Using audio editing software, I deleted (2) and (3) and decoded from what remained. The accents displayed correctly on both (1) and (4).

Then I played (1), (2) and (4). The accents were no longer visible on (4).

Finally I played (1), (3) and (4). The accents were visible on (4).

The culprit is Flmsg. Something in the Flmsg code alters the character set. Whatever that may be, I’ll just avoid text containing diacritics after using Flmsg on the program.