Unicode & Marty McFly

[richmond62]

BEFORE LC went 100% compliant it was 100% compliant, however odd that might seem.

Post Unicode compliance the terms numToCodePoint & codePointToNum were introduced, and they took over some of the roles of numToChar & charToNum.

HOWEVER prior to that ALL Unicode glyphs could be 'reached' using surrogate pairs.

Currently stacking our second load of winter logs.

BUT this evening watch this space for my surrogate pair stack.

[tperry2x]

At what version was it 100% compliant before it was 100% compliant?
I mean - if it was 100% compliant (the first time around), do you know what version of LCC that was?
If it was version 8 of the engine - then that's something else to add to the list, to see how things were handled in version 8 and see if they can be backported into the v9 engine. Is that what you mean? (apologies if I've got the wrong end of the stick).

Backported to the future.

[richmond62]

A glass of Bailey's Irish Cream,

change out of filthy, wet clothes,

turn computer on . . .

and.

[richmond62]

At what version was it 100% compliant before it was 100% compliant?

All changed at version 7.0

[richmond62]

BEFORE LC went 100% compliant it was 100% compliant, however odd that might seem.

So: I'm going to be predictably bloody-minded and decide to play some type of monster chess that employs some Cardinals that are represented in the Unicode standard like this:
-

Screenshot 2025-01-15 at 16.24.24.png (19.94 KiB) Viewed 483 times

-
Here are mine:
-

Cardinals.jpg (27.84 KiB) Viewed 482 times

-
Anyone feeling a bit 'kinky' should reflect on what one can do with a set of spare Knights, some chopsticks and an old-fashioned pencil sharpener.

[richmond62]

NOW, in OXT Lite I can get that Cardinal to appear in a field (assuming I have a font with that symbol on my computer) in a very straightforward fashion:

Code: Select all

put numToCodePoint(0x1FA50) into field "ChessDisplay"

or

Code: Select all

put numToCodePoint(129616) into field "ChessDisplay"

129616 is the decimal representation of the hexadecimal number 1FA50.

I'm using the Noto Sans Symbols 2 font: https://fonts.google.com/selection
-

Screenshot 2025-01-15 at 16.53.50.png (113.03 KiB) Viewed 475 times

-

[richmond62]

Now I have got stuck as I am over on the other side of my room playing around with LC Community 6.7.10 and
have got as far as:

Code: Select all

on mouseUp
set the useUnicode to true

worked out the 2 numbers for my surrogate pair, but cannot for the life of me remember HOW to get that up-and-running with

Code: Select all

numToChar

-

Screenshot 2025-01-15 at 17.48.17.png (124.41 KiB) Viewed 472 times

[richmond62]

https://livecode.com/tag/livecode-7/

https://livecode.com/tag/livecode-7/page/2/

[richmond62]

I recently had a 'fight' with someone in the Bulgarian department at the University of Plovdiv because they could NOT understand why a document they had written in a non-Unicode Bulgarian font would come out like donkey-droppings on my computer.

Read this and weep:

https://philip.html5.org/data/charsets-2.html

[richmond62]

OK, OK, OK: nothing that a hot bath and a glass of whisky won't sort out:
-

Screen shot 2025-01-15 at 6.57.54 PM.png (609.6 KiB) Viewed 461 times

-
And the lesson for today is:

If you are nutty in 2 ways:

1. You want to run ancient versions of LiveCode and/or retro versions of OXT

and

2. You want to bung complex Unicode characters in text fields and so on:

You will have to 'pay' for it.

[OpenXTalkPaul]

OK, OK, OK: nothing that a hot bath and a glass of whisky won't sort out:
-
Screen shot 2025-01-15 at 6.57.54 PM.png
-
And the lesson for today is:

If you are nutty in 2 ways:

1. You want to run ancient versions of LiveCode and/or retro versions of OXT

and

2. You want to bung complex Unicode characters in text fields and so on:

You will have to 'pay' for it.

I feel for ya, It's a lot of work just to go out to buy some already-chopped wood for our fireplace

Cool, so that last post is unicode in Mac Community v6? That's cool! I hadn't needed to deal with multi-byte unicode fonts or characters much. or at all around the time that v6 was the main version, and I think I kind of assumed that unicode fonts were mostly only for non-english written languages that include many glyphs, but now days there are also mega-fonts that include mega-dingbats (Emoji fonts), and Fonts with colorized versions of the font in other unicode slots. iCan you cut a Unicode character from another application and paste it in the field control and vice versa? I'm just wondering what v6 can NOT handle as far as unicode characters, just how not-fully supported was it?

For some use cases it might be easier or faster to have pre-generated a full set of the multi-byte character ranges (like a full chess piece set) that you're going to be using, stored already in a off-screen field or other container (like a global array variable) and where you could filter them by the byte values too, rather than having to generate them from numbers and then combine them for a 'UTF-32' 4-byte characters. Also maybe using binaryEncode to do number to all 4 bytes at once would be faster (if speed is an issue). Just throwing out ideas here.

[richmond62]

Of course the pre-7 and the post-7 situation re Unicode is not that simple: after all, were it that simple it would not be anything special.

So: with the advent of LC 7 one could enter Unicode text in the same way as non-Unicode (i.e. ASCII) text:

Imagine if you will a stack with a field (fld "ff") like this:
-

SShot 2025-01-16 at 10.34.21.png (47.09 KiB) Viewed 401 times

-
AND we want to put the symbol for a Cardinal directly after the words in the field.

Everything is really straightforward:
-

SShot 2025-01-16 at 10.38.22.png (84.8 KiB) Viewed 400 times

[richmond62]

However in LC 6.x (I used LC 6.1 on MacOS 10.7 for this example) 'things' are a bit more complicated:
-

Screen Shot 2025-01-16 at 10.51.37 am.png (216.94 KiB) Viewed 394 times

-
There has to be what I like to call a 'shadow' fld (you could use a variable if you preferred) to hold the text in field "ff" as you insert the Unicode character.

This is because if you do what seems intuitive:

Code: Select all

set the unicodeText of fld "ff" to ((the unicodeTextof fld "ff") & numToChar(55358) & numToChar(56912))

You get something entirely "other".

[richmond62]

amazonia.jpg (62.12 KiB) Viewed 392 times

-
Some of my Amazons were making some fairly pointed remarks to each other about Unicode before LiveCode 7.0.
-
In terms of my Devawriter the change from the pre-7 to the post-7 way of doing things with Unicode made several things happen:

1. The whole shebang ran quite a bit faster when building complex constructions in a text field.

2. The number of lines in my 'monster' SWITCH statements decreased by about 40%.

Both of which have to be good.

I will always be grateful to Mark Waddingham for that work.

[tperry2x]

However in LC 6.x 'things' are a bit more complicated
This is because if you do what seems intuitive:

Code: Select all

set the unicodeText of fld "ff" to ((the unicodeTextof fld "ff") & numToChar(55358) & numToChar(56912))

You get something entirely "other".

So my question (and my apologies if I'm missing something blindingly obvious here), is why not just use v9 of the engine if it does what you want? I'm not sure I follow this at all
- I'm only commenting, as to whether there's something I can add/modify that you need adding or modifying...

[richmond62]

The point of this thread was that someone, somewhere (it might even have been you) asked some question anent Unicode capabilities in older versions of LiveCode/Runtime Revolution.

Also, I went through this because of a serious problem with the Bulgarian Department of the University of Plovdiv (home to many living fossils) who have been submitting articles, in Bulgarian, to the editors of an academic journal (one of the editors being my wife), and because many of the lecturers there use non-standard Bulgarian fonts developed in the early 1990s on machines that run Windows XP, Vista, and Millenium ("Why should we change them: it's your problem that your computers cannot read them?"); I MAY have to knock together some standalones (that will run on those fairly ancient, steam-driven versions of Windows) to "chew" their way through those articles and convert them from one of the 7 non-unicode font mappings into Unicode Cyrillic.

Of course if the University of Plovdiv were the ONLY institution in the world with ancient 'academics' who either cannot or will not adapt the world would be a wonderful place.

If you wonder where my obsession with inter-operability, cross-platform everything comes from: look no further.

[richmond62]

Aaaaaaah:

https://openxtalk.org/forum/viewtopic.p ... ode#p13195

Certainly, LCC 7 (64-bit) on Linux is an absolute trooper. Before unicode though which I know is a deal breaker for some. So so so fast though.

LCC 7 is NOT 'before unicode': in fact, as this thread shows: unicodeText was introduced in Runtime Revolution 2.0
and modified in LiveCode 5.5.

HOWEVER the introduction of numToCodePoint & codePointToNum and the removal of the need to mess around with surrogate pairs came about in LCC 7.

It is probably better to talk about "before easy, integrated unicode" or "after clunky, awkward unicode" than simple "before".

[tperry2x]

The point of this thread was that someone, somewhere (it might even have been you) asked some question anent Unicode capabilities in older versions of LiveCode/Runtime Revolution.

Quite possibly, because at one point I wanted to use the LCC 7 engine (just because it was noticeably faster), but I think it was Paul that said Unicode didn't come into play until version 8 - and he wanted something at least version 8 up because widgets. (I might have that wrong, not sure - I was trying to find the previous post, but can't).
Perhaps he meant exactly what you'd mentioned above. After v7 you didn't have to mess around with surrogate pairs.

Also, the other fly in the ointment was that LC didn't produce a 64-bit version of the v7 engine for MacOS, so that means it won't run past MacOS 14 - so I have to be on a later version of the engine if I'm continuing to support MacOS.

...I MAY have to knock together some standalones (that will run on those fairly ancient, steam-driven versions of Windows)...

Version 8 of the engine (and standalones) supported Windows XP (service pack 2), so you don't necessarily need to go back all the way to version 6.x to produce a standalone that might run on XP. (page 3 of this)

Version 9 of the engine (and standalones created with it), supports Windows 7 upwards (page 4 of this) I mean, Windows XP (checks notes...October 2001), so it's not unreasonable to say to someone that your app won't support their 23 year old OS, however - I do accept their attitude within reason (more on that further down).

If you wonder where my obsession with inter-operability, cross-platform everything comes from: look no further.

Not at all, I totally get that. I fully understand that attitude. It should be the focus of anyone doing upgrades to ensure that what they are upgrading can also read the older formats - saving back to older formats is of course another matter, because there's nothing you can do to work around something that might be missing on an old system when that end user won't /can't modify their system. BUT, supporting a system that's 23 years old is pushing the limits a bit.

[axwald]

Hi,

do I get this right? "In LC v6 it was perfectly possible to work with Unicode, it just took some more effort than when working with 'plain text'." Is this a correct interpretation of this quote?

In terms of my Devawriter the change from the pre-7 to the post-7 way of doing things with Unicode made several things happen:

1. The whole shebang ran quite a bit faster when building complex constructions in a text field.
2. The number of lines in my 'monster' SWITCH statements decreased by about 40%.

Both of which have to be good.
I will always be grateful to Mark Waddingham for that work.

*scratching head*

[richmond62]

It was perfectly possible to work with Unicode in LC 6: but badly documented.

However I did work with Unicode in LC 6 (and still bear the scars).