(Windows Terminal) Unicode Characters outside Basic Multilingual Plane not rendering

[AmmoniumX]

PDCurses Version: 3.9

Most characters outside the Basic Multilingual Plane (Plane 0, U+0000 - U+FFFF) don't render correctly (Windows Terminal)
Example characters from Plane 1 (Supplementary Multilingual Plane, U+10000 - U+1FFFF):
U+1F600 - U+1F64F, Emoticons: 😊👍🔥 (⭐ and ⚔️ are on the BMP and therefore they do render)
U+10900 - U+1091F, Phoenician Characters: 𐤀𐤁𐤂
U+1D400 - U+1D7FF, Mathematical Alphanumeric Symbols: 𝑎𝑏𝑐

See https://www.compart.com/en/unicode/plane/U+10000 for all blocks in the Supplementary Multilingual Plane. Many of them aren't really implemented by most fonts, but there are many common ones (like the ones mentioned above) that are.

Sample code:

#include <curses.h>
#include <locale>

void setup() {
    initscr();
    cbreak();
    noecho();
    curs_set(0);
    keypad(stdscr, TRUE);
}

void cleanup() {
    endwin();
}

int main() {
    // Initialize locale
    std::locale::global(std::locale(""));

    // Initialize curses
    setup();

    std::wstring ws = L"Emoticons: (😊👍⭐⚔️🔥) \nPhoenician: (𐤀𐤁𐤂) \nMathematical Alphanumeric: (𝑎𝑏𝑐)";

    // Add the string to the main screen
    addnwstr(ws.c_str(), static_cast<int>(ws.length()));

    // Refresh the screen to show the changes.
    refresh();

    // Wait for the user to press a key.
    getch();

    // Clean up and exit
    cleanup();

    return EXIT_SUCCESS;
}

Building and running:

> cl.exe /std:c++23preview /EHsc /W4 /utf-8 /DPDC_WIDE /I "C:\Users\diego\src\PDCurses-3.9\" .\testpdcurses.cpp ..\PDCurses-3.9\wincon\pdcurses.lib user32.lib gdi32.lib advapi32.lib

> .\testpdcurses.exe
Emoticons: (����⭐⚔️��)
Phoenician: (������)
Mathematical Alphanumeric: (������)

The issue is probably within https://github.com/wmcbrine/PDCurses/blob/master/wincon/pdcdisp.c , I see that it uses the WriteConsoleW API call, which I have tested in isolation and it does print unicode characters correctly to terminal, so this may be an implementation bug with the specific way it's used, and how PDCurses handles UTF-16 surrogate pairs.

Working example using Windows API:

#include <string>

#include <Windows.h>

int main() {

    std::wstring ws = L"Emoticons: (😊👍⭐⚔️🔥) \nPhoenician: (𐤀𐤁𐤂) \nMathematical Alphanumeric: (𝑎𝑏𝑐)";

    HANDLE console = GetStdHandle(STD_OUTPUT_HANDLE);
    WriteConsoleW(console, ws.data(), static_cast<DWORD>(ws.size()), NULL, NULL);

    return EXIT_SUCCESS;
}

Building and running:

> cl.exe /std:c++23preview /EHsc /W4 /utf-8 /DPDC_WIDE .\testwinapi.cpp
> .\testwinapi.exe
Emoticons: (😊👍⭐⚔️🔥)
Phoenician: (𐤀𐤁𐤂)
Mathematical Alphanumeric: (𝑎𝑏𝑐)

[AmmoniumX]

I just realized why ⭐ and ⚔️ work, apparently they aren't on the "emoticons" block on the Supplementary Multilingual Plane, they existed before that and are considered miscellaneous symbols on the BMP

[AmmoniumX]

I just tested, and this PR fixes the unicode rendering problem: #161

Could you please merge this PR so this issue can be resolved?

[Bill-Gray]

PDCurses uses 16 bits for characters. Ask it to render Unicode 0x1d11e, for example, and you get 0xd11e; the higher bits are masked out.

Rendering of the full range of Unicode requires 21 bits to be reserved for characters.

[AmmoniumX]

PDCurses uses 16 bits for characters. Ask it to render Unicode 0x1d11e, for example, and you get 0xd11e; the higher bits are masked out.

Rendering of the full range of Unicode requires 21 bits to be reserved for characters.

Are you saying that fixing this bug would require a major architectural refactor of the library? Then why does the PR I tested work fine? It seems to just add a standalone wcwidth implementation and modifies waddch to handle surrogate pairs better. Looking at pdcdisp.c it all gets stored into a wchar_t buffer anyways, and written to the terminal via WriteConsoleW.

Is there any problem with the PR that breaks some other PDCurses functionality? Because currently, PDCurses doesn't just render most non-linguistic unicode symbols correctly, but also thousands of CJK unified ideographs, both of which are something I'd expect to work if I'm calling a function to draw a wide string

[Bill-Gray]

I am a bit surprised that it worked. (I should confess here that I am using a fork that has fullwidth and SMP support already, and therefore hadn't looked too closely at this pull request.)

I figured that this pull request would fix the halfwidth/fullwidth problem, and hadn't seen the rather nice workaround for the 16-bit limitation (i.e, put the first half of a surrogate pair into one chtype, and the other half in the next chtype in the array.)

The only problem with this method (and the reason I didn't use it) is that it'll fail with halfwidth SMP characters. (Which is a lot of them.) In that case, you only can use one chtype and you're back to having only sixteen bits to play with. (As shown at the second link above, I got around this by declaring chtype to be a 64-bit quantity instead of a 32-bit one. That gave me -- among other things -- the ability to have 21 bits for character data instead of just sixteen.)

As you say, we also have the problem of font support, especially in the Windows console. (On some of the other ports, we have more control over such things.)