I created this batch file Encode.zip (for Windows) which goes in the contrib\resources\translations

Excellent, @Kappa971. If you're using this BAT file successfully, could you make another PR that adds it?

The zip package contains the batch file which is this:

@echo off

set trans_dir=%~dp0

pushd "%trans_dir%"

echo In directory %trans_dir%:

rem (default)
rem keyb us
call :nconv_t "utf-8/en.txt" , en

rem keyb de
call :nconv_t "utf-8/de.txt" , de

rem keyb es
call :nconv_t "utf-8/es.txt" , es

rem keyb fr
call :nconv_t "utf-8/fr.txt" , fr

rem keyb it
call :nconv_t "utf-8/it.txt" , it

rem keyb nl
call :nconv_t "utf-8/nl.txt" , nl

rem keyb pl
call :nconv_t "utf-8/pl.txt" , pl

rem keyb ru
call :nconv_t "utf-8/ru.txt" , ru

exit

rem Normalization - for UTF-8 format locale
:nconv_t
echo Normalizing %~1 to %~2.lng
tools\uconv -f UTF-8 -t UTF-8 -x nfc "%~1" > "%~2.lng"
tools\dos2unix "%~2.lng"

and the tools folder containing uconv and dos2unix.

There are two problems:

1. dos2unix seems to have problems with the 0x1B character in the German, French and Spanish translations, so the files are skipped and the line break remains CRLF. The character 0x1B is indicated as Esc and seems to match the old ASCII color code (probably the first unknown character):
   
2. I have no idea if there is any difference in the uconv parameter -x nfc and -x '::nfc;'. the latter doesn't work on the windows cmd.

dos2unix seems to have problems with the 0x1B character

There is some discussion here https://www.computerhope.com/unix/dos2unix.htm#conversion about using dos2unix combined with UTF-8 files, maybe we need to make some adjustments to the dos2unix calls?

Another thought - maybe the [31;1m escape sequence could be switched over to using a color or formatting tag?

I have no idea if there is any difference in the uconv parameter -x nfc and -x '::nfc;'. the latter doesn't work on the windows cmd.

It looks like those extra symbols to the -x argument are handled, at least when I look at this code, specifically the strchr(translit, ':') comparison:

    if (translit != NULL && *translit) {
        UParseError parse;
        UnicodeString str(translit), pestr;

        /* Create from rules or by ID as needed. */

        parse.line = -1;

        if (uprv_strchr(translit, ':') || uprv_strchr(translit, '>') || uprv_strchr(translit, '<') || uprv_strchr(translit, '>')) {
            t = Transliterator::createFromRules(UNICODE_STRING_SIMPLE("Uconv"), str, UTRANS_FORWARD, parse, err);
        } else {
            t = Transliterator::createInstance(UnicodeString(translit, -1, US_INV), UTRANS_FORWARD, err);
        }

So it could be that the windows implementation of uconv doesn't have handling for the ':' colon or other symbols; in which case, leaving them out is probably fine.

@FeralChild64 - do you know more about this?

I think we want to let more people on Windows help with translation, so overall this is a great addition.

During any merge step, we will see all the diff'ed lines if MSYS2 uconv broke anything. So I don't see any harm in trying this.

If there's anything strange, it will jump out in the PR diffs.

There is some discussion here https://www.computerhope.com/unix/dos2unix.htm#conversion about using dos2unix combined with UTF-8 files, maybe we need to make some adjustments to the dos2unix calls?

Another thought - maybe the [31;1m escape sequence could be switched over to using a color or formatting tag?

It's probably enough for the respective translators to replace those codes with the new color tags, right?

So it could be that the windows implementation of uconv doesn't have handling for the ':' colon or other symbols; in which case, leaving them out is probably fine.

The funny thing is that this "Windows implementation" of uconv is taken from MSYS2 (I don't know where else to get it). On MSYS2 environment, the -x '::nfc;' parameter works, on Windows cmd it doesn't:

EDIT
I found uconv compiled with MSVC2019 here https://github.com/unicode-org/icu/releases/tag/release-72-1, but it gives the same error. However, if -x nfc is fine, then there's no need to freak out about this

@kcgen Sorry, I have no idea why Windows version behaves differently.

@kcgen Sorry, I have no idea why Windows version behaves differently.

I think we should be OK.

These tools are released by the ICU project (and they're on version 72!), I suspect our relatively simple (only UTF-8) cases should be handled without problems.

Worst-case: if we see any damaged formatting in future PRs, then someone can try the Linux/macOS script and compare the results.

If there are fatal flaws, then we can drop the batch file and then recommend only using non-Windows platforms for the conversion. But hopefully this is all just unnecessary planning :-)