-
Notifications
You must be signed in to change notification settings - Fork 764
Description
So this is a long tale, but basically when I was trying to add Unicode 16 to UnifontEX (https://stgiga.github.io/UnifontEX) at 65422 glyphs, adding 7 of the needed 17 caused the spots where there were no characters to turn into red question marks, and trying again a few days later, for science, didn't do this, but the TrueType it generated was seen by FreeType2 in TTF2PNG from the DataBeaver's Domain as being only 60K glyphs. Additionally, the entirety of Plane 14 was wiped, which was part of that. It wasn't just the Plane 14 variation Selectors that were wiped. Plane 0's got wiped too.
Additionally, going any higher than 65422 glyphs causes FontForge's Unicode Ranges menu to crash when viewed, and Linux gets farther into it than Windows does.
Opening upstream Unifont causes this, even on versions of Unifont that previously worked on the same edition of FontForge. Evidently, trying to open Unifont 15.1 permanently jacked up FontForge.
I'm projecting a highest Plane 0 Unicode version of Unicode 18, if we factor in the 17 new glyphs in Unicode 16, the upcoming Unicode 17 glyphs, and the futher-tentative Unicode 18 ones after that. Under current knowledge, that would put me at 65527, so 8 glyphs away from the limit. Unicode 19 would have to barely touch Plane 0 to be possible.
Of course, if FontForge maxes out at 65422 (which may explain Noto crashes), then Unicode 15.1 is the end. Oh and fontmake using FontTools actually causes worse problems. Not to mention that FontForge has some unique features that I simply can't jettison.
I got told that:
FontForge probably isn't an appropriate tool for developing fonts that attempt to squeeze right up to the limit.
Note that ancient (~2015-2018) versions of FontForge did go higher, but I can't use ancient versions because I'm targeting modern Unicode.
Also to those saying I should use something else, I must also impress that Unifont's TrueType builds are generated using FontForge. Having editor parity is good, even though only the 15.1 additions (of which there were 5, going from 65417 to 65422) were made with the SFDs due to it requiring a literal year to compile Unifont from source to even be able to have non-CFF Unifont 15.1. Unifont's CFF releases when opened in FontForge show up as Adobe Identity-0 CID rather than Unicode, AND there were issues for a while with their encoding beyond that.
Also the highest Plane 1 version I can go without HarfBuzz extensions is a version prior to any CFF usage, so glyf is the only route for that reason too. Meaning I have to force-compile.
I can't jettison FontForge due to stuff like the TeX table, the PfEd table with its Comments and FONTLOG sections I use, and in the case of DFONT, the BDF table. (To prevent issues with OpenType Sanitizer rejecting bitmap tables due to the assigned developer being bogged down, any formats usable as webfonts, TrueType included, don't use them. So the DFONT has them. It's bitmap AND vector. Obviously the OTB has these tables too. Also these tables double the file size.) Plus, by checking all the checkboxes in the Generate Fonts menu besides Old-style 'kern', I'm able to generate a font that targets every OS.
As such, ditching FontForge would cause compatibility issues aplenty.
In general FontForge is extremely unstable but I'm literally unable to utilize any other editor.
I'm not doing what Source Han Sans does. Part of me wants to see the maximizing through, while part of me acknowledges that UnifontEX has already had ten years of development, that FontForge fixes take time, and that it may have been a wontfix.
Also I wish FontForge could automatically make identical glyphs be one glyph mapped to multiple codepoints, because that would give me around a thousand more slots.
Also, being able to set IsFixedPitch in the post table would be helpful, as would supporting HarfBuzz beyond-64k extensions.
I know I have a LOT to say here, but I've used your software since 2014, completely self-taught. Every other font editor hasn't ran on the platforms I owned during my efforts, or costs money, something I lack for the same reason I lacked it in 2014. Yes, I was 12 when I first started using FontForge and now I'm in university.
So even though FontForge is buggy (something that a friend of mine also doesn't like), it's the only thing I can actually use. Also Unifont TrueTypes were made by FontForge so I can't really avoid it.
ALSO even back then, NOTHING in the Mac Features box would generate in a TTF, as if AAT wasn't supported at all.
The Windows version of FontForge has been less stable than the Linux version under WSLg Ubuntu.
Also there was a December 2023 bug in which adding JSTF would work, but adding BASE and/or MATH would cause quite a few tables to vanish, including some of the spec's "Critical" tables that stuff like Java and Winamp reject fonts for not having. Oh and the September 2023 version had actually removed a few tables compared to upstream Unifont, and so it didn't work there. GDEF was missing from the September 2023 version and the broken one. The table stuff in February 2024 more-or-less magically resolved itself when trying the whole process again out of OCD. The stakes for THIS bug however are higher, in that it either completely trashes the font's encoding, or it wipes 5000 characters including an entire Plane and a Plane 0 block related to most of that plane's characters. AND it causes the same crash source that upstream does post-loading of 15.1. To even do the Unicode 15.1 update I had to prevent the Unicode Ranges crash by using Goto to jump directly to the relevant blocks, and copying the relevant characters over, and it's when you do this to upgrade from Unicode 15.1 to Unicode 16 where the wheels fall off. Apparently, 65422 glyphs is the highest possible before FontForge starts going berserk. This also explains why people have issues loading fonts that go to full 65535. Also note that some operations operations can crash on that figure too. If you do manage to load beyond 65535 glyphs, which I did via merging Plane 0, Plane 1, and the UCSUR glyphs of very early Unifont 15.0 versions, exporting to SVG webfonts and BDF does work. So the limit isn't exactly a hard limit, and I was actually doing the 15.1 and 16 stuff to the main SFD and using SFDs created when building TrueType versions as the source of 15.1+ glyphs. So the edits should actually be cleaner than working with TTF directly. And yet, you run into the crash before you even get to 100 under the limit. The success was done on the same 2023 Windows FontForge I used to make the September 2023 UnifontEX release, the same Windows version I use nowadays with its easy crash on 15.1 that jacks up the program from going that high ever again.
Oh and Unifont 15.1.02 adds enough Plane 2 and Plane 3 glyphs on top of the 306 already there to make doing a regular Merge Fonts operation impossible due to the number added being multiple hundreds. So even the GUI copy itself is a workaround.
I did some calculations based on the Unicode 16 glyphs, the earmarked Unicode 17 glyphs, and the glyphs Unicode is beginning to give Unicode 18, and, give-or-take a few glyphs, I'm due to be extremely close to the limit with Unicode 18, like, less than ten slots. So unless Unicode 19 is like 12.1 or 15.1, in regards to Plane 0, Unicode 18 is the true maximum. Now, getting there requires FontForge stability improvements.
Basically, FontForge is extremely unstable and self-destructive and it always has been. Honestly some of the Unifont 16 upgrade bugs seem to be a result of memory management problems that involve random overwrites. You know, stuff where there are overflows into different areas of memory than the write is for, in ways that make the program prone to imploding with even the slightest twitch.
And Python is a high-level language.
Python doing this type of memory management epicfail is when you know something is seriously wrong with your code.
Oh and the SFD for a font this big is a bit over 100MiB, but it compresses extremely efficiently, even in my all-formats zip, of which it's an eighth of the uncompressed size (863MiB so it's a 99-minute CD uncompressed minus 7MiB, in Zip it's 99.5MiB, and in maximum 7z it's 50MiB, a business card CD.)
FontForge having to deal with a 100MiB SFD in memory evidently isn't graceful. I'm on AMD64. So this could ALSO be why Linux is slightly more stable because Windows still attempts to observe Win32 RAM use limits on 64bit programs without tricks. So this could also be a problem with memory filling.
Basically FontForge is apparently going horribly wrong internally. Also I published the UnifontEX SFD as part of GPL source disclosure compliance, and it's in the downloads.
Also, crashes become even more likely if you merge before deleting Unifont's placeholders, and/or start going above a 1:1 copy to an n:1 copy. Meaning that if you're trying to copy from multiple versions of Unifont open together, you have greater odds of crashing and this scales. Even three total projects isn't really safe.
Oh and every save, U+0000 loses its Unicode status afterwards so you have to go into Glyph Info to remind FontForge that U+0000 is a valid codepoint. "Set From Name" and "Set From Value" both work.
Horrifyingly-enough, some of the compiled SFDs regressed to UnicodeBMP despite having the Plane 1 Copyleft symbol and the Plane 2 and 3 characters in Unifont-JP and 15.1.02+. This naturally caused problems, and doing UnicodeFull had its own issues. These were generated by headless FontForge. The generated TrueTypes don't have this problem.
Oh and to make TrueType in 16.x, you actually have to edit the string C89 in the Makefile to C99, but Unifont doesn't tell you to do so. Unifont 15.1 didn't need this. Also even getting compilation working took a year and multiple computers. Evidently FontForge in Unifont headless usage ALSO has severe issues, and there's also some issues it has with actually building itself. I don't know what is going on with FontForge code, but it's definitely more-hacky than my JavaScript compression program BWTC32Key, which I threw TrueType into to make the new format WOFF3, and it beats WOFF1 and WOFF2.
Also of note is that FontForge WOFF2 actually produces a bigger file than WOFF1, and that's without the WOFF1 adding a VDMX table and using ttf2woff Zopfli recompression. So somehow WOFF2 miserably fails here. That too is a FontForge export.
Basically, FontForge needs a tune-up, because of its multiple severe bugs.
Also, I had to use default values for BASE and JSTF because double-clicking on certain sections in their menus related to extents and some other parts causes a full editor crash.
My Macs (Panther to last x86 macOS) hate the DFONT and FontForge doesn't read its own valid BDFs properly.
The gist of this issue is that FontForge falls apart easily.
I know you already use upstream Unifont as a fallback, but do you use their Plane 1? If you don't, you know where to find me.
Also, FontForge SVG glyph import of https://stgiga.github.io/gigaware/Bonnotodho.zip and https://stgiga.github.io/gigaware/1319stroke.zip
doesn't work on the relevant SVGs (I was trying to make TrueTypes out of these), even the ones that aren't Inkscape SVGs. Since I don't know how Unifont vectorizes bitmaps to SFDs I can't make the 16x16 versions or my Favicons into UnifontEX Private Use Area glyphs either.
With regards to SVG, if your FONTLOG/Comments has a -- in it, when it gets written as a comment tag in the SVG, the -- is interpreted as the end of the comment even without it being --> as is the normal way to end an HTML/etc comment (SVG and HTML can actually interbreed to make stuff like this which is quite literally an HTML5 Canvas interactive toy shoved into an SVGZ (GZipped SVG) as part of extreme 10x minification from 30KB to 3KiB. Ungzipped, it's readable as an SVG file or an HTML file without changing the contents. So yes, HTML comments DO exist in SVG). Basically, the standard HTML comments that also work in SVG break if even a partial comment ending is present. FontForge when writing the FONTLOG/Comments to an SVG webfont doesn't care. This broke older SVG versions of UnifontEX (and by extension my SVGZ builds). Perhaps there should be something to inform the user that their FONTLOG/Comments contains characters that end the comment prematurely. Because there's also CDATA comments in SVG, and they use a different comment ending scheme. For context, even a PGP/GPG header used for signing will count as an ending HTML comment and wreck things in SVG webfont export. Apparently there's now been a wave of people liking and even developing for iOS 6, including Discord clients, so SVG webfonts have become undead even though WebKit and Blink browser engines still technically support them. Plus in theory you could port over the SVG-in-OpenType to them for animated color emoji with no glyph limits at all, no CFF2 or HarfBuzz glyf required. Basically, fixing the SVG comment bug (warning the user) that ends up commenting out the entire font if -- is present in FONTLOG/Comments is something that should be done
TL;DR: loyal user finds major problems with FontForge after over a decade of use.