If we look at the code in Data.Text.Encoding.encodeUtf8
encodeUtf8 :: Text -> ByteString
encodeUtf8 (Text arr off len)
| len == 0 = B.empty
| otherwise = unsafeDupablePerformIO $ do
fp <- mallocByteString (len*4)
...
it preallocates a 4 bytes for each 16-bit code unit in our text.
If this was 4 bytes for each character, it'd be correct.
However the maximum expansion from UTF-16 -> UTF-8 is 50%, as characters < 0xffff, which are all encoded with as one code, unit get encoded with at most 3 bytes in UTF-8.
Any 4 byte expansion requires an input that takes two code units, so this is overly conservative.
tl;dr replace 4 with 3, everything will still work and we'll leak 25% less memory when we happen to encounter a bunch of CJK text.
If we look at the code in
Data.Text.Encoding.encodeUtf8it preallocates a 4 bytes for each 16-bit code unit in our text.
If this was 4 bytes for each character, it'd be correct.
However the maximum expansion from UTF-16 -> UTF-8 is 50%, as characters < 0xffff, which are all encoded with as one code, unit get encoded with at most 3 bytes in UTF-8.
Any 4 byte expansion requires an input that takes two code units, so this is overly conservative.
tl;dr replace 4 with 3, everything will still work and we'll leak 25% less memory when we happen to encounter a bunch of CJK text.