Psych.dump raises Encoding::CompatibilityError for UTF-16/UTF-32 strings

[Halvanhelv]

Summary

Psych.dump raises Encoding::CompatibilityError for any string in a non-ASCII-compatible encoding (UTF-16LE/BE, UTF-32LE/BE). An empty string is enough to trigger it.

This is asymmetric with the load side: Psych.load explicitly supports UTF-16 input and has tests for it (test_transcode_utf16le / test_transcode_utf16be in test/psych/test_encoding.rb), but dumping a UTF-16 string crashes.

require "psych"

Psych.load("--- こんにちは！".encode("UTF-16LE"))  # => "こんにちは！"  (works)
Psych.dump("".encode("UTF-16LE"))                  # => raises

Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp with UTF-16LE string)
    lib/psych/visitors/yaml_tree.rb:303:in 'String#match?'
    lib/psych/visitors/yaml_tree.rb:303:in 'Psych::Visitors::YAMLTree#visit_String'

Affects every dump entry point: Psych.dump, YAML.dump, and to_yaml on a String, or on any Hash/Array/nested structure containing one — including when the string is used as a hash key.

Cause

visit_String matches the string against US-ASCII regexp literals:

• yaml_tree.rb:303 — o.match?(/\n(?!\Z)/)
• yaml_tree.rb:313 — o.match?(/^[^[:word:]][^"]*$/)
• yaml_tree.rb:316 — /\A0[0-7]*[89]/.match?(o)

A string whose encoding is not ASCII-compatible cannot be matched against an ASCII regexp at all, so the first of these raises regardless of the string's content.

The guard above them only covers ASCII_8BIT:

def binary? string
  string.encoding == Encoding::ASCII_8BIT && !string.ascii_only?
end

so UTF-16/32 falls straight through.

Which encodings are affected

The crashing set maps exactly onto Encoding#ascii_compatible?:

encoding	ascii_compatible?	Psych.dump
UTF-8, ISO-8859-1, EUC-JP, Windows-1252, Shift_JIS, ASCII-8BIT	true	works
UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE	false	raises

Every other non-UTF-8 encoding transcodes and round-trips correctly, which suggests the UTF-16/32 family is simply an unhandled case rather than a deliberate restriction.

Expected

A UTF-16/32 string should dump like the equivalent UTF-8 string does — it should not be tagged !binary, and it should round-trip.

Environment

Reproduced on psych 5.4.0 (Ruby 4.0.4) and on master 79be592 (5.5.0), same file and line on both.

I have a patch and will open a PR referencing this issue.