Fix BufferedTokenizer to properly resume after a buffer full condition respecting the encoding of the input string#16968

Merged

andsel merged 11 commits intoelastic:mainfrom andsel:fix/buffered_tokenizer_clean_state_in_case_of_line_too_big_respecting_character_encoding

Feb 5, 2025

+426-13

Commits on Jan 31, 2025

Re-established existing tests with the addition of the encoding case transformation
andsel
committed
Fixed the test to compare Java strings byte rappresentation so that has the same encoding
andsel
committed
Re-imported previously reverted code
andsel
committed
Try to make the test reporduce the encoding problem
andsel
committed
Fixed test to verify the case of encoding preservation
andsel
committed
Switch from RubyArray addAll to concat method to preserve the encoding and avoid implicit deconding in addAll iterator
andsel
committed
Covered with more use cases and verifies encoding of the returned Ruby strings
andsel
committed
Updates the point of return Java String to return the one encoded with data input encoding, to do not change encoding
andsel
committed
Minor, removed unused import
andsel
committed

Commits on Feb 3, 2025

Force UTF8 for empty string when flush is invoked without a previous extract call
andsel
committed

Commits on Feb 4, 2025

Addressed some concerns raised in PR review
andsel
committed