tokenizer-chinese only supports Mandarin - regional dialects fail
#340 · Rust · ★ 21k · MPL-2.0 · updated 1mo ago
The tokenizer-chinese feature uses jieba-rs, which only supports Mandarin Chinese word segmentation. Chinese regional dialects (Cantonese, Hokkien, Hakka) fail to segment correctly. Test Results Same sentence ("we are going shopping today") in four dialects, using jieba-rs 0.7.4:…
replies in ~1h8/9 outside PRs mergedmaintainer left guidancehas CONTRIBUTING
Cleared 1m ago