Cleared 1m ago · valeriansaliou/sonic #340 · Rust · ★ 21k · MPL-2.0

tokenizer-chinese only supports Mandarin - regional dialects fail

The tokenizer-chinese feature uses jieba-rs, which only supports Mandarin Chinese word segmentation. Chinese regional dialects (Cantonese, Hokkien, Hakka) fail to segment correctly. Test Results Same sentence ("we are going shopping today") in four dialects, using jieba-rs 0.7.4:…

What we checked

Similar unclaimed issues