EPS
← All batches·2605.06886

TajPersLexon: A Tajik-Persian Lexical Resource and Hybrid Model for Cross-Script Low-Resource NLP

topic: general_aitop score: 100released: 2026-05-12first surfaced: 2026-05-11arXivPDFnew_researchthreats2026-05-112026-05-12

Authors: Mullosharaf K. Arabov

arXiv · PDF

Summary

The author presents TajPersLexon, a curated 40,112-pair Tajik-Persian parallel lexicon for cross-script lexical tasks (retrieval, transliteration, alignment) in a low-resource setting. The benchmark compares three families of methods—lightweight hybrid pipelines, neural sequence-to-sequence, and retrieval—on CPU only. Neural and retrieval baselines hit 98–99% top-1 accuracy, essentially solving exact lexical matching, but large multilingual sentence transformers fail. The hybrid model offers a practical accuracy-efficiency trade-off, achieving 96.4% accuracy on an OCR post-correction task, and is interpretable and fast. All experiments use fixed random seeds for full reproducibility; dataset, code, and models will be released.

Main takeaways:

  • TajPersLexon is a 40k-pair Tajik-Persian lexicon for cross-script word/short-phrase matching (Cyrillic Tajik ↔ Persian script).
  • Neural seq2seq and retrieval methods nearly solve the task (98–99% top-1 accuracy) on exact lexical matching.
  • Multilingual sentence transformers fail despite being large, showing that sentence-level embeddings don't transfer to exact lexical tasks.
  • The lightweight hybrid model (rule-based + small neural components) is interpretable, CPU-friendly, and achieves 96.4% on real-world OCR correction.
  • All code, data, and models will be open-sourced with fixed seeds for reproducibility.

Relevance

No obvious connection to my persona or midtraining work—this is a low-resource NLP lexicon task. Included because the methodological lesson (large sentence embeddings fail on exact lexical tasks; hybrid approaches win on efficiency) is a useful general reminder about task-method fit.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.06886v1 Announce Type: new Abstract: This work introduces TajPersLexon, a curated Tajik--Persian parallel lexical resource of 40,112 word and short-phrase pairs for cross-script lexical retrieval, transliteration, and alignment in low-resource settings. We conduct a comprehensive CPU-only benchmark comparing three methodological families: (i) a lightweight hybrid pipeline, (ii) neural sequence-to-sequence models, and (iii) retrieval methods. Our evaluation establishes that the task is essentially solvable, with neural and retrieval baselines achieving 98-99% top-1 accuracy. Crucially, we demonstrate that while large multilingual sentence transformers fail on this exact lexical matching, our interpretable hybrid model offers a favorable accuracy-efficiency trade-off for practical applications, achieving 96.4% accuracy in an OCR post-correction task. All experiments use fixed random seeds for full reproducibility. The dataset, code, and models will be publicly released.