lexparse

Lightweight English NLP microservice: dependency parsing, POS tagging, morphological features (FEATS), and MWE (Multi-Word Expression) detection via slot/gap matching.

Built with Rust + ONNX Runtime.

Model	Size	HuggingFace
deberta-v3-xsmall ewt+gum	fastest, lowest RAM	ghotriw/deberta-v3-xsmall-biaffine-dep-pos-feats-en-ewt-gum
deberta-v3-small ewt+gum	faster, less RAM	ghotriw/deberta-v3-small-biaffine-dep-pos-feats-en-ewt-gum

Setup

1. Download model

./download_models.sh

2. Build and run

cargo build --release
./target/release/parser-service

Service listens on 0.0.0.0:3000.

API

POST /parse

Accepts a JSON array of sentences, returns an SSE stream.

curl -X POST http://localhost:3000/parse \
  -H 'content-type: application/json' \
  -d '{"sentences": ["She came up with a great idea.", "He spilled the beans."]}'

SSE events:

Event	Payload
result	{ index: number; result: SentenceResult } — emitted per sentence as it completes
progress	{ done, total, percent } — emitted after each sentence
done	{ results: SentenceResult[] } — full array on completion
error	error message string

result events arrive incrementally; clients that only need the full batch can ignore them and use done.

SentenceResult:

{
  tokens: {
    id: number;      // 1-based index in the sentence
    word: string;    // original surface form
    lemma: string;   // base form used for matching
    upos: string;    // Universal POS tag (VERB, NOUN, ADJ, …)
    feats: string;   // CoNLL-U morphological features, e.g. "Number=Plur|Tense=Past"; "_" if none
    head: number;    // syntactic head id; 0 = ROOT
    rel: string      // UD dependency relation (nsubj, obj, …)
  }[];

  mwes: {
    id: number;              // lexicon entry id
    pos: string | null;      // syntactic category of the whole MWE ("verb", "noun", …)
    phrase: string;          // canonical dictionary form, e.g. "give up", "spill the beans"
    definition: string | null; // Wiktionary gloss, if available
    surface: string;         // actual text spanned in the sentence (includes gap words if discontinuous)
    categories: ("idiom" | "phrasal_verb" | "proverb" | "fixed_collocation")[];
    has_slot: boolean;       // pattern has a wildcard slot (e.g. "spill someone's beans")
    token_ids: number[];     // 1-based ids of matched tokens (gap words excluded)
    words: string[];         // surface forms of matched tokens (parallel to token_ids)
    discontinuous: boolean   // matched tokens are not contiguous in the sentence
  }[];
}

Morphological features (feats)

Each token's feats is a CoNLL-U FEATS string: Category=Value pairs joined by |, sorted alphabetically by category, or _ when the token has no features. Examples:

were     →  Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin
dogs     →  Number=Plur
quickly  →  _

Predicted per token by a multi-label head (one independent classifier per category). Categories present in the model (UD English EWT+GUM tagset):

Category	Values
Number	Sing, Plur, Ptan
Person	1, 2, 3
Tense	Past, Pres
VerbForm	Fin, Inf, Part, Ger
Mood	Ind, Imp, Sub
Voice	Pass
Degree	Pos, Cmp, Sup
Case	Nom, Acc, Gen
Gender	Masc, Fem, Neut, Fem,Masc (composite = both)
PronType	Art, Dem, Prs, Int, Rel, Ind, Neg, Tot, Rcp, Emp
Definite	Def, Ind
NumType	Card, Ord, Mult, Frac
NumForm	Word, Digit, Roman, Combi
Polarity	Pos, Neg
Poss	Yes
Reflex	Yes
Abbr	Yes
Foreign	Yes
Typo	Yes
Style	Arch, Coll, Expr, Slng, Vrnc
ExtPos	ADP, ADV, CCONJ, NOUN, PRON, PROPN, SCONJ

Only models trained with the FEATS head populate this field; UPOS-only checkpoints return _ for every token. The exact category/value inventory is defined by feats_vocab in model/vocabs.json.

GET /health

Returns ok.

Rebuilding the lexicon

dic/lexicon.jsonl is a generated artifact. Rebuild when you update the Wiktionary dump or builder_config.toml.

builder_config.toml controls:

• phrase_blocklist — phrases to exclude regardless of category (too generic, rare, or replaced by a corrected entry in dic/custom.jsonl)
• target_categories — Wiktionary category → internal type mapping
• pos_filter — POS allowlist per type

dic/custom.jsonl — hand-curated additions and corrections loaded at runtime alongside the generated lexicon (same LexiconEntry JSONL format). Supports typed slots: ["slot:pron"] (first fill token must be UPOS=PRON), ["slot:noun"] (NOUN/PROPN/PRON/DET), ["slot"] (unconstrained).

1. Download the Wiktionary dump

./download_dict.sh

Saves the decompressed dump to tmp/raw-wiktextract-data.jsonl.

2. Build the lexicon

cargo run --release --bin lexicon_builder

Testing

# Unit and lexicon-level tests (no model required)
cargo test

# Full regression tests including MWE detection (requires model artifacts)
cargo test -- --include-ignored

Utility Scripts

The src/bin/ directory contains several utility scripts:

• lexicon_builder — builds the primary lexicon from the Wiktionary dump.
• correct_idioms — LLM-assisted script to reclassify false positive idioms using Gemini.
• extract_wiktextract_test — extracts a JSONL test dataset of example sentences for phrases in the lexicon.
• eval_wiktextract — evaluates parser accuracy over the extracted test sentences.
• tokenize_lines — helper to tokenize a stream of text.

Configuration

Env var	Default	Description
PARSER_ADDR	0.0.0.0:3000	Listen address
PARSER_IDLE_UNLOAD_SECS	300	Unload model after N seconds idle
PARSER_BATCH_SIZE	1	Number of sentences to batch during parsing
PARSER_INTRA_THREADS	ORT default	Number of intra-op threads
PARSER_MEM_PATTERN	0	ORT memory pattern caching (0 or 1)
PARSER_CPU_ARENA	1	ORT CPU Arena allocator (0 or 1)