Unable to set vllm serve parameters using hf_overrides
#912 · Go · ★ 655 · Apache-2.0 · updated 1mo ago
It appears that parameters set using --hf_overrides are ignored by the vllm backend. Either when certain parameters are supplied (ie. tensor_parallel_size or max_model_len) Is there any other way to provide runtime arguments to the vllm backend outside using --hf_overrides? vllm…
13/17 outside PRs mergedpoints at a filehas repro steps13 comments — busy
Cleared 3m ago