Cleared 3m ago · docker/model-runner #912 · Go · ★ 655 · Apache-2.0

Unable to set vllm serve parameters using hf_overrides

It appears that parameters set using --hf_overrides are ignored by the vllm backend. Either when certain parameters are supplied (ie. tensor_parallel_size or max_model_len) Is there any other way to provide runtime arguments to the vllm backend outside using --hf_overrides? vllm…

What we checked

Similar unclaimed issues