vLLM

A self-hosted vLLM server exposing an OpenAI-compatible endpoint, selected with LLM_PROVIDER=vllm, used to extract facts from conversations.

Enable it

LLM_PROVIDER=vllm
OPENAI_BASE_URL=http://localhost:8000/v1
OPENAI_LLM_MODEL=Qwen/Qwen2.5-7B-Instruct
# No API key: vLLM is served locally with no auth.

Configuration

Variable	Default	Description
LLM_PROVIDER	openai	Set to vllm.
OPENAI_BASE_URL	https://api.openai.com/v1	Point at your vLLM server: http://localhost:8000/v1.
OPENAI_LLM_MODEL	gpt-4.1-nano	The model served by vLLM, e.g. Qwen/Qwen2.5-7B-Instruct.
OPENAI_API_KEY	(none)	Not required; the local server has no auth.

Notes

• OpenAI-compatible: the provider reuses the OpenAI request handling and only differs by base URL and model.
• The provider also reads VLLM_BASE_URL as its own base-URL fallback (default http://localhost:8000/v1).
• Local server with no authentication, so no API key is needed.