use remote ai model for Tabby itself

[itforxp]

I have rent GPU server and want point local installed Tabby to that server. Is it possible at the moment?
Something like :
export TABBY_BACKEND_LLAMA=http://<REMOTE_MODEL_URL>
export TABBY_BACKEND_AUTHORIZATION="Bearer <YOUR_API_KEY>"
docker run -it -p 8080:8080 -v $HOME/.tabby:/data registry.tabbyml.com/tabbyml/tabby server --remote

---

Please reply with a 👍 if you want this feature.

[wsxiaoys]

Hi - https://tabby.tabbyml.com/docs/references/models-http-api/llama.cpp/ contains an example connecting tabby to a remote model HTTP server

[itforxp]

thanks. how do i run tabby to use the remote model instead of the local one. the current tabby binary options force me to run the local model

[itforxp]

also i see :

~/.tabby/config.toml

[model.completion.http]
kind = "llama.cpp/completion"
api_endpoint = "http://localhost:8888"
prompt_template = "<PRE> {prefix} <SUF>{suffix} <MID>" 

how can i provide Bearer Token to authorize on remote ollama side?

[wsxiaoys]

You could set api_key field for authorization. https://tabby.tabbyml.com/docs/references/models-http-api/openai/ contains relevant examples.

[itforxp]

Thanks! But what about : how to disable local tabby's served ai model? I tried to kill local tabby model, but it restarted rapidly

[wsxiaoys]

Tabby initiates three default models if they are not configured remotely. For more information, please refer to our documentation at: https://tabby.tabbyml.com/docs/administration/model/

Could you please confirm if you have set up all three models?

[itforxp]

i still have only local models

[wsxiaoys]

Please ensure that you have correctly configured the ~/.tabby/config.toml file, and verify that your tabby command is utilizing config.toml.

[10373064]

Please ensure that you have correctly configured the ~/.tabby/config.toml file, and verify that your tabby command is utilizing config.toml.

I'm having the same problem, been going through the problem all afternoon and still can't find a solution.
I have finished reading the relevant documents
The configuration does not seem to be taking effect
Using tabby_x86_64-windows-msvc 0.24
Configuration in config.toml.

[logs]
level = “debug” # “silent” or “error” or “debug”

[model.chat.http]
kind = “openai/chat”
model_name = “qwen2.5-coder-14b-instruct”
api_endpoint = “https://dashscope.aliyuncs.com/compatible-mode/4”
api_key = “sk-xxxx”

[model.completion.http]
kind = “openai/completion”
model_name = “qwen2.5-coder-14b-instruct”
api_endpoint = “https://dashscope.aliyuncs.com/compatible-mode/v1”
api_key = “sk-xxxx”

Start command . \tabby.exe serve

How to verify that the configuration in config.toml is working using