Instructions to use DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst")
model = AutoModelForCausalLM.from_pretrained("DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst

SGLang

How to use DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst with Docker Model Runner:
```
docker model run hf.co/DeepAuto-AI/ldm_soup_Llama-3.1-8B-Inst
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model Card for DeepAutoAI/ldm_soup_Llama-3.1-8B-Inst

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in

compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Overview

DeepAutoAI/ldm_soup_Llama-3.1-8B-Inst is developed by deepAuto.ai and builds upon the VAGOsolutions/Llama-3.1-SauerkrautLM-8B-Instruct model. Our approach leverages the base model’s pretrained weights and optimizes them for the Winogrande and ARC-Challenge datasets by training a latent diffusion model on the pretrained weights.

Through this process, we learn the distribution of the base model's weight space, enabling us to explore optimal configurations. We then sample multiple sets of weights, using the model-soup averaging technique to identify the best-performing weights for both datasets. These weights are merged using linear interpolation to create the final model weights for DeepAutoAI/ldm_soup_Llama-3.1-8B-Inst.

This approach has led to improved performance on previously unseen leaderboard tasks, all without any additional task-specific training.

The work is currently in progress