DEVELOPER KNOWLEDGE BASE

NEXUS AI Documentation & CLI Guide

Learn how to compile models with TensorRT-LLM, serve high-throughput endpoints via vLLM, and integrate with the Python SDK.

1. Install the NEXUS CLI

The NEXUS CLI interacts directly with your local NVIDIA drivers to inspect GPU VRAM and launch quantized container runtimes.

# Install via Homebrew or curl
curl -sSL https://nexusvllm.com/install.sh | bash

# Authenticate with your Pro API key
nexus auth login --token nex_live_xxxxxxxx

2. Launch Instant vLLM Server

Run any model from the NEXUS / Hugging Face catalog with continuous batching and PagedAttention:

docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:latest   --model meta-llama/Meta-Llama-3.1-70B-Instruct   --quantization fp8   --max-model-len 16384   --tensor-parallel-size 2   --gpu-memory-utilization 0.95

3. Compile to TensorRT-LLM Engine

TensorRT-LLM provides maximum FLOPS utilization on NVIDIA Hopper (H100) and Blackwell (B200) architectures:

# Convert weights to TensorRT-LLM checkpoint format
python3 convert_checkpoint.py --model_dir ./Meta-Llama-3.1-70B \
  --output_dir ./tllm_checkpoint \
  --dtype fp8

# Build the optimized binary engine
trtllm-build --checkpoint_dir ./tllm_checkpoint \
  --output_dir ./engine \
  --gemm_plugin auto

4. Connect via OpenAI SDK (Python & cURL)

OPENAI-COMPATIBLE

NexusVLLM exposes a drop-in OpenAI-compatible endpoint at https://nexusvllm.com/api/v1. You can use your existing OpenAI SDK client without changing your application code:

# Python: pip install openai
from openai import OpenAI

client = OpenAI(
    base_url="https://nexusvllm.com/api/v1",
    api_key="nv_live_YOUR_API_TOKEN"  # Get token from /settings/tokens
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a high-performance LRU cache in Python."}
    ],
    temperature=0.7,
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Query via cURL:

curl -X POST https://nexusvllm.com/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer nv_live_YOUR_API_TOKEN" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello NexusVLLM!"}],
    "temperature": 0.7
  }'