For users with 8 * W7900 (gfx1100, 48 GB RAM), the following command serves as reference parameters to start a Qwen 3.8 Flash Next for internal use.

Note: not all options are useful; parameters need to be tuned.

set -x
docker run --rm \
    --name vllm-rocm \
    --pids-limit=-1 \
    --group-add=video \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --device /dev/kfd \
    --device /dev/dri \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -v ~/.cache/vllm:/root/.cache/vllm \
    -v /nvme2:/nvme2 \
    --env "HF_TOKEN=$HF_TOKEN" \
    --env "NCCL_PROTO=Simple" \
    --env "VLLM_PLE_CPU_OFFLOAD=1" \
    --env "FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE" \
    --env "FLASHINFER_PCIE_IPC_ALLREDUCE=1" \
    -p 8000:8000 \
    --ipc=host \
    --entrypoint=/usr/local/bin/vllm \
    vllm/vllm-openai-rocm:nightly \
    serve /nvme2/Qwen3.8-Flash-Next-W4A16-AutoRound \
    --served-model-name Qwen3.8-Flash-Next \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --no-enable-flashinfer-autotune \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice \  
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \

Reference: https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next?hardware=mi300x&variant=bf16

Note: Using Qwen3.8-Flash-Next (bf16 original version) gives Out Of Memory, hence W4A16 version is used; will try original version when ngram can be offloaded to host RAM.

Note: using nightly version of the container, tag qwen38-flash-next seems not recent enough (“transformer: unrecognized model”).

Note about num_speculative_tokens: in vllm qsa_cache.py, QSAKeyStateCache.get_kv_cache_spec(...):

        span = self.compress_ratio + vllm_config.num_speculative_tokens
        capacity = self.compress_ratio * cdiv(span, self.compress_ratio)
        assert self.cache_config.block_size % capacity == 0, (
            f"QSA ring capacity {capacity} must divide the attention block "
            f"size {self.cache_config.block_size}"
        )

For Qwen 3.8 Flash Next, compress_ratio = 4 and block_size = 416, hence allowed num_speculative_tokens values are:

0, 1, 2, 3, 4, 9, 10, 11, 12, 25, 26, 27, 28, 45, 46, 47, 48, 97, 98, 99, 100, 201, 202, 203, 204, 409, 410, 411, 412.

Note: VLLM_PLE_CPU_OFFLOAD possibly requires vllm-project/vllm#53899: Support PLE-Offload for Qwen3.8-Flash-Next, no effect for now.