Alibaba open-weights 176B Qwen4 preview model

Alibaba has released weights for Qwen3.8-Flash-Next, a multimodal mixture-of-experts model with 176B total parameters — including 51B N-gram embedding parameters — that activates 6B parameters per token. The native context window is 262,144 tokens, extensible to 1M with YaRN.

The architecture is the notable part. Three of every four layers use Gated DeltaNet, which compresses history into a fixed-size recurrent state and so does not accumulate KV cache as sequences lengthen. The fourth layer uses Qwen Sparse Attention, which aggregates the sequence into micro-blocks and scores importance at the block level rather than per token, cutting the indexing overhead that grows with context in earlier sparse-attention designs.

Alibaba's published figures put the QSA attention kernel at up to 7.6x faster than full attention during prefill and 4.9x during decoding. In a serving test at 1M-token context with a 90% prefix-cache hit rate, the model reached 8.6x the prefill throughput of Qwen3.7-Plus.

NVIDIA, whose blog is the source here, says it provides best-effort Day-0 functional support via SGLang, vLLM and TensorRT LLM, plus post-training recipes through NeMo AutoModel and NeMo RL. It reports over 16K tokens/second per GPU and over 200 tokens/second per user on GB300 NVL72, a 72-GPU Blackwell Ultra rack with a 130 TB/s NVLink domain. The model also runs on DGX Station, DGX Spark clusters, and four RTX PRO 6000 workstation GPUs.

All performance numbers are self-reported by Alibaba and NVIDIA; no independent evaluation or accuracy benchmarks accompany the post. Weights are on Hugging Face and ModelScope.