Cambricon Siyuan 590 Running vLLM: Compile-Time Bottlenecks and Operator Adaptation Checklist
The Cambricon Siyuan 590 is often included in comparisons for domestic compute selection in China
The Cambricon Siyuan 590 is often included in comparisons for domestic compute selection in China, but the real bottleneck when running vLLM is not the chip's peak compute power—it is the engineering effort of operator adaptation during the compilation phase. This article provides a reproducible method for mapping out an adaptation checklist, helping decision-makers assess migration costs before project initiation.
Why Siyuan 590's vLLM Adaptation Bottlenecks Concentrate in the Compilation Phase
vLLM's high performance relies on kernel-level optimizations such as PagedAttention. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, PagedAttention reduces memory fragmentation by managing the KV Cache through paging, and it is the core mechanism behind vLLM's throughput gains. This means that any non-CUDA platform running vLLM must reimplement these kernels—and Cambricon's software stack is built on the CANN heterogeneous computing architecture, which differs significantly from the CUDA ecosystem.
According to the official positioning of CANN by the Ascend community, CANN is the core software stack of the heterogeneous computing architecture, providing operator libraries, graph compilation, and runtime. Cambricon's software stack follows the same "framework adaptation layer + operator implementation layer" path, but operator implementations cannot be directly reused. This is where the compile-time bottleneck arises: vLLM's operator graph must complete operator mapping within the target platform's compiler. Operators that fail to map will fall back to inefficient implementations or trigger errors outright.
How to Map the Operator Adaptation Checklist: A Three-Step Reproducible Method
Step 1: Statically scan operator dependencies. Use vLLM's built-in model configuration and profiling tools to record the operator call sequence along the inference path of the target model (e.g., the Qwen series MoE architecture). This step does not depend on the target platform and can be completed in an x86 + CUDA environment. The output is an "operator list," not an "adaptation result."
Step 2: Cross-reference the target platform's operator support table. Compare the list from Step 1 item by item against the official support list of the Cambricon CANN operator library, marking each operator into three categories: fully supported, partially supported (with semantic differences), and unsupported. Partially supported items are the real pitfalls in the compilation phase—semantic differences (such as data types, dimension layouts, or reduce semantics) do not cause compile-time errors but produce incorrect results at runtime.
Step 3: Validate fallback paths for "unsupported" and "partially supported" items. vLLM has fallback implementations for some operators (e.g., flash attention falling back to xformers or eager mode), but fallbacks change memory usage and the computational graph structure. It is recommended to run a minimal inference test case on the target platform and observe the actual mapping result of each operator in the compilation log, rather than only checking whether the final run succeeds.
Validation Criteria and Risk Boundaries for Compile-Time Adaptation
Whether adaptation is complete cannot be judged by "the model runs." Three validation criteria are recommended: first, completeness of the compilation log, confirming no silent fallbacks; second, operator-level correctness, performing numerical comparisons for each partially supported operator; third, end-to-end performance comparison, comparing throughput and time-to-first-token (TTFT) before and after adaptation under the same model and concurrency settings.
It should be clarified that this article does not provide cross-platform performance comparison figures between Cambricon and any other platform—Mingxin only has data on its own tested platforms, and cross-platform extrapolation has no basis. According to the PyTorch Documentation's official definitions of framework-side memory management and operator behavior, changes in the framework adaptation layer affect memory allocation strategies and data loading paths, which constitute architectural differences rather than performance gaps.
For decision-makers in platform selection, it is recommended to require vendors to provide a "three-piece set" before project initiation: an operator adaptation checklist, sample compilation logs, and fallback path documentation. If a vendor cannot provide a complete record of compile-time operator mapping, subsequent runtime performance issues will be difficult to diagnose. To validate adaptation effectiveness under real workloads, gate-based validation can be performed in a joint test environment to check whether TTFT reduction and throughput gains fall within the expected range.
Key Q&A
Q: What is the main bottleneck for Siyuan 590 running vLLM? A: The bottleneck lies in compile-time operator adaptation, not the chip's peak compute power. vLLM relies on kernel-level optimizations such as PagedAttention, whose operators must be reimplemented under the CANN architecture. Operators that fail to map will fall back to inefficient implementations.
Q: How do you map the operator adaptation checklist? A: Use a three-step method: first, statically scan operator dependencies in a CUDA environment; second, cross-reference the target platform's operator support table and classify each item; third, validate fallback paths for unsupported items and inspect the compilation log.
Q: How do you verify that adaptation is complete? A: Check three criteria: no silent fallbacks in the compilation log, correct operator-level numerical comparisons, and end-to-end performance meeting targets. "The model runs" is not a sufficient criterion.
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- PyTorch Documentation — https://pytorch.org/docs/stable/index.html
- CANN-Ascend Heterogeneous Computing Architecture-Ascend Community — https://www.hiascend.com/software/cann