Jittor shows why Ascend inference is more than a compiler problem

The Jittor Industry-Academia Alliance was unveiled at Tsinghua University on May 29, 2025. The photograph matters because accelerator portability is an ecosystem problem: framework researchers, model builders, chip vendors, and operators all have to maintain the bridge.[6]

On a Tsinghua University stage in May 2025, two officials pulled a red cloth from a plaque while representatives of universities, model companies, and chip vendors stood in a long row behind them. The new Jittor Industry-Academia Alliance brought together 24 initial council members across the AI supply chain, including Huawei, Tencent, Zhipu AI, JD, and Moore Threads.[6] It was the kind of ceremony that can look more institutional than technical. In this case, the group portrait captured the problem unusually well.

China does not get a durable alternative AI stack merely by manufacturing an accelerator or releasing an open-weight model. The software between them has to translate model operations into hardware work, control memory and communication, expose a usable service, and survive changes on both sides. Jittor, a deep-learning framework developed by Tsinghua's computer graphics group, is instructive because its team has pursued more than one way to own that middle layer.[1][2][5]

The important point is not one unbroken stack. The 2025 Jittor-on-CANN port tests whether a compiler and graph abstraction can keep upper-layer model code stable across accelerators. JittorInfer takes a separate route: a C++ serving runtime that explicitly draws on code and ideas from llama.cpp, ggml, and torchair, then uses Huawei's CANN Graph Engine and Ascend-specific work. Taken together, the two efforts test related but distinct propositions—framework-level portability and accelerator-native serving. Treating them as one direct code lineage would erase much of the engineering that the second path exposes.[2][4][5]

The original wager was below the model

Jittor's design predates the current large-model race. Its 2020 paper describes a fully just-in-time compiled framework built around meta-operators: a small set of NumPy-like primitives that can be composed and compiled into specialized CPU or GPU code. A unified graph executor then manages those operations while applying techniques such as operator fusion, cross-iteration fusion, and unified memory management.[1]

That architecture changes where portability work can live. A framework dominated by hand-written, backend-specific kernels accumulates a long translation list whenever a new accelerator arrives. A compiler-centered framework can, in principle, move more of the adaptation into operator generation, graph lowering, and a bounded collection of hardware interfaces. “In principle” is important: a new backend still needs compilers, libraries, memory semantics, numerical checks, and tuned implementations of expensive operations. The abstraction does not abolish hardware work; it gives that work an address.

That compiler-first design gives the project a portability thesis wider than one accelerator. The useful AI-China signal is therefore not simply “another framework.” It is a long-running attempt to make the compiler and graph layer a point of leverage across heterogeneous compute.[1]

The Ascend port shows where the abstraction holds

In January 2025, the Jittor team documented its integration with Huawei's CANN 8.0 stack for Ascend 910-series processors. At the top of the example, the visible migration was almost comically small: replace use_cuda=1 with use_acl=1. The team reported that existing Jittor programs could then move from Nvidia to Ascend without changes to their upper-layer model code.[2]

The work beneath that flag was not small. The release describes CANN interface integration, a FlashAttention implementation, model training and inference checks, and numerical comparisons at the operator level. At that stage, the named large-model coverage was LLaMA 7B, ChatGLM, and ChatRWKV through JittorLLMs; image and multimodal examples extended the path to ResNet, ViT, Stable Diffusion, and CLIP. It also exposed Jtorch, intended to route much PyTorch-style model code through Jittor's backend.[2]

This is the first boundary to keep visible. A one-line switch is the desired user experience, not evidence that every PyTorch workload will migrate untouched. Coverage depends on the operators, shapes, dtypes, model architectures, and distributed patterns that have actually been implemented and tested. Portability is earned model by model even when the compiler makes each new model cheaper to support.

JittorInfer takes a separate runtime route

The team's DeepSeek adaptation made the need for a specialized runtime concrete. In April 2025, it reported Ascend support for DeepSeek-V2-Lite at 16 billion parameters, DeepSeek-V2 at 236 billion, and DeepSeek-R1 at 671 billion. Its account names the work rather than hiding it behind a compatibility badge: fused and unfused mixture-of-experts operators, extended rotary position embeddings, matrix absorption for multi-head latent attention, INT4 weight quantization, tensor and expert parallelism, and continuous-batching changes.[3]

When JittorInfer v0.1.0 was released in August, it was not simply the original Jittor Python/JIT stack repackaged for serving. The MIT-licensed repository is a standalone C++ runtime whose README credits ideas and code from llama.cpp, ggml, and torchair. Its public path requires Huawei's CANN toolkit, uses CANN's Graph Engine, and includes a llama-server with an OpenAI-compatible interface. The tagged release and its benchmark material center on DeepSeek-V2-Lite and DeepSeek-V2.[4][5]

The most revealing optimizations sit outside matrix multiplication. JittorInfer groups a sequence of fine-grained operations into one reusable graph dispatch so the CPU does not repeatedly wake the NPU. It moves token sampling onto the accelerator so a full logits vector does not shuttle back to the CPU on every decoding step. It replaces broadcast wake-ups in the HTTP task queue with targeted notification, and swaps a balanced tree for a bitset when tracking occupied KV-cache blocks.[4]

This is a different abstraction from the 2020 Jittor framework, but it sharpens the same supply-chain point. An accelerator is not productive because its peak arithmetic exists on a spec sheet. It becomes productive when dispatch, data movement, cache bookkeeping, batching, and the serving boundary stop leaving it idle. JittorInfer treats the host CPU, NPU, runtime, and HTTP service as one latency path.

The benchmark is a lead, not a verdict

The performance numbers are promising but tightly bounded. Jittor's August release reports tests on Ascend 910-series hardware with DeepSeek-V2-Lite on one card and DeepSeek-V2 on one eight-card server. Its repository compares throughput across concurrency levels of 1, 2, 4, and 8 against vLLM-Ascend v0.7.3 and, for the single-card table, v0.9.1. The reported gains vary materially with workload and concurrency.[4][5]

Those are project-published results, not an independent evaluation. The v0.1.0 repository does disclose useful setup details outside the headline tables: its environment notes identify a Kunpeng 920 host, Ascend 910B3 accelerators, CANN 8.2 RC1 alpha, and -O2 compilation; its throughput script uses a 15-second warm-up, five-token prompts, and 110–120-token decodes. But the tables do not label their metric or unit, while the script hard-codes private model and benchmark-tool paths and supplies no matched vLLM command or configuration. The linkage between table and script, plus baseline precision and quantization parity, therefore remains incomplete.[4][5]

The right reading is directional: the team has opened an implementation and published a comparison worth reproducing. It has not supplied an end-to-end reproduction recipe, and the result is not evidence that JittorInfer is faster across arbitrary models, cards, sequence lengths, or production traffic.

That distinction is especially important because the comparison target is moving. vLLM-Ascend, CANN, MindSpore, and model architectures will continue to change. A percentage lead tied to one runtime version can disappear; a maintainable method for profiling and retuning the stack is more durable.

The supply-chain test is breadth

As of July 12, 2026, JittorInfer's README and release metadata still lead with v0.1.0 and a DeepSeek-centered benchmark. The default branch is broader than that presentation: a June 10, 2026 merge added graph builders and configuration paths for Qwen3 dense models, Qwen3 mixture-of-experts models, and Llama 2 alongside DeepSeek. That code is a real breadth signal, but the presence of builders and example configurations is not the same as a documented, benchmarked production-support contract.[5]

Three signals would show that the expansion is becoming infrastructure. First, the newer model paths should become an explicit tested-support matrix, with architecture, dtype, quantization, context-length, and hardware coverage. Second, benchmark recipes should connect every published table to commands outsiders can rerun, including a matched baseline. Third, tagged releases and user documentation should catch up with the default branch while CANN and model upgrades leave the OpenAI-compatible service contract intact.

The alliance in the cover photograph could help with precisely that unglamorous work. Its membership spans framework research, models, hardware, cloud and industrial users.[6] If those participants contribute requirements, test capacity, kernels, bug reports, and deployment evidence, Jittor can turn institutional breadth into software coverage. If the alliance remains ceremonial while the runtime stays narrow, it will not.

The falsifier is clear: if the team's two Ascend paths cannot turn new model code into documented, tested support and reproducible results, then the Jittor name describes a collection of targeted adaptations rather than a supply-chain layer. But if framework-level graph abstraction and the separate serving runtime keep reducing the cost of each new adaptation, the project offers something strategically useful—a place where China's model and accelerator ecosystems can meet without pretending that hardware independence arrives automatically with the chip.

cronfeed.work