分类目录归档：技术笔记

openFuyao InferNex AI推理集成部署 310P(300I Pro) 环境问题记录及解决

AI推理集成部署（InferNex）是一个专为云原生环境下AI推理服务优化所设计的端到端集成部署方案。该方案基于Kubernetes Gateway API Inference Extension (GIE) 和主流LLM技术栈构建，通过Helm Chart将开源网关、智能路由、高性能推理后端、全局KVCache管理、扩缩容决策框架及推理可观测体系等核心加速模块无缝集成。它提供从请求接入、动态路由、推理执行到资源管理与监控的完整加速链路，旨在提升推理吞吐量并降低TTFT/TPOT时延，实现一站式的高效AI服务部署体验。

hccn 问题

似乎是 hccn 找不到

[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg 
Name:             vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg
Namespace:        ai-inference
Priority:         0
Service Account:  default
Node:             master1/10.17.30.131
Start Time:       Tue, 07 Apr 2026 09:31:33 +0800
Labels:           app.kubernetes.io/instance=infernex-vllm-pd-2p1d-01
                  app.kubernetes.io/name=inference-backend
                  openfuyao.com/dpSize=1
                  openfuyao.com/engine=vllm
                  openfuyao.com/model=qwen-qwen3-8b
                  openfuyao.com/pdGroupID=qwen3-8b-pd-01
                  openfuyao.com/pdRole=decode
                  openfuyao.com/ppSize=1
                  openfuyao.com/tpSize=1
                  pod-template-hash=6cd64bc69c
Annotations:      checksum/config: 476b32f01fc96ff2896aee7fce288cd2b58cdb2ac825d1a22518798806847a2c
                  huawei.com/AscendReal: Ascend310P-0
                  huawei.com/kltDev: Ascend310P-0
Status:           Pending
IP:               
IPs:              
<none>
Controlled By:    ReplicaSet/vllm-pd-2p1d-01-decode-6cd64bc69c
Init Containers:
  mooncake-config-init:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image ID:      
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/sh
      -c
    Args:
      set -e
      CONFIG_PATH="/app/mooncake.json"
      mkdir -p "$(dirname "$CONFIG_PATH")"
      cat > /tmp/mooncake_config.tpl << 'EOF'
        local_hostname: "$POD_IP"
        metadata_server: "redis://redis-service:6379"
        master_server_address: "mooncake-master-service:30089"
        device_name: ""
        protocol: "ascend"
        global_segment_size: 42949672960
        use_ascend_direct: true

      EOF
      POD_IP_VALUE="${POD_IP:-0.0.0.0}"
      sed "s/\$POD_IP/${POD_IP_VALUE}/g" /tmp/mooncake_config.tpl | yq eval - -o=json > "$CONFIG_PATH"

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAME:  vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
    Mounts:
      /app from mooncake-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
  huggingface-download:
    Container ID:  
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image ID:      
    Port:          
<none>
    Host Port:     
<none>
    Command:
      hf
      download
      Qwen/Qwen3-8B
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
    Mounts:
      /root/.cache from rootcache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
Containers:
  decode-engine:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0
    Image ID:      
    Port:          8000/TCP (decode-port)
    Host Port:     0/TCP (decode-port)
    Command:
      /bin/bash
      -c
    Args:
      # PHYSICAL_DEVICES stands for the physical devices assigned to the container, use for vllm ascend 0.10.x
      export PHYSICAL_DEVICES=$(ls /dev/davinci* 2>/dev/null | grep -o '[0-9]\+' | sort -n | paste -sd',' -)

      # start vllm service
      vllm serve Qwen/Qwen3-8B \
        --served-model-name Qwen/Qwen3-8B \
        --trust-remote-code \
        --no-enable-prefix-caching \
        --port 8000 \
        --tensor-parallel-size 1 \
        --max-model-len 10000 \
        --max-num-batched-tokens 40960 \
        --data-parallel-size 1 \
        --pipeline-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --kv-transfer-config '{"engine_id":"'$POD_NAME'","kv_connector":"MultiConnector","kv_connector_extra_config":{"connectors":[{"kv_buffer_device":"npu","kv_connector":"MooncakeConnectorV1","kv_connector_extra_config":{"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2},"use_ascend_direct":true},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"},{"kv_buffer_device":"npu","kv_connector":"AscendStoreConnector","kv_connector_extra_config":{"backend":"mooncake","decode":{"dp_size":1,"tp_size":1},"lookup_rpc_port":"0","prefill":{"dp_size":1,"tp_size":2}},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"}],"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2}},"kv_port":"20001","kv_rank":1,"kv_role":"kv_consumer"}'

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                    8
      huawei.com/Ascend310P:  1
      memory:                 64Gi
    Requests:
      cpu:                    4
      huawei.com/Ascend310P:  1
      memory:                 32Gi
    Liveness:                 http-get http://:decode-port/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Readiness:                http-get http://:decode-port/v1/models delay=0s timeout=5s period=10s #success=1 #failure=3
    Startup:                  http-get http://:decode-port/v1/models delay=30s timeout=5s period=30s #success=1 #failure=60
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
      POD_NAME:              vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg (v1:metadata.name)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /app from mooncake-config (ro)
      /dev/shm from shm (rw)
      /etc/ascend_install.info from installinfo (rw)
      /etc/hccn.conf from hccnconf (rw)
      /root/.cache from rootcache (rw)
      /usr/bin/hccn_tool from hccntool (rw)
      /usr/local/Ascend/driver/lib64 from lib64 (rw)
      /usr/local/Ascend/driver/version.info from version (rw)
      /usr/local/bin/npu-smi from npusmi (rw)
      /usr/local/dcmi from dcmi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  mooncake-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  
<unset>
  shm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  24Gi
  dcmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/dcmi
    HostPathType:  
  npusmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/npu-smi
    HostPathType:  File
  lib64:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/lib64
    HostPathType:  
  version:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/version.info
    HostPathType:  File
  installinfo:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ascend_install.info
    HostPathType:  File
  hccntool:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin/hccn_tool
    HostPathType:  File
  hccnconf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/hccn.conf
    HostPathType:  File
  rootcache:
    Type:          HostPath (bare host directory volume)
    Path:          /home/llm_cache
    HostPathType:  
  kube-api-access-jswfw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              
<none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  5d                      default-scheduler  0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  5d (x2 over 5d)         default-scheduler  0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  9m32s                   default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  9m20s (x24 over 9m29s)  default-scheduler  0/1 nodes are available: 1 Insufficient huawei.com/Ascend310P. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         8m58s                   default-scheduler  Successfully assigned ai-inference/vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg to master1
  Warning  FailedMount       44s (x12 over 8m58s)    kubelet            MountVolume.SetUp failed for volume "hccntool" : hostPath type check failed: /usr/bin/hccn_tool is not a file

暂时绕过：

touch /usr/bin/hccn_tool
chmod +x /usr/bin/hccn_tool

huggingface-download 失败

[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp 
Name:             vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp
Namespace:        ai-inference
Priority:         0
Service Account:  default
Node:             master1/10.17.30.131
Start Time:       Tue, 07 Apr 2026 09:45:19 +0800
Labels:           app.kubernetes.io/instance=infernex-vllm-pd-2p1d-01
                  app.kubernetes.io/name=inference-backend
                  openfuyao.com/dpSize=1
                  openfuyao.com/engine=vllm
                  openfuyao.com/model=qwen-qwen3-8b
                  openfuyao.com/pdGroupID=qwen3-8b-pd-01
                  openfuyao.com/pdRole=decode
                  openfuyao.com/ppSize=1
                  openfuyao.com/tpSize=1
                  pod-template-hash=6cd64bc69c
Annotations:      checksum/config: 476b32f01fc96ff2896aee7fce288cd2b58cdb2ac825d1a22518798806847a2c
                  cni.projectcalico.org/containerID: 32b3384131b69054ca45acc8afe5e272b0ca681ea6d0611b3fec7316e3532e80
                  cni.projectcalico.org/podIP: 192.168.137.155/32
                  cni.projectcalico.org/podIPs: 192.168.137.155/32
                  huawei.com/AscendReal: Ascend310P-0
                  huawei.com/kltDev: Ascend310P-0
Status:           Pending
IP:               192.168.137.155
IPs:
  IP:           192.168.137.155
Controlled By:  ReplicaSet/vllm-pd-2p1d-01-decode-6cd64bc69c
Init Containers:
  mooncake-config-init:
    Container ID:  containerd://4ede488dd17e33f3a980aee6fa4eac3093ad6366c3854fe85436f81f6e1df7bb
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image ID:      hub.oepkgs.net/openfuyao/mikefarah/yq@sha256:4facc66fdcc785ec961ef7f2185f53f862f462eefe1d50c2eb311c2bb26823e3
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/sh
      -c
    Args:
      set -e
      CONFIG_PATH="/app/mooncake.json"
      mkdir -p "$(dirname "$CONFIG_PATH")"
      cat > /tmp/mooncake_config.tpl << 'EOF'
        local_hostname: "$POD_IP"
        metadata_server: "redis://redis-service:6379"
        master_server_address: "mooncake-master-service:30089"
        device_name: ""
        protocol: "ascend"
        global_segment_size: 42949672960
        use_ascend_direct: true

      EOF
      POD_IP_VALUE="${POD_IP:-0.0.0.0}"
      sed "s/\$POD_IP/${POD_IP_VALUE}/g" /tmp/mooncake_config.tpl | yq eval - -o=json > "$CONFIG_PATH"

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 07 Apr 2026 09:45:20 +0800
      Finished:     Tue, 07 Apr 2026 09:45:20 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:  vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
    Mounts:
      /app from mooncake-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
  huggingface-download:
    Container ID:  containerd://84e15b7d9e2f7382181e309c1558174ec48e58ad0ae14f92ae0dfff284da76e5
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image ID:      cr.openfuyao.cn/openfuyao/huggingface-download@sha256:ac86348b5e6934a020c21c4f0ebf81b520194ba8e549f1847ecc7521b82d9a8d
    Port:          
<none>
    Host Port:     
<none>
    Command:
      hf
      download
      Qwen/Qwen3-8B
    State:          Running
      Started:      Tue, 07 Apr 2026 09:47:33 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 Apr 2026 09:45:20 +0800
      Finished:     Tue, 07 Apr 2026 09:47:32 +0800
    Ready:          False
    Restart Count:  1
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
    Mounts:
      /root/.cache from rootcache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
Containers:
  decode-engine:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0
    Image ID:      
    Port:          8000/TCP (decode-port)
    Host Port:     0/TCP (decode-port)
    Command:
      /bin/bash
      -c
    Args:
      # PHYSICAL_DEVICES stands for the physical devices assigned to the container, use for vllm ascend 0.10.x
      export PHYSICAL_DEVICES=$(ls /dev/davinci* 2>/dev/null | grep -o '[0-9]\+' | sort -n | paste -sd',' -)

      # start vllm service
      vllm serve Qwen/Qwen3-8B \
        --served-model-name Qwen/Qwen3-8B \
        --trust-remote-code \
        --no-enable-prefix-caching \
        --port 8000 \
        --tensor-parallel-size 1 \
        --max-model-len 10000 \
        --max-num-batched-tokens 40960 \
        --data-parallel-size 1 \
        --pipeline-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --kv-transfer-config '{"engine_id":"'$POD_NAME'","kv_connector":"MultiConnector","kv_connector_extra_config":{"connectors":[{"kv_buffer_device":"npu","kv_connector":"MooncakeConnectorV1","kv_connector_extra_config":{"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2},"use_ascend_direct":true},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"},{"kv_buffer_device":"npu","kv_connector":"AscendStoreConnector","kv_connector_extra_config":{"backend":"mooncake","decode":{"dp_size":1,"tp_size":1},"lookup_rpc_port":"0","prefill":{"dp_size":1,"tp_size":2}},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"}],"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2}},"kv_port":"20001","kv_rank":1,"kv_role":"kv_consumer"}'

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                    8
      huawei.com/Ascend310P:  1
      memory:                 64Gi
    Requests:
      cpu:                    4
      huawei.com/Ascend310P:  1
      memory:                 32Gi
    Liveness:                 http-get http://:decode-port/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Readiness:                http-get http://:decode-port/v1/models delay=0s timeout=5s period=10s #success=1 #failure=3
    Startup:                  http-get http://:decode-port/v1/models delay=30s timeout=5s period=30s #success=1 #failure=60
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
      POD_NAME:              vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp (v1:metadata.name)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /app from mooncake-config (ro)
      /dev/shm from shm (rw)
      /etc/ascend_install.info from installinfo (rw)
      /etc/hccn.conf from hccnconf (rw)
      /root/.cache from rootcache (rw)
      /usr/bin/hccn_tool from hccntool (rw)
      /usr/local/Ascend/driver/lib64 from lib64 (rw)
      /usr/local/Ascend/driver/version.info from version (rw)
      /usr/local/bin/npu-smi from npusmi (rw)
      /usr/local/dcmi from dcmi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  mooncake-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  
<unset>
  shm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  24Gi
  dcmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/dcmi
    HostPathType:  
  npusmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/npu-smi
    HostPathType:  File
  lib64:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/lib64
    HostPathType:  
  version:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/version.info
    HostPathType:  File
  installinfo:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ascend_install.info
    HostPathType:  File
  hccntool:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin/hccn_tool
    HostPathType:  File
  hccnconf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/hccn.conf
    HostPathType:  File
  rootcache:
    Type:          HostPath (bare host directory volume)
    Path:          /home/llm_cache
    HostPathType:  
  kube-api-access-9jvbd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              
<none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  3m20s               default-scheduler  0/1 nodes are available: 1 Insufficient huawei.com/Ascend310P. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Normal   Scheduled         2m15s               default-scheduler  Successfully assigned ai-inference/vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp to master1
  Normal   Pulled            2m14s               kubelet            Container image "hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1" already present on machine
  Normal   Created           2m14s               kubelet            Created container: mooncake-config-init
  Normal   Started           2m14s               kubelet            Started container mooncake-config-init
  Normal   Pulled            1s (x2 over 2m14s)  kubelet            Container image "cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2" already present on machine
  Normal   Created           1s (x2 over 2m14s)  kubelet            Created container: huggingface-download
  Normal   Started           1s (x2 over 2m14s)  kubelet            Started container huggingface-download

查看错误日志：

# 看当前这次的日志
kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download

# 看上一次失败的日志
kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download --previous


[root@master1 ~]# kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download
[root@master1 ~]# 
[root@master1 ~]# kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download --previous
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 240, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3285, in repo_info
    return method(
           ^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3020, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1053, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/hf", line 8, in 
<module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/hf.py", line 113, in main
    app()
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1152, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1135, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 795, in main
    return _main(
           ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 188, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1514, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/download.py", line 224, in download
    _print_result(run_download())
                  ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/download.py", line 185, in run_download
    return snapshot_download(
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 324, in snapshot_download
    raise LocalEntryNotFoundError(
huggingface_hub.errors.LocalEntryNotFoundError: Got: ConnectError: [Errno 101] Network is unreachable
An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.

确认是网络问题：节点无法访问 HuggingFace（Network is unreachable），而且本地也没有缓存。解决方案：换国内镜像源（推荐）在 Deployment 的 huggingface-download init container 里加一个环境变量：

env:
  - name: HF_ENDPOINT
    value: "https://hf-mirror.com"

最好给 decode-engine 也加上，否则报同样的错。

看日志可能没有任何输出

kubectl -n ai-inference logs deployments/vllm-pd-2p1d-01-decode huggingface-download  -f

此时查看 llm 目录大小即可，可以看到不断在变化：

$ watch -n 2 -d 'du -sh /home/llm_cache/'
426M    /home/llm_cache/

310p 运行报错

[root@master1 ~]# kubectl  -n ai-inference logs  vllm-pd-2p1d-01-decode-7d487c49cd-qw89v
Defaulted container "decode-engine" out of: decode-engine, mooncake-config-init (init), huggingface-download (init)
...
INFO 04-07 05:15:19 [__init__.py:217] Platform plugin ascend is activated
(EngineCore_DP0 pid=94) INFO 04-07 05:15:33 [ascend_config.py:55] Linear layer sharding enabled with config: None. Note: This feature works optimally with FLASHCOMM2 and DSA-CP enabled; using it without these features may result in significant performance degradation.
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] EngineCore failed to start.
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] Traceback (most recent call last):
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 59, in run_engine_core
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     super().__init__(
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self._init_executor()
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.driver_worker.init_worker(all_kwargs=[kwargs])
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 313, in init_worker
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.worker = worker_class(**kwargs)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                   ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker.py", line 116, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     check_ascend_device_type()
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/utils.py", line 708, in check_ascend_device_type
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     assert _ascend_device_type == cur_device_type, f"Current device type: {cur_device_type} does not match the installed version's device type: {_ascend_device_type}, please check your installation package."
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] AssertionError: Current device type: AscendDeviceType._310P does not match the installed version's device type: AscendDeviceType.A2, please check your installation package.
(EngineCore_DP0 pid=94) Process EngineCore_DP0:
(EngineCore_DP0 pid=94) Traceback (most recent call last):
(EngineCore_DP0 pid=94)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=94)     self.run()
(EngineCore_DP0 pid=94)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=94)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 72, in run_engine_core
(EngineCore_DP0 pid=94)     raise e
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 59, in run_engine_core
(EngineCore_DP0 pid=94)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=94)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=94)     super().__init__(
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=94)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=94)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=94)     self._init_executor()
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=94)     self.driver_worker.init_worker(all_kwargs=[kwargs])
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 313, in init_worker
(EngineCore_DP0 pid=94)     self.worker = worker_class(**kwargs)
(EngineCore_DP0 pid=94)                   ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker.py", line 116, in __init__
(EngineCore_DP0 pid=94)     check_ascend_device_type()
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/utils.py", line 708, in check_ascend_device_type
(EngineCore_DP0 pid=94)     assert _ascend_device_type == cur_device_type, f"Current device type: {cur_device_type} does not match the installed version's device type: {_ascend_device_type}, please check your installation package."
(EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) AssertionError: Current device type: AscendDeviceType._310P does not match the installed version's device type: AscendDeviceType.A2, please check your installation package.

经过排查是 hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0 镜像是针对 910 构建的，找到了官方说明

310p 应该使用带有 310p 后缀的镜像。在镜像仓库寻找后替换为 quay.io/ascend/vllm-ascend:v0.18.0rc1-310p-openeuler 尝试。

对比 sha256 发现 hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0 镜像是完全的 quay.io/ascend/vllm-ascend:v0.13.0 镜像，sha256 完全一致。

310p vllm-ascend 报错

但是替换为其他带有 310p 后缀的 image ，启动后会报错：

Every 1.0s: kubectl -n ai-inference get pod                                                            master1: Thu Apr  9 08:36:06 2026

NAME                                          READY   STATUS             RESTARTS        AGE
cache-indexer-deployment-65d5b449f6-x9l46     1/1     Running            0               17h
inference-gateway-istio-5f9b7d78f6-7kbrw      1/1     Running            26 (15h ago)    17h
infernex-epp-5cc456bd-4vvmv                   1/1     Running            0               17h
mooncake-master-deployment-74cc5666b7-fr4fq   1/1     Running            0               17h
redis-server-deployment-67566b9765-m66lc      1/1     Running            0               17h
vllm-pd-2p1d-01-decode-7687ccb7b-vg98n        0/1     CrashLoopBackOff   161 (31s ago)   16h
vllm-pd-2p1d-01-prefill-66f7564d7f-tdd84      0/1     Pending            0               43h
vllm-pd-2p1d-01-prefill-fd68f87cf-mhtdk       0/1     Pending            0               40h
vllm-pd-2p1d-01-proxy-7ff4f59865-h8xbw        1/1     Running            0               17h

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
(APIServer pid=1)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/kv_transfer/ascend_multi_connector.py", line 5, in 
<module>
(APIServer pid=1)     from vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_layerwise_connector import MooncakeLayerwiseConnector
(APIServer pid=1)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_layerwise_connector.py", line 25, in 
<module>
(APIServer pid=1)     from mooncake.engine import TransferEngine  # type: ignore
(APIServer pid=1)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ModuleNotFoundError: No module named 'mooncake'
(APIServer pid=1) [ERROR] 2026-04-08-08:58:03 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-7687ccb7b-vg98n | grep -i image:
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image:         quay.io/ascend/vllm-ascend:main-310p

针对该问题，openfuyao 给出的方案如下：

@tl.s InferNex在310P环境部署问题排查：

vllm-ascend的310P镜像没有加入mooncake，所以在prefill/decode之间kvcache数据传输无法支持。
https://github.com/vllm-project/vllm-ascend/blob/main/Dockerfile.310p

建议使用聚合模式部署，可参考InferNex聚合模式示例，部署时将`inference-backend.services[0].kvTransferConfig` 配置项删除，即可不使用mooncake相关能力：
https://gitcode.com/openFuyao/InferNex/blob/0.22.2/examples/vllm-aggregated-random-values.yaml

vllm-ascend针对310P在线推理文档：
https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

310P的推理还未验证过，可以尝试v0.13.0或者 v0.18.0rc1，这两个版本vllm官方有文档支持。

aggregated 模式卡个数错误

环境中只有一张 310p，但是默认申请两张，需要修改两处，分别是资源申请个数 1 ，以及 vllm 启动参数 tensor_parallel_size 设定为 1 。

resources:
          limits:
            cpu: "8"
            huawei.com/Ascend310P: "1"
            memory: 64Gi
          requests:
            cpu: "4"
            huawei.com/Ascend310P: "1"
            memory: 32Gi
...
  # start vllm service
          vllm serve Qwen/Qwen3-8B \
            --served-model-name Qwen/Qwen3-8B \
            --trust-remote-code \
            --enable-prefix-caching \
            --port 8000 \
            --tensor-parallel-size 1 \

bf16 数据类型报错

(EngineCore pid=35) [PID: 35] 2026-04-09-08:27:33.217.113 AclNN_Parameter_Error(EZ1001): Tensor self not implemented for DT_BFLOAT16, should be in dtype support list [DT_FLOAT,DT_FLOAT16,DT_INT8,DT_INT16,DT_INT32,DT_INT64,DT_UINT8,DT_BOOL,DT_DOUBLE,].
(EngineCore pid=35) 
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-09-08:27:49 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据 AI 说法，暂未查证：Ascend 310P 芯片不支持 bfloat16（bf16）数据类型，但 vLLM 在初始化 rotary embedding（RoPE）时使用了 torch.ones(…, dtype=torch.bfloat16)，导致 ACL 算子报错。 310P 的算子库支持的浮点类型只有 float32 和 float16，不包含 bf16。

通过参数强制使用 float16 --dtype half 解决：

bashvllm serve Qwen/Qwen3-8B \
  --dtype half \   # 强制使用 float16 而非 bfloat16
  ...其他参数

npu_dynamic_quant 算子报错

解决以上问题后 pod 可以运行更久

(EngineCore pid=35) INFO 04-09 08:43:11 [weight_utils.py:574] Time spent downloading weights for Qwen/Qwen3-8B: 1.077223 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:08<00:32,  8.07s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:16<00:24,  8.33s/it]

但最终依然 error ，跟踪报错信息如下：

(EngineCore pid=35) INFO 04-09 08:47:55 [default_loader.py:384] Loading weights took 35.12 seconds
(EngineCore pid=35) INFO 04-09 08:47:57 [model_runner_v1.py:2589] Loading model weights took 17.6043 GB
.(EngineCore pid=35) INFO 04-09 08:48:11 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/4de24ceb58/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=35) INFO 04-09 08:48:11 [backends.py:1048] Dynamo bytecode transform time: 13.07 s
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] EngineCore failed to start.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] Traceback (most recent call last):
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     super().__init__(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.model_runner.profile_run()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     super().profile_run()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return super()._dummy_run(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     outputs = self._model_forward(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 322, in forward
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 597, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     output = TorchCompileWithNoGuardsWrapper.__call__(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 182, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._call_with_optional_nvtx_range(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return callable_fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 845, in compile_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2196, in _call_user_compiler
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     raise BackendCompilerFailed(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2171, in _call_user_compiler
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     compiled_fn = compiler_fn(gm, example_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     compiled_gm = compiler_fn(gm, example_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/__init__.py", line 2437, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self.compiler_fn(model_, inputs_, **self.kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwds)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 1063, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.configure_post_pass()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 847, in configure_post_pass
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.pass_manager.configure(self.vllm_config)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/graph_fusion_pass_manager.py", line 55, in configure
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.passes.append(AddRMSNormQuantFusionPass(config))
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 493, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     AddRMSNormDynamicQuantPattern(vllm_config, eps=eps).register(self.pattern_match_passes)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/base_pattern.py", line 49, in register
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     pm.register_replacement(pattern_fn, replacement_fn, example_inputs, pm.fwd_only, pm_pass)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1552, in register_replacement
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     pattern, gm = gen_pattern_and_search_gm(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwds)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1760, in gen_pattern_and_search_gm
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     search_gm = trace_fn(search_fn, flat_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 2115, in fwd_only
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     gm = make_fx(fn, decompositions, tracing_mode="real")(*args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2429, in wrapped
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return make_fx_tracer.trace(f, *args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2356, in trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._trace_inner(f, *args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2318, in _trace_inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     t = dispatch_trace(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         ^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_compile.py", line 53, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return disable_fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1303, in dispatch_trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     graph = tracer.trace(root, concrete_args)  # type: ignore[arg-type]
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py", line 868, in trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     (self.create_arg(fn(*args)),),
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                      ^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1361, in wrapped
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     out = f(*tensors)  # type:ignore[call-arg]
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]           ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 300, in pattern
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     quantized_output = torch.ops.npu.npu_dynamic_quant(out0)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1409, in __torch_function__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_stats.py", line 28, in wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1534, in __torch_dispatch__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return proxy_call(self, func, self.pre_dispatch, args, kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 994, in proxy_call
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     out = func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0xfffec83c17d0>' raised:
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] RuntimeError: npu_dynamic_quant:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:82 NPU function error: call aclnnDynamicQuantV2 failed, error code is 561103
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] [ERROR] 2026-04-09-08:48:12 (PID:35, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] [PID: 35] 2026-04-09-08:48:12.623.243 AclNN_Parameter_Error(EZ1001): DynamicQuant launch kernel failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         TraceBack (most recent call last):
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Tiling failed
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Tiling Failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Kernel GetWorkspace failed. opType: 21
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         DynamicQuant launch kernel failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) Process EngineCore:
(EngineCore pid=35) Traceback (most recent call last):
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=35)     self.run()
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore pid=35)     self._target(*self._args, **self._kwargs)
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1103, in run_engine_core
(EngineCore pid=35)     raise e
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35)     super().__init__(
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35)     self.model_runner.profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35)     super().profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35)                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35)     return super()._dummy_run(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35)     outputs = self._model_forward(
(EngineCore pid=35)               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 322, in forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 597, in __call__
(EngineCore pid=35)     output = TorchCompileWithNoGuardsWrapper.__call__(
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 182, in __call__
(EngineCore pid=35)     return self._call_with_optional_nvtx_range(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(EngineCore pid=35)     return callable_fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 845, in compile_wrapper
(EngineCore pid=35)     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(EngineCore pid=35)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2196, in _call_user_compiler
(EngineCore pid=35)     raise BackendCompilerFailed(
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2171, in _call_user_compiler
(EngineCore pid=35)     compiled_fn = compiler_fn(gm, example_inputs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
(EngineCore pid=35)     compiled_gm = compiler_fn(gm, example_inputs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/__init__.py", line 2437, in __call__
(EngineCore pid=35)     return self.compiler_fn(model_, inputs_, **self.kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35)     return func(*args, **kwds)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 1063, in __call__
(EngineCore pid=35)     self.configure_post_pass()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 847, in configure_post_pass
(EngineCore pid=35)     self.pass_manager.configure(self.vllm_config)
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/graph_fusion_pass_manager.py", line 55, in configure
(EngineCore pid=35)     self.passes.append(AddRMSNormQuantFusionPass(config))
(EngineCore pid=35)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 493, in __init__
(EngineCore pid=35)     AddRMSNormDynamicQuantPattern(vllm_config, eps=eps).register(self.pattern_match_passes)
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/base_pattern.py", line 49, in register
(EngineCore pid=35)     pm.register_replacement(pattern_fn, replacement_fn, example_inputs, pm.fwd_only, pm_pass)
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1552, in register_replacement
(EngineCore pid=35)     pattern, gm = gen_pattern_and_search_gm(
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35)     return func(*args, **kwds)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1760, in gen_pattern_and_search_gm
(EngineCore pid=35)     search_gm = trace_fn(search_fn, flat_inputs)
(EngineCore pid=35)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 2115, in fwd_only
(EngineCore pid=35)     gm = make_fx(fn, decompositions, tracing_mode="real")(*args)
(EngineCore pid=35)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2429, in wrapped
(EngineCore pid=35)     return make_fx_tracer.trace(f, *args)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2356, in trace
(EngineCore pid=35)     return self._trace_inner(f, *args)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2318, in _trace_inner
(EngineCore pid=35)     t = dispatch_trace(
(EngineCore pid=35)         ^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_compile.py", line 53, in inner
(EngineCore pid=35)     return disable_fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1303, in dispatch_trace
(EngineCore pid=35)     graph = tracer.trace(root, concrete_args)  # type: ignore[arg-type]
(EngineCore pid=35)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py", line 868, in trace
(EngineCore pid=35)     (self.create_arg(fn(*args)),),
(EngineCore pid=35)                      ^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1361, in wrapped
(EngineCore pid=35)     out = f(*tensors)  # type:ignore[call-arg]
(EngineCore pid=35)           ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 300, in pattern
(EngineCore pid=35)     quantized_output = torch.ops.npu.npu_dynamic_quant(out0)
(EngineCore pid=35)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1409, in __torch_function__
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_stats.py", line 28, in wrapper
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1534, in __torch_dispatch__
(EngineCore pid=35)     return proxy_call(self, func, self.pre_dispatch, args, kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 994, in proxy_call
(EngineCore pid=35)     out = func(*args, **kwargs)
(EngineCore pid=35)           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0xfffec83c17d0>' raised:
(EngineCore pid=35) RuntimeError: npu_dynamic_quant:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:82 NPU function error: call aclnnDynamicQuantV2 failed, error code is 561103
(EngineCore pid=35) [ERROR] 2026-04-09-08:48:12 (PID:35, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(EngineCore pid=35) [PID: 35] 2026-04-09-08:48:12.623.243 AclNN_Parameter_Error(EZ1001): DynamicQuant launch kernel failed.
(EngineCore pid=35)         TraceBack (most recent call last):
(EngineCore pid=35)         Tiling failed
(EngineCore pid=35)         Tiling Failed.
(EngineCore pid=35)         Kernel GetWorkspace failed. opType: 21
(EngineCore pid=35)         DynamicQuant launch kernel failed.
(EngineCore pid=35) 
(EngineCore pid=35) 
(EngineCore pid=35) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore pid=35) 
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-09-08:48:30 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据 ai 的说法：根本原因：norm_quant 融合 Pass 在编译阶段向 310P 发起 npu_dynamic_quant 算子，而 310P 不支持该动态量化算子（或当前 CANN 版本不兼容），导致 Tiling 失败。

openfuyao 社区给出的回应如下：

@tl.s 看了一下报错日志，应该是310卡不支持vllm-ascend默认开启的算子 DynamicQuantV2，可以加上启动配置项 –enforce-eager 和 –no-quant 尝试一下。在InferNex中，默认未直接提供的vllm启动参数可以在 inference-backend.services[0].pd.prefill/decode.extraArgs添加。例如： extraArgs:

“–enforce-eager”

“–no-quant “

若还是不行，可以尝试更换模型，按照官方310P文档内的示例部署，如 Qwen2.5-7B-Instruct。 https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

OOM

[root@master1 fuyao-26.3-rc3]# kubectl  -n ai-inference logs deployments/vllm-pd-2p1d-01 -f 
Defaulted container "aggregated-engine" out of: aggregated-engine, huggingface-download (init)

INFO 04-13 06:21:29 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:21:29 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:21:29 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:21:29 [__init__.py:239] Platform plugin ascend is activated
INFO 04-13 06:21:42 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 04-13 06:21:42 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
WARNING 04-13 06:21:44 [__init__.py:80] The quantization method 'ascend' already exists and will be overwritten by the quantization config <class 'vllm_ascend._310p.quantization.modelslim_config.AscendModelSlimConfig310'>.
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297] 
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.18.0
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]   █▄█▀ █     █     █     █  model   Qwen/Qwen2.5-7B-Instruct
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297] 
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen2.5-7B-Instruct', 'model': 'Qwen/Qwen2.5-7B-Instruct', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['Qwen/Qwen2.5-7B-Instruct'], 'block_size': 128, 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': True, 'max_num_batched_tokens': 40960, 'kv_events_config': KVEventsConfig(enable_kv_cache_events=True, publisher='zmq', endpoint='tcp://*:5557', replay_endpoint=None, buffer_steps=10000, hwm=100000, max_queue_size=100000, topic='kv-events')}
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_ADDR
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_PORT_HTTP_API
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_PROTO
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_HOST
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_USE_V1
(APIServer pid=1) INFO 04-13 06:22:18 [model.py:533] Resolved architecture: Qwen2ForCausalLM
(APIServer pid=1) WARNING 04-13 06:22:18 [model.py:1920] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 04-13 06:22:18 [model.py:1582] Using max model len 4096
(APIServer pid=1) INFO 04-13 06:22:18 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=40960.
(APIServer pid=1) INFO 04-13 06:22:18 [vllm.py:754] Asynchronous scheduling is enabled.
(APIServer pid=1) WARNING 04-13 06:22:18 [vllm.py:788] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=1) WARNING 04-13 06:22:18 [vllm.py:799] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1) INFO 04-13 06:22:18 [vllm.py:964] Cudagraph is disabled under eager mode
(APIServer pid=1) WARNING 04-13 06:22:26 [platform.py:749] Parameter '--disable-cascade-attn' is a GPU-specific feature. Resetting to False for Ascend.
(APIServer pid=1) WARNING 04-13 06:22:26 [platform.py:838] Ignored parameter 'disable_flashinfer_prefill'. This is a GPU-specific feature not supported on Ascend. Resetting to False.
(APIServer pid=1) INFO 04-13 06:22:26 [ascend_config.py:425] Dynamic EPLB is False
(APIServer pid=1) INFO 04-13 06:22:26 [ascend_config.py:426] The number of redundant experts is 0
(APIServer pid=1) INFO 04-13 06:22:26 [platform.py:297] Compilation disabled, using eager mode by default
(APIServer pid=1) INFO 04-13 06:22:26 [platform.py:502] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
(APIServer pid=1) INFO 04-13 06:22:26 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant
INFO 04-13 06:22:53 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:22:53 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:22:53 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:22:53 [__init__.py:239] Platform plugin ascend is activated
(EngineCore pid=35) INFO 04-13 06:23:03 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
(EngineCore pid=35) INFO 04-13 06:23:03 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
(EngineCore pid=35) INFO 04-13 06:23:03 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='Qwen/Qwen2.5-7B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [40960], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=35) WARNING 04-13 06:23:08 [camem.py:66] Failed to import vllm_ascend_C:/vllm-workspace/vllm-ascend/vllm_ascend/vllm_ascend_C.cpython-311-aarch64-linux-gnu.so: undefined symbol: _ZN9pp_matmul17GetPpMatmulTilingERKNS_10MatMulInfoERKNS_12HardwareInfoERjRNS_18PpMatmulTilingDataE. Sleep mode will be disabled. 
(EngineCore pid=35) INFO 04-13 06:23:08 [ascend_config.py:425] Dynamic EPLB is False
(EngineCore pid=35) INFO 04-13 06:23:08 [ascend_config.py:426] The number of redundant experts is 0
INFO 04-13 06:23:22 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:23:22 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:23:22 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:23:22 [__init__.py:239] Platform plugin ascend is activated
....(EngineCore pid=35) INFO 04-13 06:24:35 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.137.164:42089 backend=hccl
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=35) INFO 04-13 06:24:36 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=35) WARNING 04-13 06:24:36 [worker.py:306] Bind cpus failed in rank0: Can not get running npu info. Skip binding cpu.
(EngineCore pid=35) INFO 04-13 06:24:37 [model_runner_v1.py:2562] Starting to load model Qwen/Qwen2.5-7B-Instruct...
(EngineCore pid=35) INFO 04-13 06:24:56 [weight_utils.py:574] Time spent downloading weights for Qwen/Qwen2.5-7B-Instruct: 4.228922 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:10<00:30, 10.12s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:20<00:20, 10.32s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:30<00:10, 10.21s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:40<00:00,  9.99s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:40<00:00, 10.08s/it]
(EngineCore pid=35) 
(EngineCore pid=35) INFO 04-13 06:25:46 [default_loader.py:384] Loading weights took 40.53 seconds
(EngineCore pid=35) INFO 04-13 06:25:48 [model_runner_v1.py:2589] Loading model weights took 16.2391 GB
.(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] EngineCore failed to start.
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] Traceback (most recent call last):
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     super().__init__(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     self.model_runner.profile_run()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     super().profile_run()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super()._dummy_run(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     outputs = self._model_forward(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 583, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 439, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self.forward(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 444, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states, residual = layer(positions, hidden_states, residual)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 311, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.mlp(hidden_states)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 114, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     gate_up, _ = self.gate_up_proj(x)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                  ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/linear.py", line 215, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super().forward(input_)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 582, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 228, in apply
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return dispatch_unquantized_gemm()(layer, x, layer.weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 55, in default_unquantized_gemm
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return torch.ops.vllm.unquantized_gemm(x, weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/parameter.py", line 126, in __torch_function__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super().__torch_function__(func, types, args, kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 27, in unquantized_gemm
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return torch.nn.functional.linear(x, weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] RuntimeError: NPU out of memory. Tried to allocate 2.89 GiB (NPU 0; 21.02 GiB total capacity; 17.36 GiB already allocated; 17.36 GiB current active; 2.15 GiB free; 17.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
(EngineCore pid=35) Process EngineCore:
(EngineCore pid=35) Traceback (most recent call last):
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=35)     self.run()
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore pid=35)     self._target(*self._args, **self._kwargs)
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1103, in run_engine_core
(EngineCore pid=35)     raise e
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35)     super().__init__(
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35)     self.model_runner.profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35)     super().profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35)                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35)     return super()._dummy_run(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35)     outputs = self._model_forward(
(EngineCore pid=35)               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 583, in forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 439, in __call__
(EngineCore pid=35)     return self.forward(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 444, in forward
(EngineCore pid=35)     hidden_states, residual = layer(positions, hidden_states, residual)
(EngineCore pid=35)                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 311, in forward
(EngineCore pid=35)     hidden_states = self.mlp(hidden_states)
(EngineCore pid=35)                     ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 114, in forward
(EngineCore pid=35)     gate_up, _ = self.gate_up_proj(x)
(EngineCore pid=35)                  ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/linear.py", line 215, in forward
(EngineCore pid=35)     return super().forward(input_)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 582, in forward
(EngineCore pid=35)     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 228, in apply
(EngineCore pid=35)     return dispatch_unquantized_gemm()(layer, x, layer.weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 55, in default_unquantized_gemm
(EngineCore pid=35)     return torch.ops.vllm.unquantized_gemm(x, weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/parameter.py", line 126, in __torch_function__
(EngineCore pid=35)     return super().__torch_function__(func, types, args, kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 27, in unquantized_gemm
(EngineCore pid=35)     return torch.nn.functional.linear(x, weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) RuntimeError: NPU out of memory. Tried to allocate 2.89 GiB (NPU 0; 21.02 GiB total capacity; 17.36 GiB already allocated; 17.36 GiB current active; 2.15 GiB free; 17.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-13-06:26:05 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据官网说明，似乎单张 310P 只能跑个 0.6B

Run the following script to start the vLLM server on NPU (Qwen3-0.6B:1 card, Qwen2.5-7B-Instruct:2 cards, Pangu-Pro-MoE-72B: 8 cards)

https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

而且需要一些参数

vllm serve Qwen/Qwen3-0.6B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --enforce-eager \
    --dtype float16

Helm Update 报错

每次更新 helm 都需要经历以下几个回合，才能成功：

[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: cannot patch "vllm-pd-2p1d-01" with kind Deployment: Deployment.apps "vllm-pd-2p1d-01" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"infernex-vllm-pd-2p1d-01","app.kubernetes.io/name":"inference-backend","openfuyao.com/dpSize":"1","openfuyao.com/engine":"vllm","openfuyao.com/model":"qwen-qwen3-0.6b","openfuyao.com/pdRole":"aggregate","openfuyao.com/tpSize":"2"}}: field is immutable
[root@master1 fuyao-26.3-rc3]# kubectl -n ai-inference delete deployments.apps vllm-pd-2p1d-01 
deployment.apps "vllm-pd-2p1d-01" deleted from ai-inference namespace
[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade infernex/charts/pd-orchestrator/charts/resourcescalinggroup/templates/webhook-wait-hook.yaml failed: 1 error occurred:
        * jobs.batch "infernex-resourcescalinggroup-wait-webhook" is forbidden: unable to create new content in namespace scaling-system because it is being terminated

[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: failed to create resource: namespaces "scaling-system" not found
[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 

Release "infernex" has been upgraded. Happy Helming!
NAME: infernex
LAST DEPLOYED: Mon Apr 13 14:33:47 2026
NAMESPACE: ai-inference
STATUS: deployed
REVISION: 27
TEST SUITE: None

istio httproute 错误

[root@master1 fuyao-26.3-rc3]# kubectl  -n ai-inference describe httproutes.gateway.networking.k8s.io qwen-qwen3-0.6b-httproute 
Name:         qwen-qwen3-0.6b-httproute
Namespace:    ai-inference
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=infernex-epp
              app.kubernetes.io/version=0.21.0
Annotations:  meta.helm.sh/release-name: infernex
              meta.helm.sh/release-namespace: ai-inference
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2026-04-13T06:31:19Z
  Generation:          1
  Resource Version:    3664436
  UID:                 21dca6e8-483a-4b03-8a78-82559c45a7e3
Spec:
  Parent Refs:
    Group:  gateway.networking.k8s.io
    Kind:   Gateway
    Name:   inference-gateway
  Rules:
    Backend Refs:
      Group:   inference.networking.k8s.io
      Kind:    InferencePool
      Name:    qwen-qwen3-0.6b
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
    Timeouts:
      Request:  300s
Status:
  Parents:
    Conditions:
      Last Transition Time:  2026-04-13T06:31:19Z
      Message:               Route was valid
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-04-13T06:31:19Z
      Message:               InferencePool.Name invalid; the name of the InferencePool must be used, not the hostname.
      Observed Generation:   1
      Reason:                InvalidDestination
      Status:                False
      Type:                  ResolvedRefs
    Controller Name:         istio.io/gateway-controller
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   inference-gateway
Events:       
<none>

总结

helm 默认申请 2 张 310P ，需手动修改

resources:
limits:
  cpu: "8"
  huawei.com/Ascend310P: "1"
  memory: 64Gi
requests:
  cpu: "4"
  huawei.com/Ascend310P: "1"
  memory: 32Gi

huggerface 下载需手动配置国内源

- name: HF_ENDPOINT
  value: https://hf-mirror.com

vllm 启动参数需手动调整

vllm serve Qwen/Qwen3-0.6B \
--served-model-name Qwen/Qwen3-0.6B \
--trust-remote-code \
--enable-prefix-caching \
--port 8000 \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--enforce-eager \
--dtype float16 \
--max-num-batched-tokens 40960 \
--data-parallel-size 1 \
--gpu-memory-utilization 0.8 \
--block-size 128 \
--kv-events-config '{"enable_kv_cache_events": true, "publisher":"zmq", "topic":"kv-events"}'

默认服务采用 ClusterAPI 使用 Gateway istio 暴露服务，目前还没有正常工作
vllm pod 看到频繁打印

(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK

Refs

Ascend 310P + openFuyao + NPU-Operator 故障排查

发表评论

[TOC]

故障 pod describe

[root@master1 ~]# kubectl -n kube-system describe pod ascend-device-plugin-ll46f 
Name:                 ascend-device-plugin-ll46f
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      ascend-device-plugin-sa
Node:                 master1/10.17.30.131
Start Time:           Mon, 30 Mar 2026 11:08:32 +0800
Labels:               app.kubernetes.io/managed-by=npu-operator
                      controller-revision-hash=7df5dcb887
                      helm.sh/chart=npu-operator-0.15.0
                      name=ascend-device-plugin-ds
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/containerID: c1f2adcaeaaf2bdcf0a6e09730f68231a293074e31d58f61997f714dfb520878
                      cni.projectcalico.org/podIP: 192.168.137.118/32
                      cni.projectcalico.org/podIPs: 192.168.137.118/32
                      scheduler.alpha.kubernetes.io/critical-pod: 
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Running
IP:                   192.168.137.118
IPs:
  IP:           192.168.137.118
Controlled By:  DaemonSet/ascend-device-plugin
Init Containers:
  init-permission:
    Container ID:  containerd://4406968a522bea48dfefebae81ec53644312762af4781c25de689952ed6c2d27
    Image:         cr.openfuyao.cn/openfuyao/busybox:1.36.1
    Image ID:      cr.openfuyao.cn/openfuyao/busybox@sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176
    Port:          
<none>
    Host Port:     
<none>
    Command:
      sh
      -c
      chown 9000:9000 /var/log/mindx-dl /var/log/mindx-dl/devicePlugin
      chmod 750 /var/log/mindx-dl/devicePlugin

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Mar 2026 15:28:32 +0800
      Finished:     Mon, 30 Mar 2026 15:28:32 +0800
    Ready:          True
    Restart Count:  1
    Environment:    
<none>
    Mounts:
      /var/log/mindx-dl/devicePlugin from log-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfldg (ro)
Containers:
  device-plugin-01:
    Container ID:  containerd://fcc0c4742285847e2621a9a9217502307fc7e28644fbf86b32f9c11d67a2c0ab
    Image:         cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0
    Image ID:      cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin@sha256:a5b9612b21bcd35384f9f19a05b2d7915b865e7b2be6a30bfd7806a9b8a86f58
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/bash
      -c
      --
    Args:
      device-plugin  -useAscendDocker=true -volcanoType=false -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log -logLevel=0
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 31 Mar 2026 10:28:58 +0800
      Finished:     Tue, 31 Mar 2026 10:28:58 +0800
    Ready:          False
    Restart Count:  274
    Limits:
      cpu:     500m
      memory:  500Mi
    Requests:
      cpu:     500m
      memory:  500Mi
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /tmp from tmp (rw)
      /usr/local/Ascend/driver from hiai-driver (ro)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/lib/kubelet/pod-resources from pod-resource (rw)
      /var/log/mindx-dl/devicePlugin from log-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfldg (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:  
  pod-resource:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pod-resources
    HostPathType:  
  hiai-driver:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver
    HostPathType:  
  log-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/mindx-dl/devicePlugin
    HostPathType:  DirectoryOrCreate
  tmp:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp
    HostPathType:  
  kube-api-access-gfldg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              openfuyao.com/npu.present=
Tolerations:                 CriticalAddonsOnly op=Exists
                             device-plugin=v2:NoSchedule
                             huawei.com/Ascend910:NoSchedule op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulled   16m (x205 over 18h)     kubelet  (combined from similar events): Successfully pulled image "cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0" in 403ms (403ms including waiting). Image size: 48017174 bytes.
  Warning  BackOff  2m47s (x5216 over 18h)  kubelet  Back-off restarting failed container device-plugin-01 in pod ascend-device-plugin-ll46f_kube-system(8edcd384-ab2d-4998-8077-5ac58801c79e)
  Normal   Pulling  66s (x227 over 19h)     kubelet  Pulling image "cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0"

故障 pod /dev 检查

[root@master1 fuyao-26.3-rc3]# kubectl  -n kube-system exec -it daemonsets/ascend-device-plugin -- ls /dev
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
autofs           null               tty10  tty34  tty58    vcs5
bsg              ppp                tty11  tty35  tty59    vcs6
btrfs-control    ptmx               tty12  tty36  tty6     vcsa
bus              pts                tty13  tty37  tty60    vcsa1
core             random             tty14  tty38  tty61    vcsa2
cpu_dma_latency  raw                tty15  tty39  tty62    vcsa3
cuse             relationship_ctrl  tty16  tty4   tty63    vcsa4
davinci0         rfkill             tty17  tty40  tty7     vcsa5
davinci_manager  rtc0               tty18  tty41  tty8     vcsa6
devmm_svm        sda                tty19  tty42  tty9     vcsu
dri              sda1               tty2   tty43  ttyAMA0  vcsu1
fb0              sda2               tty20  tty44  ttyS0    vcsu2
fd               sg0                tty21  tty45  ttyS1    vcsu3
full             sg1                tty22  tty46  ttyS2    vcsu4
fuse             sg2                tty23  tty47  ttyS3    vcsu5
hidraw0          shm                tty24  tty48  uhid     vcsu6
hidraw1          snapshot           tty25  tty49  uinput   vfio
hisi_hdc         sr0                tty26  tty5   urandom  vga_arbiter
hwrng            sr1                tty27  tty50  usbmon0  vhost-net
input            stderr             tty28  tty51  usbmon1  vhost-vsock
kmsg             stdin              tty29  tty52  usbmon2  vport2p1
loop-control     stdout             tty3   tty53  vcs      zero
mapper           termination-log    tty30  tty54  vcs1
mem              tty                tty31  tty55  vcs2
mqueue           tty0               tty32  tty56  vcs3
net              tty1               tty33  tty57  vcs4

故障 pod 驱动检查

[root@master1 fuyao-26.3-rc3]# kubectl  -n kube-system exec -it daemonsets/ascend-device-plugin -- ls -lha /usr/local/Ascend/driver
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
total 44K
drwxr-xr-x  8 root root 4.0K Mar 27 08:03 .
drwxr-xr-x  3 root root 4.0K Mar 31 02:34 ..
drwxr-xr-x  2 root root 4.0K Mar 27 08:01 bin
-r--r--r--  1 root root   20 Mar 27 08:01 build.info
dr-xr-x---  2 root root 4.0K Mar 27 08:01 device
dr-x------ 41 root root 4.0K Mar 27 08:01 kernel
drwxr-xr-x  6 root root 4.0K Mar 27 08:01 lib64
-r--r-----  1 root root   56 Mar 27 08:01 scene.info
dr-xr-x---  2 root root 4.0K Mar 27 08:01 script
drwxr-xr-x  2 root root 4.0K Mar 27 08:01 tools
-r--r--r--  1 root root  352 Mar 27 08:03 version.info

故障 pod 日志

[root@master1 ~]# kubectl -n kube-system logs daemonsets/ascend-device-plugin --previous
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
[INFO]     2026/03/31 06:46:54.593254 1       hwlog/api.go:108    devicePlugin.log's logger init success
[INFO]     2026/03/31 06:46:54.593449 1       main.go:187    ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO]     2026/03/31 06:46:54.593494 1       main.go:188    ascend device plugin starting scene is center
[INFO]     2026/03/31 06:46:54.787930 1       devmanager/devmanager.go:104    the dcmi version is 24.1.rc3
[ERROR]    2026/03/31 06:46:54.788019 1       devmanager/devmanager.go:211    get error card quantity: 0
[ERROR]    2026/03/31 06:46:54.788052 1       devmanager/devmanager.go:195    get card list failed for init
[ERROR]    2026/03/31 06:46:54.788101 1       main.go:203    init devmanager failed, err: auto init failed, err: get card list failed for init

故障 pod 驱动检查

[root@master1 ~]# kubectl -n kube-system exec -it daemonsets/ascend-device-plugin -- bash -c 'find /usr/local/Ascend/driver -name libdcmi.so 2>/dev/null; echo $LD_LIBRARY_PATH'
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
/usr/local/Ascend/driver/lib64/driver/libdcmi.so
command terminated with exit code 137
[root@master1 ~]# ps -ef | grep -E 'dmp_daemon|slogd' | grep -v grep
root       21578       1  0 Mar30 ?        00:00:19 /usr/sbin/rsyslogd -n -i/var/run/rsyslogd.pid

检查服务状态？

[root@master1 ~]# systemctl status ascend-dmi
Unit ascend-dmi.service could not be found.
[root@master1 ~]# systemctl status ascend-dkms
Unit ascend-dkms.service could not be found.
[root@master1 ~]# systemctl status npu-smi
Unit npu-smi.service could not be found.
[root@master1 ~]# find / -name dmp_daemon 2>/dev/null
[root@master1 ~]# find / -name slogd 2>/dev/null
[root@master1 ~]# ls -l /var/dmp_daemon /var/slogd 2>/dev/null
[root@master1 ~]#

dcmi 问题，需硬件排查

检查卡获取

#include <stdlib.h>
#include <stdio.h>
#include "dcmi_interface_api.h"

int my_get_card_list();

int main(int argc,char *argv[])
{
    my_get_card_list();
    return 0;
}

int my_get_card_list()
{
    printf("\n==================================card id info list=========================\n");
    dcmi_init();
    int card_num = 0;
    int card_list[16] = {0};
    int ret = dcmi_get_card_list(&card_num, card_list, 16);
    if (ret != DCMI_OK) {
        printf("dcmi get card list failed ret=%d\n", ret);
    }
    printf("card_num=%d, card_list:[",card_num);
    for (int i = 0; i < card_num; i++) {
        printf("%d ", card_list[i]);
    }
}

cc ./test1.c -o test1 -I /usr/local/dcmi -L /usr/local/dcmi -ldcmi

-I头文件（.h）搜索路径
-L库文件（.so/.a）搜索路径
-l链接的库名（去掉 lib 前缀）

nerdctl run --rm \
  -v /usr/local/Ascend:/usr/local/Ascend \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v $(pwd):/build \
  ubuntu:18.04 bash -c "
    sed -i -e 's@http*://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g' \
           -e 's@http*://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g' \
           /etc/apt/sources.list
    apt update && apt install -y gcc
    cd /build
    cc ./test1.c -o test1 \
      -I /usr/local/dcmi \
      -L /usr/local/dcmi \
      -L /usr/local/Ascend/driver/lib64/common \
      -L /usr/local/Ascend/driver/lib64/driver \
      -ldcmi \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/common \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/driver \
      -Wl,-rpath,/usr/local/dcmi
  "

分析二进制：

[root@master1 ascend_debug]# ldd ./test1 | grep -i dcmi
        libdcmi.so => /usr/local/Ascend/driver/lib64/driver/libdcmi.so (0x0000ffffa6dd0000)
[root@master1 ascend_debug]# LD_DEBUG=libs ./test1 2>&1 | grep -i dcmi
    284830:     find library=libdcmi.so [0]; searching
    284830:      search path=/usr/local/Ascend/driver/lib64/common/tls/aarch64/atomics:/usr/local/Ascend/driver/lib64/common/tls/aarch64:/usr/local/Ascend/driver/lib64/common/tls/atomics:/usr/local/Ascend/driver/lib64/common/tls:/usr/local/Ascend/driver/lib64/common/aarch64/atomics:/usr/local/Ascend/driver/lib64/common/aarch64:/usr/local/Ascend/driver/lib64/common/atomics:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver/tls/aarch64/atomics:/usr/local/Ascend/driver/lib64/driver/tls/aarch64:/usr/local/Ascend/driver/lib64/driver/tls/atomics:/usr/local/Ascend/driver/lib64/driver/tls:/usr/local/Ascend/driver/lib64/driver/aarch64/atomics:/usr/local/Ascend/driver/lib64/driver/aarch64:/usr/local/Ascend/driver/lib64/driver/atomics:/usr/local/Ascend/driver/lib64/driver:/usr/local/dcmi/tls/aarch64/atomics:/usr/local/dcmi/tls/aarch64:/usr/local/dcmi/tls/atomics:/usr/local/dcmi/tls:/usr/local/dcmi/aarch64/atomics:/usr/local/dcmi/aarch64:/usr/local/dcmi/atomics:/usr/local/dcmi            (RUNPATH from file ./test1)
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/libdcmi.so
    284830:      search path=/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/dcmi/tls/aarch64/atomics:/usr/local/dcmi/tls/aarch64:/usr/local/dcmi/tls/atomics:/usr/local/dcmi/tls:/usr/local/dcmi/aarch64/atomics:/usr/local/dcmi/aarch64:/usr/local/dcmi/atomics:/usr/local/dcmi          (RUNPATH from file ./test1)
    284830:       trying file=/usr/local/dcmi/tls/aarch64/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/aarch64/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/libc.so.6
    284830:       trying file=/usr/local/dcmi/aarch64/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/aarch64/libc.so.6
    284830:       trying file=/usr/local/dcmi/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/libc.so.6
    284830:     calling init: /usr/local/Ascend/driver/lib64/driver/libdcmi.so
    284830:     calling fini: /usr/local/Ascend/driver/lib64/driver/libdcmi.so [0]
[root@master1 ascend_debug]# readlink -f /usr/local/dcmi/libdcmi.so
/usr/local/dcmi/libdcmi.so
[root@master1 ascend_debug]# readlink -f /usr/local/Ascend/driver/lib64/driver/libdcmi.so
/usr/local/Ascend/driver/lib64/driver/libdcmi.so
[root@master1 ascend_debug]# sha256sum /usr/local/dcmi/libdcmi.so /usr/local/Ascend/driver/lib64/driver/libdcmi.so
13a38cae84bad0f06367ff9280016e372c0608ca16465b5ae5f000d3844ee401  /usr/local/dcmi/libdcmi.so
13a38cae84bad0f06367ff9280016e372c0608ca16465b5ae5f000d3844ee401  /usr/local/Ascend/driver/lib64/driver/libdcmi.so

跟踪 strace

宿主机跑
strace -f -o /tmp/host.strace -e trace=file,ioctl ./test1
容器里跑
strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

新增挂载继续跟踪

volumeMounts:
        - name: hdc-basic
          mountPath: /etc/hdcBasic.cfg
          readOnly: true
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
      volumes:
      - name: hdc-basic
        hostPath:
          path: /etc/hdcBasic.cfg
          type: File
      - name: localtime
        hostPath:
          path: /etc/localtime
          type: File

跟踪并查看日志

kubectl -n kube-system exec -it ascend-device-plugin-69q5t -c device-plugin-01 -- bash

strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

root@ascend-device-plugin-69q5t:/tmp# strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

==================================card id info list=========================
card_num=0, card_list:[

root@ascend-device-plugin-69q5t:/tmp# cat /var/log/nputools_LOG_INFO.log > /tmp/nputools_LOG_INFO.log
root@ascend-device-plugin-69q5t:/tmp# cat /var/log/nputools_LOG_ERR.log > /tmp/nputools_LOG_ERR.log
cat: /var/log/nputools_LOG_ERR.log: No such file or directory
root@ascend-device-plugin-69q5t:/tmp# cat /tmp/nputools_LOG_INFO.log 
[2026/04/01 11:18:12][0583][root][127.0.0.1][dcmi_api.c,dcmi_board_init,86]:dcmi board init success. device_count=1.
[2026/04/01 11:18:12][0583][root][127.0.0.1][dcmi_api.c,dcmi_init,119]:dcmi init all success.

检查代码2

#include <stdio.h>
#include <stdlib.h>
#include "dcmi_interface_api.h"

#ifndef DCMI_OK
#define DCMI_OK 0
#endif

/* 头文件里没看到这个声明，手动补一个 */
extern int dcmi_get_card_num_list(int *card_num, int *card_list, int list_length);

static void print_list(const char *name, int ret, int num, int *list) {
    printf("%s ret=%d num=%d list=[", name, ret, num);
    for (int i = 0; i < num; ++i) {
        printf("%d ", list[i]);
    }
    printf("]\n");
}

int main(void) {
    int ret = dcmi_init();
    printf("dcmi_init ret=%d\n", ret);
    if (ret != DCMI_OK) {
        return 1;
    }

    int card_num = 0;
    int card_list[16] = {0};

    ret = dcmi_get_card_list(&card_num, card_list, 16);
    print_list("dcmi_get_card_list", ret, card_num, card_list);

    int card_num2 = 0;
    int card_list2[16] = {0};
    ret = dcmi_get_card_num_list(&card_num2, card_list2, 16);
    print_list("dcmi_get_card_num_list", ret, card_num2, card_list2);

    for (int i = 0; i < card_num && i < 16; ++i) {
        int dev_num = -1;
        ret = dcmi_get_device_num_in_card(card_list[i], &dev_num);
        printf("dcmi_get_device_num_in_card card=%d ret=%d dev_num=%d\n",
               card_list[i], ret, dev_num);
    }

    return 0;
}

主机编译

cc ./test2.c -o test2 -I /usr/local/dcmi -L /usr/local/dcmi -ldcmi

容器编译

nerdctl run --rm \
  -v /usr/local/Ascend:/usr/local/Ascend \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v $(pwd):/build \
  ubuntu:18.04 bash -c "
    sed -i -e 's@http*://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g' \
           -e 's@http*://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g' \
           /etc/apt/sources.list
    apt update && apt install -y gcc
    cd /build
    cc ./test2.c -o test2 \
      -I /usr/local/dcmi \
      -L /usr/local/dcmi \
      -L /usr/local/Ascend/driver/lib64/common \
      -L /usr/local/Ascend/driver/lib64/driver \
      -ldcmi \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/common \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/driver \
      -Wl,-rpath,/usr/local/dcmi
  "

拷入容器运行

kubectl -n kube-system cp ./test2 ascend-device-plugin-69q5t:/tmp/

# 主机运行
[root@master1 ascend_debug]# ./test2
dcmi_init ret=0
dcmi_get_card_list ret=0 num=1 list=[176 ]
dcmi_get_card_num_list ret=0 num=1 list=[176 ]
dcmi_get_device_num_in_card card=176 ret=0 dev_num=1

# 容器运行
root@ascend-device-plugin-69q5t:/tmp# ./test2 
dcmi_init ret=0
dcmi_get_card_list ret=0 num=0 list=[]
dcmi_get_card_num_list ret=0 num=0 list=[]

虚拟机场景

经过许老师认真定位，最终发现是因为非裸金属环境。虚拟机场景需要定制镜像。

根据官网文档：

如果在虚拟机场景下部署Ascend Device Plugin，需要在Ascend Device Plugin的镜像中安装systemd，推荐在Dockerfile中加入RUN apt-get update && apt-get install -y systemd命令进行安装。

为了使用 nerdctl 构建镜像首先安装 buildkit

wegt https://github.com/moby/buildkit/releases/download/v0.29.0/buildkit-v0.29.0.linux-arm64.tar.gz
tar zxvf buildkit-v0.29.0.linux-arm64.tar.gz
cp bin/* /usr/local/bin/

之后找一个新终端启动 buildkit ，这里是为了 nerdctl 构建 image, 如果不需要则不用启动。

buildkitd --oci-worker=false --containerd-worker=true --containerd-worker-namespace=k8s.io

Dockerfile 如下：

镜像源部分按需修改


FROM hub.oepkgs.net/openfuyao/ascendhub/ascend-k8sdeviceplugin:v6.0.0

替换 apt 镜像源

RUN sed -i \ -e ‘s@http://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g‘ \ -e ‘s@http://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g‘ \ /etc/apt/sources.list

安装 systemd

RUN apt-get update && \ apt-get install -y –no-install-recommends systemd systemd-sysv && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*

设置 systemd 为 init

STOPSIGNAL SIGRTMIN+3


> STOPSIGNAL SIGRTMIN+3 是告诉容器运行时（containerd/docker）停止这个容器时应该发送哪个信号。
> 如果你的容器里 不跑 systemd 作为主进程（比如 entrypoint 是业务程序），这行可以删掉，没有任何作用。
> 如果确实用 systemd 管理容器内服务，保留它能避免 kubectl delete pod 时等待 30 秒超时再强杀的问题。

构建命令如下：

nerdctl build \ –namespace k8s.io \ -t hub.oepkgs.net/openfuyao/ascendhub/ascend-k8sdeviceplugin:v6.0.0-systemd \ -f Dockerfile \ .


之后将出问题的镜像替换为新构建的镜像即可。

npu-operator 有同样的问题，一样修改即可。

## 修复确认
> 最终在 node 中能看到 npu 资源即成功。

[root@master1 ~]# kubectl describe node master1 Name: master1 Roles: control-plane,master,node,worker Labels: accelerator=huawei-Ascend310P beta.kubernetes.io/arch=arm64 beta.kubernetes.io/os=linux … servertype=Ascend310P-8 workerselector=dls-worker-node Annotations: baseDeviceInfos: {“Ascend310P-0”:{“IP”:””,”SuperDeviceID”:0}} … Capacity: cpu: 16 ephemeral-storage: 129724184Ki huawei.com/Ascend310P: 1 hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 32595632Ki pods: 110 Allocatable: cpu: 16 ephemeral-storage: 119553807777 huawei.com/Ascend310P: 1 hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 32493232Ki pods: 110 … Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits

cpu 15565m (97%) 16910m (105%) memory 17492Mi (55%) 30900Mi (97%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) hugepages-32Mi 0 (0%) 0 (0%) hugepages-64Ki 0 (0%) 0 (0%) huawei.com/Ascend310P 0 0 …

KDE Plasma6 禁用全局菜单，恢复正常应用菜单

发表评论

前情提要

不知道从什么时候开始，KDE Plasma 默认启用类似 macOS 的全局应用菜单。

即应用窗口标题栏下方不显示菜单，而是移动到顶部菜单栏中“全局菜单”小组件中。

但问题是，Linux 桌面生态生态复杂，X11 Wayland Qt GTK 等等技术太过复杂，很难保证常用软件都能够正常显示全局菜单。

比如我最近在使用 Joplin ，就发现除了菜单栏根本找不到任何入口。

于是搜索了一番后，终于找到了关闭全局菜单，恢复正常的应用菜单的方法。

恢复方法

第一步：移除“全局菜单组件”

Edit Mode > Add or Manage Widgets > Global Menu > Remove all instances (button in the top right corner of the widget)

进入编辑模式，添加或管理组件，删除“全局菜单”小组间

第二步：移除应用菜单

实测完成第一步即可，第二步按照自己实际情况决定是否要做。

Settings > Colors & Themes > Window Decorations > Configure Titlebar Buttons…

There remove the Application Menu (“Hamburger”) button from your titlebars.

进入设置，颜色与主题，窗口装饰元素，配置菜单栏按钮，将“应用菜单”按钮移除，点击应用，即可。

第三步：重启应用

此时应该就能看到应用菜单了，如果看不到再重启一下即可。

Refs

Disable global menus, restore “normal” menus in apps

终极指南：在 Linux 裸机服务器上快速部署 Moltbot (原 Clawbot) 并集成飞书

发表评论

引言

2026 年初，一个名为 Moltbot（前身为 Clawbot）的开源 AI 代理（Agent）框架席卷了开发者社区。该框架允许用户将强大的 AI 模型（例如 OpenAI 的 GPT 系列、Anthropic 的 Claude 等）与 WhatsApp、Telegram、Discord 等日常通讯工具集成，从而通过聊天即可操控电脑、执行任务、获取信息。特斯拉前 AI 主管 Andrej Karpathy 的推荐更是使其迅速走红，其 GitHub 项目在短时间内获得了超过 60,000 个星标。

本指南将提供一份以安全为核心的详尽教程，旨在引导您逐步在 Linux 裸机服务器上安全地完成 Moltbot 的部署。

准备工作

在开始之前，请确保你已具备以下条件：

项目	要求	说明
Linux 服务器	Ubuntu 22.04+ 或其他现代发行版	推荐至少 2GB RAM 和 2 核 CPU。
基础 Linux 知识	熟悉 SSH、命令行操作	本教程将提供所有需要的命令。

第一步：准备服务器环境

基于安全最佳实践，我们强烈建议不要直接使用 root 用户运行任何应用程序。因此，第一步是创建一个专用的非特权用户，并为其授予 sudo 权限。

以 root 用户 SSH 登录你的服务器：

ssh root@your_server_ip

创建一个新用户（我们称之为 moltbot）：

adduser moltbot

系统会提示你为新用户设置密码和其他信息。

将新用户添加到 sudo 组，以便执行需要管理员权限的操作：

usermod -aG sudo moltbot

切换到新用户：

su - moltbot

更新系统软件包：

sudo apt update && sudo apt upgrade -y

此后，所有操作都应在该 moltbot 用户下进行。

第二步：安装 Moltbot

Moltbot 的核心依赖是 Node.js (版本 ≥ 22)。官方提供了一个便捷的一键安装脚本，可自动处理相关依赖并完成 Moltbot 的安装。

执行官方安装脚本：

curl -fsSL https://molt.bot/install.sh | bash

该脚本会自动检测您的操作系统，安装适配的 Node.js 版本，并部署 Moltbot 命令行工具（CLI）。

安装成功后，将自动进入交互式配置流程，如下图所示：

第一个选项是风险提示，输入 yes 确认已知晓。

第二步在模式选择中，建议选择 QuickStart 以进行快速配置。

第三步：配置使用 Qwen

在选择模型服务商时，本教程以阿里云的 Qwen（通义千问）为例，其提供充足的免费额度，非常适合初学者入门。待您熟悉系统后，可根据需求更换为其他模型。

选择 Qwen 模型后，终端会提供一个授权链接，请复制并在浏览器中打开。

打开链接后，您会看到授权页面。如果您尚未登录阿里云账户，请根据提示完成登录。

登录成功后，系统会提示您选择具体的 Qwen 模型版本。

此处选择默认模型即可。随后，安装向导会询问是否配置频道（Channel），建议暂时跳过，后续再进行配置。

接着，在技能（Skills）配置步骤中，同样选择 No。

等待安装流程结束，在最后的启动方式选择中，选择 TUI（文本用户界面）。

如果成功进入 TUI 聊天界面，说明 Moltbot 已正确安装。您可以输入 Hello 进行初步测试。

测试无误后，使用 Ctrl+C 组合键退出 TUI 界面，以便进行后续的后台服务配置。

第四步：配置 Moltbot 并启动 Gateway

检查 Gateway 服务状态：

安装向导会自动将 Gateway 配置为系统服务并在后台启动。

clawdbot gateway status

如果服务正常运行，您将看到 active (running) 的状态提示。

服务状态

如果检查状态时发现服务异常（例如 inactive 或 failed），可以尝试使用内置的诊断工具进行修复：

clawdbot doctor --repair

修复后，再次检查状态，应可看到服务已恢复正常。

服务状态

配置选项：

您可以根据需要编辑 Moltbot 的主配置文件。

使用文本编辑器（如 vim 或 nano）打开配置文件：

sudo vim ~/.clawdbot/clawdbot.json

重载并重启服务：

每次修改配置文件后，需要重载服务以使更改生效。虽然某些版本的 Moltbot 支持自动重载，但手动重启是更可靠的方式。

systemctl --user daemon-reload
clawdbot gateway restart

至此，Moltbot Gateway 已在后台稳定运行，并监听本地端口，等待连接。

第五步：访问后台

Moltbot 自带一个 Web UI 用于管理和测试。运行以下命令可获取访问方式：

clawdbot dashboard

命令会输出一段 SSH 端口转发指令，格式如下：

后台访问方法

在您的本地电脑（而非服务器）上打开一个终端，并执行上述 SSH 命令。该命令会将服务器的 Web UI 端口 18789 映射到您本地的同名端口。

ssh -N -L 18789:127.0.0.1:18789 xxx@xxx.xxx.xxx.xxx -p xxx

命令运行后，在本地浏览器访问 http://127.0.0.1:18789 即可打开 Web UI 。

您可以发送一条 Hello 消息进行测试，如果收到回复，则说明一切正常。

后台界面

第六步：接入飞书

首先，在服务器上为 Moltbot 安装飞书插件：

clawdbot plugins install @m1heng-clawd/feishu

接下来，登录飞书开放平台，进入「开发者后台」，点击「创建企业自建应用」。

填写应用名称和描述后，完成创建。

创建成功后，进入应用的「凭据与基础信息」页面，复制并妥善保存 App ID 和 App Secret，它们将在后续配置中用到。

然后添加机器人，如下操作

暂时停留在飞书后台，我们先返回服务器终端，完成 Moltbot 的飞书相关配置。

飞书的其他配置先暂停，回到服务器配置 Clawdbot 的飞书参数

添加飞书配置

clawdbot config set channels.feishu.appId "飞书 app id"

clawdbot config set channels.feishu.appSecret "飞书 app secret"

clawdbot config set channels.feishu.enabled true

# 推荐使用 websocket
clawdbot config set channels.feishu.connectionMode websocket

clawdbot config set channels.feishu.dmPolicy pairing

clawdbot config set channels.feishu.groupPolicy allowlist

clawdbot config set channels.feishu.requireMention true

配置完成之后，重启

clawdbot gateway restart

重启完成后回到飞书，找到「事件和回调」，选择长连接模式，如下图

如果配置成功，说明连接已建立。继续下面的配置，添加事件，选择「接收消息」事件

事件添加完成之后，还需要开通权限，有以下权限全部勾选

权限	Scope（范围）	Description（说明）
contact:user.base:readonly	用户信息	获取基础用户信息
im:message	消息全部勾选	发送和接收消息

如下图

以上步骤全部完成后，即可与机器人对话。但在此之前需要先创建一个版本

注意：每次修改配置后都需要重新发布版本，建议全部配置完成后再统一发布。

发布完成后，回到飞书客户端，可以看到应用已上线，点击打开应用

向机器人发送 Hello，即可收到 Moltbot 的回复

恭喜您！至此，您已成功在 Linux 服务器上部署了 Moltbot，并将其与飞书集成。现在您可以开始探索其强大的功能，或根据官方文档进行更深度的定制。如果在部署过程中遇到任何问题，欢迎留言交流。

常用命令

# 检查常见安全问题
clawdbot security audit

# 自动修复文件权限等问题
clawdbot security audit --fix

# 查看 Moltbot 日志
clawdbot logs --follow

参考文献

Windows 配置 Claude Code 解决 settings.json 不生效

发表评论

TL;DR

默认情况下 Windows 安装的 Claude Code 会从这个位置读取配置：

C:\Users\
<YOUR_NAME>\.claude

其他系统则类似的找到 ~/.claude 路径。

官方流程在安装结束后就完成了，可以直接登录使用。如果需要修改配置将 Claude Code 接入第三方 API，就需要修改这里的 settings.json 配置文件，可以使用 CC-Switch 或是手动编写，但是修改后会发现不生效。

经过调研，发现是最新版的 Claude Code 修改了一个参数，导致启动时一定要登录，这里可以手动修改一下 `C:\Users\

\.claude.json` 中的这个参数，即可正常使用： “` hasCompletedOnboarding: true “` 将配置中的 `false` 改为 `true` ，再重新运行 `Claude Code` 即可。 > 这个问题卡了我两天，一度打算放弃回到其他 OS，最后终于在 [这里](https://github.com/farion1231/cc-switch/issues/404#issuecomment-3677768452) 找到了答案，因此特别记录一下。 # Refs – [《 Claude Code 终极版 FAQ 指南》](https://linux.do/t/topic/803265) – [Windows 用 Claude Code 的最优解是 WSL 吗？](https://linux.do/t/topic/1422264) – [请问一下claude code windows版本claude.exe是否settings.json不生效？](https://linux.do/t/topic/1327596) – [配置好 setting.json 文件之后，Claude任然要求登录？怎么办？ #404](https://github.com/farion1231/cc-switch/issues/404#issuecomment-3677768452)

Windows 配置 Claude Code 全流程

发表评论

今天终于跑通了 Windows 下运行 Claoude Code 的全流程，不借助 WSL ，原生运行。起因是自己需要一个可以长期运行任务的云桌面，这方面还是 Windows 最好用。不得不说相比于 Linux/macOS ，Windows 下运行 Claude Code 实在太多坑了。

感谢 LD 巨佬哈雷彗星的《 Claude Code 终极版 FAQ 指南》，在这份指南的指导下,还有开源社区朋友的帮助下，终于跑通了全流程，在这里记录一下，避免更多的朋友踩坑。

ClaudeCode

环境说明

先说一下我的环境：

OS: Windows 10 Enterprise LTSC 2021
Shell: PowerShell 5 -> PowerShell 7

我是在全新安装的 Windows 10 Enterprise LTSC 2021 系统中进行，使用系统自带的 PowerShell 5 开始，流程中会安装最新的 PoserShell 7 并使用 PS7 完成后续。根据论坛大佬的说法，PS7 很好用，自带的 PS5 不好用，因此下面的流程包含了安装 PS7 的流程，大家可以自己体验一下。

主要流程

以下内容主要来自《 Claude Code 终极版 FAQ 指南》和实践补充。

打开 PowerShell 5 并安装 WinGet:

$progressPreference = 'silentlyContinue'
Install-PackageProvider -Name NuGet -Force | Out-Null
Install-Module -Name Microsoft.WinGet.Client -Force -Repository PSGallery | Out-Null
Write-Host "Using Repair-WinGetPackageManager cmdlet to bootstrap WinGet..."
Repair-WinGetPackageManager -AllUsers

在 PoserShell 5 中使用 WinGet 安装 PoserShell 7

winget install Microsoft.PowerShell

打开 PoserShell 7 并使用 winget 安装其他依赖，后续全部在 PS7 中进行：


# 必须，安装 fnm 用于管理 node 环境，或使用现有环境
winget install Schniz.fnm
# 可选，安装 Git
winget install --id Git.Git -e --source winget
# 可选，安装 notepad4
winget install zufuliu.notepad4
# 可选，安装 Windows Terminal
winget install -e --id Microsoft.WindowsTerminal
# 可选，安装 Notepad++
winget install -e --id Notepad++.Notepad++

下面使用 fnm 安装特定版本的 node 运行环境，首先需要对 FNM 给予一个环境启动，否则安装的 node 无法使用，因此首先预配置环境：

New-Item –Path $Profile –Type File –Force
notepad $profile

之后将以下内容写入该文件并保存：

fnm env --use-on-cd --shell powershell | Out-String | Invoke-Expression

下面开始正式安装 nodejs:

fnm install lts/krypton
fnm use lts/krypton

全局安装 Claude Code:

npm install -g @anthropic-ai/claude-code

之后就可以愉快的使用 Claude Code 了。

Claude Code 在 Windows PowerShell7 运行

常见问题

配置 `settings.json` 不生效

默认情况下这样安装的 Claude Code 会从这个位置读取配置：

C:\Users\
<YOUR_NAME>\.claude

官方流程在安装结束后就完成了，可以直接登录使用。如果你需要修改配置将 Claude Code 接入第三方 API，就需要修改这里的 settings.json 配置文件，可以使用 CC-Switch 或是手动编写，但是修改后会发现不生效。

经过调研，发现是最新版的 Claude Code 修改了一个参数，导致启动时一定要登录，这里可以手动修改一下 `C:\Users\

AI 生图精品提示词｜第二期：城市星球

发表评论

若没有特别说明，默认使用 AiLoft 提供的 Nano Banana Pro 模型生成。

城市星球系列封面图

本次带来《城市星球》系列，先看效果图：

Refs: https://x.com/TechieBySA/status/1999577563295826208

广州城市星球

北京城市星球

厦门城市星球

香港城市星球

芝加哥城市星球

澳门城市星球

提示词如下：

Create a hyperrealistic miniature planet showcasing [GuangZhou] with famous landmarks seamlessly curving around the spherical surface. Position bold 3D white text ”[CITY]” naturally integrated across the lush green central parkland with realistic shadows and dimensional depth. Capture from a top-down orbiting angle that emphasizes the dramatic planet curvature. Use soft golden hour daylight filtering through partly cloudy skies, casting gentle shadows on emerald grass and surrounding trees. The background should blend into a swirling atmospheric sky. Apply vibrant greens, warm earth tones, and soft blues. Render in polished photorealistic style with fine architectural detail.

可以讲示例中的 GuangZhou 换成其他城市，例如：

[GuangZhou]
[BeiJing]
[XiaMen]
[ZhengZhou]
[HongKong]
[Chicago]
[Macau]

效果都很不错，快用 AiLoft 试试吧。

AI 生图精品提示词｜第一期

发表评论

若没有特别说明，默认使用 AiLoft 提供的 Nano Banana Pro 模型生成。

本期封面图：

英文封面图

中文封面图

3D Q 版迷你概念商店

Refs: https://x.com/dotey/status/1995190286775881780

说明：

🍌 nano banana prompt
3D chibi-style miniature concept store of {Brand Name}

提示词：

3D chibi-style miniature concept store of {Brand Name}, creatively designed with an exterior inspired by the brand's most iconic product or packaging (such as a giant {brand's core product, e.g., chicken bucket/hamburger/donut/roast duck}). The store features two floors with large glass windows clearly showcasing the cozy and finely decorated interior: {brand's primary color}-themed decor, warm lighting, and busy staff dressed in outfits matching the brand. Adorable tiny figures stroll or sit along the street, surrounded by benches, street lamps, and potted plants, creating a charming urban scene. Rendered in a miniature cityscape style using Cinema 4D, with a blind-box toy aesthetic, rich in details and realism, and bathed in soft lighting that evokes a relaxing afternoon atmosphere. --ar 2:3

Brand name: Starbucks

效果图： Starbucks 3D chibi-style miniature concept store of Starbucks

麦当劳 3D chibi-style miniature concept store of 麦当劳

McDonald 3D chibi-style miniature concept store of McDonald

和明星自拍

refs: https://x.com/canghecode/status/1996593241421181403

提示词:

“我在[电影名称]的片场和[电影角色]自拍。

保持人物与参考图像完全一致，面部特征、骨骼结构、肤色、表情、姿势和外貌 100%相同。1:1 宽高比，4K 细节。”

实例:

参考图片：

哈利波特

我在[疯狂动物城]的片场和[Judy Hopps]、[Nick Wilde]自拍。

保持人物与参考图像完全一致，面部特征、骨骼结构、肤色、表情、姿势和外貌 100%相同。1:1 宽高比，4K 细节。

效果图：

哈利波特在疯狂动物城拍摄现场与主演合影

3D 微缩电影场景

refs: https://x.com/tsubame_99/status/1997181713546727621

Present an exquisite, miniature 3D cartoon-style scene of the user-specified subject, clearly viewed from a 45° top-down perspective.

Place the subject’s most iconic representation, character, or landmark prominently at the center, complemented by proportionally-sized icons of key elements, symbolic items, charming figures, props, and other details illustrating the subject’s core theme or narrative. The scene should be detailed, finely crafted, and playful.

Rendered with Cinema 4D, the modeling should be refined, smoothly rounded, and rich in texture, accurately capturing realistic PBR materials. Gentle, lifelike lighting and soft shadows should create a warm, comfortable ambiance.

Maintain a clean, minimalist layout and a solid-color background to highlight the primary content.

At the top-center of the scene, prominently display the subject name in a large font size.

All texts should be displayed in the language specified or entered by the user, without any background, and may subtly overlap with the scene elements to enhance overall design integration.

**Parameters:**
Aspect ratio: {User input, default 1:1}
Subject Name / Topic: {User input}

实例

Subject Name / Topic: Squid Game
Subject Name / Topic: 名侦探柯南

效果图：

名侦探柯南 3D 微缩电影场景

3D 动态城市天气场景

refs: https://x.com/dotey/status/1993729800922341810?s=46

说明：

🍌 nano banana pro prompt (with gemini)

Dynamically generate a current weather card based on a given city name.

提示词：

Present a clear, 45° top-down view of a vertical (9:16) isometric miniature 3D cartoon scene, highlighting iconic landmarks centered in the composition to showcase precise and delicate modeling.

The scene features soft, refined textures with realistic PBR materials and gentle, lifelike lighting and shadow effects. Weather elements are creatively integrated into the urban architecture, establishing a dynamic interaction between the city's landscape and atmospheric conditions, creating an immersive weather ambiance.

Use a clean, unified composition with minimalistic aesthetics and a soft, solid-colored background that highlights the main content. The overall visual style is fresh and soothing.

Display a prominent weather icon at the top-center, with the date (x-small text) and temperature range (medium text) beneath it. The city name (large text) is positioned directly above the weather icon. The weather information has no background and can subtly overlap with the buildings.

The text should match the input city's native language.
Please retrieve current weather conditions for the specified city before rendering.

City name:【广州】

效果图

广州动态城市天气场景

3D 诗词场景

Refs: https://x.com/ykszs017/status/1997268159800434783

A vertical (9:16), intricately detailed 45° top-down isometric miniature 3D cartoon scene. The central composition is a richly composed and densely modeled visualization of an ancient Chinese poem's imagery. The scene features soft but highly tactile PBR textures with matte materials (like hand-sculpted clay with visible tool marks, aged wood grain, rough ceramic, and textured fabrics). Layered, lyrical lighting with soft shadows and ambient occlusion emphasizes the depth of every small feature. Key visual elements derived from the verses are creatively integrated into the landscape. The scene must be densely filled with micro-details to create a lived-in, immersive poetic ambiance: add scattered pebbles, moss in cracks, fallen leaves, dew drops on plants, intricate architectural carvings, tiny wildlife (like birds or fish), and environmental clutter relevant to the poem. Use a clean, unified composition that balances richness with aesthetics, set against a soft, solid-colored background that complements the scene's color palette. The overall visual style is fresh, soothing, yet elaborate and full of life. Text layout: At the top-center, display the poem title in large, elegant Chinese font. Directly beneath it, display the Dynasty and Author in medium font. The core verses of the poem are arranged below the main scene or subtly floating near relevant objects within the composition, with no background plate. Poem Title: 【天净沙·秋思】 Dynasty/Author: 【元·马致远】 Core Verses Context: 枯藤老树昏鸦，小桥流水人家，古道西风瘦马。夕阳西下，断肠人在天涯 。(Visual Focus: An old tree, tightly entwined with withered vines, has cracked branches, and a few crows are perched or flapping their wings. A small stone bridge spanning a cold and clear stream, with mist and moisture clinging to the water surface; Two or three compact houses by the stream, with very faint smoke from their chimneys, were torn apart by the cold wind. An ancient path, dusty by the autumn wind, extends deeper into the picture. A bony and thin horse, with dull fur, had its mane flapped sideways by the wind. A solitary traveler, wearing an old cloak, with a slender figure, strolled slowly in the wind. The withered and yellow grass lay on the ground, and the cracked dirt road had ruts. The setting sun was close to the horizon, its golden-red rays shattered by the wind, casting long shadows on the travelers, horses and ancient paths. The overall scene presents a desolate, dry and cold, and spacious atmosphere of late autumn.）

效果图：

天净沙·秋思

彻底解决阿里云和 tailscale 冲突

发表评论

如果你在一台阿里云服务器安装并启动了 tailscale，大概率会出现阿里云服务器无法上网的问题，根本原因为阿里云服务器默认DNS与tailscale网段产生冲突。

由于阿里云和 tailscale 都使用了 100.64.0.0/10 这个网段。100.64.0.0/10 在 RFC 6598 中被保留为 运营商级 NAT (Carrier-Grade NAT) 地址段，用于 ISP 做 NAT 时避免与内网冲突。Tailscale 把它当成“只允许来自 Tailscale 接口的地址段”是符合规范的。但阿里云在 VPC 内把 100.100.2.136、100.100.2.138 作为内网 DNS 服务地址，初衷是：地址在公网不可路由，避免外泄；与经典网络互通时不会冲突。

由于动机不同，目标不同，造成二者冲突。

目前比较流行的方法，是关闭 tailscale 的 iptables 规则生成，但这是不安全的。浏览各种解决方案后我认为脚本轮询的方案最可靠。

脚本轮训解决

/usr/local/bin/fix-ts-dns.sh

#!/bin/bash
while true; do
  if ! iptables -C ts-input -s 100.100.2.136/32 -j ACCEPT 2>/dev/null; then
    iptables -I ts-input 1 -s 100.100.2.136/32 -j ACCEPT
  fi
  sleep 30
done

/etc/systemd/system/fix-ts-dns.service

[Unit]
Description=Keep Aliyun DNS whitelist in Tailscale chain
After=tailscaled.service

[Service]
Type=simple
ExecStart=/usr/local/bin/fix-ts-dns.sh
Restart=always

[Install]
WantedBy=multi-user.target

chmod +x /usr/local/bin/fix-ts-dns.sh
systemctl enable --now fix-ts-dns.service

脚本轮询，检查到存在 tailscale 的规则表且不存在白名单时，自动插入一条放通阿里云网段的规则。

解决方案来源：https://www.xugj520.cn/archives/aliyun-cgnat-tailscale-conflict.html

Refs

阿里云 CGNAT 地址与 Tailscale 的 DNS 冲突：一次真实的排查记录与可行方案

阿里云使用 CGNAT 地址当作内部 DNS 服务地址, 导致和 Tailscale 冲突

Tailscale 与阿里云八字不合的解决方法（上）：治标

Tailscale 与阿里云八字不合的解决方法（下）：寻根

Tailscale 和阿里云DNS 水土不服的解法

阿里云安装Docker和Tailscale及遇到的问题

IP pool

macOS 单独为鼠标或触控板开启自然滚动

发表评论

macOS 系统配置“自然滚动”，虽然鼠标和触控板都可以配置，但是两个控制是同步的，更改任意一个另一个都会跟着动。

一直是这样，也没有要更改的意思。

使用 LinearMouse 这个软件可以实现单独为触控板开启自然滚动，鼠标不开启自然滚动。

对于我来说，操作自然多了。简单记录一下。

hccn 问题

huggingface-download 失败

310p 运行报错

310p vllm-ascend 报错

aggregated 模式卡个数错误

bf16 数据类型报错

npu_dynamic_quant 算子报错

OOM

Helm Update 报错

istio httproute 错误

总结

Refs

故障 pod describe

故障 pod /dev 检查

故障 pod 驱动检查

故障 pod 日志

故障 pod 驱动检查

检查服务状态？

检查卡获取

跟踪 strace

新增挂载继续跟踪

跟踪并查看日志

检查代码2

虚拟机场景

替换 apt 镜像源

安装 systemd

设置 systemd 为 init

前情提要

恢复方法

Refs

引言

准备工作

第一步：准备服务器环境

第二步：安装 Moltbot

第三步：配置使用 Qwen

第四步：配置 Moltbot 并启动 Gateway

第五步：访问后台

第六步：接入飞书

添加飞书配置

常用命令

参考文献

TL;DR

环境说明

主要流程

常见问题

配置 settings.json 不生效

3D Q 版迷你概念商店

和明星自拍

3D 微缩电影场景

3D 动态城市天气场景

3D 诗词场景

脚本轮训解决

Refs

Refs

配置 `settings.json` 不生效