Skip to content

optimize get_rdma_nics.sh to support nsys profile #8099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wangyuwen1999 wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

optimize get_rdma_nics.sh to support nsys profile #8099

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ function __JUDGE_NIC_TYPE__() {
gpu_first=true
xpu_first=true
cpu_first=true
# Cache ibdev2netdev output once to avoid repeated subprocess forks in the loop.

@PaddlePaddle-bot PaddlePaddle-bot Jul 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 这次只缓存了 package 资源脚本，但仓库里还有同名脚本副本仍保留旧的多次 ibdev2netdev 调用。

fastdeploy/cache_manager/transfer_factory/utils.py 会调用本文件；但 scripts/get_rdma_nics.sh 和 tests/e2e/utils/get_rdma_nics.sh 仍在 gpu / xpu / cpu_ib 分支里直接执行 ibdev2netdev | awk，e2e 用例也会调用后者。在 nsys profile 场景下这些入口仍可能卡在多次 fork 上，和本 PR 的优化目标不一致。

建议修复方式：将同样的 _IBDEV2NETDEV_CACHE 缓存逻辑同步到这两个副本，或改成单一脚本来源，避免三份实现继续漂移。

# Only needed for rdma-based types; cpu uses ip only.
_IBDEV2NETDEV_CACHE=""
if [[ "$type" != "cpu" ]]; then
_IBDEV2NETDEV_CACHE=$(ibdev2netdev 2>/dev/null)
fi

for (( xgbe_no=0; xgbe_no < XGBE_NUM; xgbe_no++ ))
do
Expand All @@ -41,7 +47,7 @@ function __JUDGE_NIC_TYPE__() {
grep -qxF "$NIC_ROOTPORT" ${gpu_root_port_filename} 2>/dev/null && NIC_TYPE="GPU_NIC"

if [[ "$type" == "gpu" && "$NIC_TYPE" == "GPU_NIC" ]]; then
ibdev=$(ibdev2netdev 2>/dev/null | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
ibdev=$(echo "$_IBDEV2NETDEV_CACHE" | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
if [ -n "$ibdev" ] && ip link show "${NICNAME_TYPE}${xgbe_no}" | grep -q "state UP"; then
if $gpu_first; then
printf "KVCACHE_RDMA_NICS=%s" "$ibdev"
Expand All @@ -53,7 +59,7 @@ function __JUDGE_NIC_TYPE__() {
fi

if [[ "$type" == "xpu" && "$NIC_TYPE" == "GPU_NIC" ]]; then
ibdev=$(ibdev2netdev 2>/dev/null | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
ibdev=$(echo "$_IBDEV2NETDEV_CACHE" | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
if [ -n "$ibdev" ] && ip link show "${NICNAME_TYPE}${xgbe_no}" | grep -q "state UP"; then
if $xpu_first; then
printf "KVCACHE_RDMA_NICS=%s,%s" "$ibdev" "$ibdev"
Expand Down Expand Up @@ -86,7 +92,7 @@ function __JUDGE_NIC_TYPE__() {
fi

if [[ "$type" == "cpu_ib" && "$NIC_TYPE" == "CPU_NIC" ]]; then
ibdev=$(ibdev2netdev 2>/dev/null | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
ibdev=$(echo "$_IBDEV2NETDEV_CACHE" | awk -v nic="${NICNAME_TYPE}${xgbe_no}" '$5 == nic {print $1}')
if [ -n "$ibdev" ] && ip link show "${NICNAME_TYPE}${xgbe_no}" | grep -q "state UP" && \
ip a show "${NICNAME_TYPE}${xgbe_no}" | grep -q "inet "; then
if $cpu_ib_first; then
Expand Down Expand Up @@ -183,11 +189,13 @@ function __main__() {

if [[ "$type" == "cpu_ib" ]]; then
first=true
# Cache ibdev2netdev output once outside the loop
_IBDEV2NETDEV_CACHE=$(ibdev2netdev 2>/dev/null)
for bond in $(ls -d /sys/class/net/bond* 2>/dev/null); do
bond_if=$(basename "$bond")
__NEW_GPU_ROOTPORT_FILE__

ibdev=$(ibdev2netdev 2>/dev/null | grep -w "$bond_if" | awk '{print $1}')
ibdev=$(echo "$_IBDEV2NETDEV_CACHE" | grep -w "$bond_if" | awk '{print $1}')
if [ -n "$ibdev" ] && ip link show "$bond_if" | grep -q "state UP" && \
ip a show "$bond_if" | grep -q "inet "; then
if $first; then
Expand Down
Loading