TL;DR
AMD Ryzen Threadripper PRO 3995WX在8通道DDR4-3200下
Linux内建的/dev/shm可以达到9M~10M 4k IOPS
而16x Samsung 980 PRO 1TB安装于此台主机上
未最佳化的RAID 0与系统在高队列多执行绪下可达到6M 4k IOPS
主要硬件
AMD Ryzen Threadripper PRO 3995WX
8x 64GB Samsung DDR4-3200 ECC RDIMM
ASUS Pro WS WRX80E-SAGE SE WIFI (custom bios version)
7x ASUS HYPER M.2 X16 GEN 4 CARD
16x Samsung 980 PRO 1TB FW:5B2QGXA7 (Slot 1~7各2个+主机板2个)
mdadm RAID0 devices=16 chunk-size=128kB
default xfs
default mount
2x Crucial MX500
mdadm RAID1 装系统
韧体设定
NPS4
TSME disable
作业系统
Ubuntu 22.04 LTS 5.15.0
其他没说明的都是默认
说明
原本是准备了30个Samsung 980 PRO 1TB尝试最大化效能
但后来发现有理论上与工程上的问题
根据AMD的文件 理论上的问题是
Each Processor has the capability to support four P links and four G links.
P links are connected to the PCIe root ports.
G links have the option to be connected to a PCIe root port or as cross
socket links that can be used to communicate cross-socket traffic.
但这张主机板是单插槽 所以G links应该是用于PCIe
There are four bandwidth injection points into the Infinity Fabric of a
single IOD: one per quadrant.
If the board designer chooses to use both G link and P link, then Px and Gx
share the bandwidth injection point in the IOD quadrant x.
When using both Px and Gx as bandwidth injection points, the system achieves
around 20 lanes of effective bandwidth from that quadrant, even though
32 lanes are being used.
根据这段 就算SSD全部直连CPU 最多也只能用到20个
(20 lanes * 4 roots / 4 lanes per ssd)的频宽
这个事实蛮合理的 因为DDR4-3200 8通道理论频宽204.8GB/s
而128x PCIe 4.0 理论频宽为256GB/s
中间还有infinity fabric, IOD, CCD等东西卡著
怎么算都会网络塞车
When a G link is used to connect to a second PCIe root port, we will have a
configuration in which traffic from two PCIe root ports must be distributed
to a single DMA BW injection point.
In the above configuration, the upstream bandwidth is equally distributed
between P and G links, assuming an equal number of lanes being used in
both P and G links.
根据这段，我应该要安装各2个在P0, G0, P1, G1, P2, G2, P3, G3
但华硕的工作站主机板说明书都不会提供主机板架构
(这点Supermicro gigabyte asrock rack 都做得比较好
工作站或服务器级主机板手册都会提供架构图
Supermicro的M12SWA-TF就有提供这张图
https://i.imgur.com/F4UQPzH.jpg )
反正最后用试的试出来
使用16个SSD 每个插槽各安装2个 主机板上装2个
这样能让每个PCIe Root Complex底下有4个SSD
工程上的问题是
这张板子的BIOS Version 1003进不了Linux
网络上其他人的留言
https://unix.stackexchange.com/questions/692552/#700549
而就算退版回0701能进Linux 只要安装超过14个SSD就会有问题
(乡民: 购买工作站或服务器 请找品牌厂+SI
有钱就上DELL HPE 没钱至少也选Supermicro或其他台厂白牌"整机"
[丽台 华硕 技嘉 微星 广达 纬颖 ...]
自组遇到原厂没有测过的issue 没有SI支援很可怕的
我就遇到了 华硕的工程部门 没有超过10颗的SSD
无法测试同时安装超过10颗以上的状况)
理论性能测试
使用fio 3.28 参数比照CrystalDiskMark 8 nvme
fio