Test KIOXIA BG5 M.2 NVMe SSD

[watson81]

I ordered 3 KIOXIA BG4 SSDs from Amazon, specifically model BG40ZNS256G. This is the 256GB version of the KBG40ZNS128G you have listed here. However, I received BG5 model SSDs instead, specifically model KBG50ZNS256G. Thus, I gathered information and ran your benchmark scripts to contribute back. I've attempted to mimic #326 to make this easier for you.

These tests were all run on an 8GB Raspberry Pi 5 with a Waveshare PCIe to M.2 PoE HAT configured to use PCIe Gen 3.

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ sudo lspci -s "0000:01:00.0" -vvvv
0000:01:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller BG5 (DRAM-less) (prog-if 02 [NVM Express])
	Subsystem: KIOXIA Corporation NVMe SSD Controller BG5 (DRAM-less)
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at 1b80000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable- Count=1/32 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable+ Count=65 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x1 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [150 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [260 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: LaneErr at lane: 0
	Capabilities: [360 v1] Data Link Feature <?>
	Capabilities: [380 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [400 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
			  PortCommonModeRestoreTime=60us PortTPowerOnTime=150us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=216064ns
		L1SubCtl2: T_PwrOn=150us
	Capabilities: [470 v1] Lane Margining at the Receiver <?>
	Kernel driver in use: nvme

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ sudo nvme id-ctrl /dev/nvme0n1
NVME Identify Controller:
vid       : 0x1e0f
ssvid     : 0x1e0f
sn        : 323PD743Q1Y5        
mn        : KBG50ZNS256G NVMe KIOXIA 256GB          
fr        : 11200104
rab       : 2
ieee      : 8ce38e
cmic      : 0
mdts      : 9
cntlid    : 0
ver       : 0x10400
rtd3r     : 0x186a0
rtd3e     : 0x7a120
oaes      : 0x200
ctratt    : 0x2
rrls      : 0
cntrltype : 1
fguid     : 00000000-0000-0000-0000-000000000000
crdt1     : 0
crdt2     : 0
crdt3     : 0
nvmsr     : 1
vwci      : 0
mec       : 1
oacs      : 0x17
acl       : 3
aerl      : 7
frmw      : 0x14
lpa       : 0x1e
elpe      : 255
npss      : 4
avscc     : 0
apsta     : 0x1
wctemp    : 356
cctemp    : 358
mtfa      : 50
hmpre     : 16384
hmmin     : 3584
tnvmcap   : 256,060,514,304
unvmcap   : 0
rpmbs     : 0
edstt     : 9
dsto      : 1
fwug      : 1
kas       : 0
hctma     : 0x1
mntmt     : 273
mxtmt     : 358
sanicap   : 0xa0000002
hmminds   : 1
hmmaxd    : 1024
nsetidmax : 0
endgidmax : 0
anatt     : 0
anacap    : 0
anagrpmax : 0
nanagrpid : 0
pels      : 32
domainid  : 0
megcap    : 0
sqes      : 0x66
cqes      : 0x44
maxcmd    : 0
nn        : 1
oncs      : 0xdf
fuses     : 0x1
fna       : 0
vwc       : 0x7
awun      : 31
awupf     : 0
icsvscc   : 0
nwpc      : 0
acwu      : 31
ocfs      : 0
sgls      : 0
mnan      : 0
maxdna    : 0
maxcna    : 0
subnqn    : nqn.2019-10.com.kioxia:KBG50ZNS256G NVMe KIOXIA 256GB:323PD743Q1Y5
ioccsz    : 0
iorcsz    : 0
icdoff    : 0
fcatt     : 0
msdbd     : 0
ofcs      : 0
ps      0 : mp:4.00W operational enlat:1 exlat:1 rrt:0 rrl:0
            rwt:0 rwl:0 idle_power:- active_power:-
            active_power_workload:-
ps      1 : mp:2.50W operational enlat:1 exlat:1 rrt:1 rrl:1
            rwt:1 rwl:1 idle_power:- active_power:-
            active_power_workload:-
ps      2 : mp:1.30W operational enlat:1 exlat:1 rrt:2 rrl:2
            rwt:2 rwl:2 idle_power:- active_power:-
            active_power_workload:-
ps      3 : mp:0.0500W non-operational enlat:800 exlat:1000 rrt:3 rrl:3
            rwt:3 rwl:3 idle_power:- active_power:-
            active_power_workload:-
ps      4 : mp:0.0030W non-operational enlat:3000 exlat:44900 rrt:4 rrl:4
            rwt:4 rwl:4 idle_power:- active_power:-
            active_power_workload:-

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ sudo nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning			: 0
temperature				: 38°C (311 Kelvin)
available_spare				: 100%
available_spare_threshold		: 50%
percentage_used				: 0%
endurance group critical warning summary: 0
Data Units Read				: 953,241 (488.06 GB)
Data Units Written			: 803,353 (411.32 GB)
host_read_commands			: 7,677,324
host_write_commands			: 4,167,531
controller_busy_time			: 22
power_cycles				: 43
power_on_hours				: 54
unsafe_shutdowns			: 15
media_errors				: 0
num_err_log_entries			: 0
Warning Temperature Time		: 0
Critical Composite Temperature Time	: 0
Temperature Sensor 1           : 38°C (311 Kelvin)
Thermal Management T1 Trans Count	: 0
Thermal Management T2 Trans Count	: 0
Thermal Management T1 Total Time	: 0
Thermal Management T2 Total Time	: 0

Despite being sold to me as "new", it shows a non-trivial amount of wear. Boo

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ sudo modinfo nvme
name:           nvme
filename:       (builtin)
version:        1.0
license:        GPL
file:           drivers/nvme/host/nvme
author:         Matthew Wilcox <[email protected]>
parm:           use_threaded_interrupts:int
parm:           use_cmb_sqes:use controller's memory buffer for I/O SQes (bool)
parm:           max_host_mem_size_mb:Maximum Host Memory Buffer (HMB) size per controller (in MiB) (uint)
parm:           sgl_threshold:Use SGLs when average request segment size is larger or equal to this size. Use 0 to disable SGLs. (uint)
parm:           io_queue_depth:set io queue depth, should >= 2 and < 4096
parm:           write_queues:Number of queues to use for writes. If not set, reads and writes will share a queue set.
parm:           poll_queues:Number of queues to use for polled IO.
parm:           noacpi:disable acpi bios quirks (bool)

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ sudo DEVICE_UNDER_TEXT=/dev/nvme0n1 DEVICE_MOUNT_PATH=/mnt/nvme0n1 ./disk-benchmark.sh

Full results attached: full-benchmark.txt
Summary:

Benchmark	Result
fio 1M sequential read	890 MB/s
iozone 1M random read	790 MB/s
iozone 1M random write	760 MB/s
iozone 4K random read	78 MB/s
iozone 4K random write	244 MB/s

[watson81]

patrick@raspberrypi5🌐  in ~/stress $ dmesg | grep -C 2 -i nvme
[    1.877235] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[    1.883042] pcieport 0000:00:00.0: AER: enabled with IRQ 38
[    1.888732] nvme nvme0: pci function 0000:01:00.0
[    1.893465] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    1.901000] hwmon hwmon1: temp1_input not attached to any thermal zone
[    1.907556] hwmon hwmon1: temp2_input not attached to any thermal zone
[    1.915219] nvme nvme0: allocated 64 MiB host memory buffer.
[    1.922398] nvme nvme0: 4/0/0 default/read/poll queues
[    1.933386]  nvme0n1: p1 p2 p3 p4
[    1.936902] brcm-pcie 1000120000.pcie: host bridge /axi/pcie@120000 ranges:
[    1.943895] brcm-pcie 1000120000.pcie:   No bus range found for /axi/pcie@120000, using [bus 00-ff]