1. lspci
it reads the information of the pci bus and device.
>sudo lspci -vvv
with parameter -vvv , it will show the device configure space and capabilities. For example:
Here is a bridge. First it has type1 configure space header as:
64:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
Physical Slot: 6
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 30
NUMA node: 0
Bus: primary=64, secondary=65, subordinate=65, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: b6000000-b61fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Intel Corporation Sky Lake-E PCI Express Root Port A
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee01000 Data: 4021
Masking: 00000002 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #6, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
Kernel driver in use: pcieport
Here is a device, it has the pci configure space type0 header as:
65:00.0 Non-Volatile memory controller: XXX Technology Group Ltd. Device xxx (rev 02) (prog-if 02 [NVM Express])
Subsystem: XXX Technology Group Ltd. Device 0100
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 31
NUMA node: 0
Region 0: Memory at b6000000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s unlimited, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=2 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [178 v1] #19
Capabilities: [198 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 109a
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000b6004000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: nvme
as we know, pci capabilities is chain structure link by pointers as:
And pcie has 3 types of capabilities: 1) pci capabilities 2) pci express capabilities 3) pci express extended capabilites.
Why 3? maybe need more functions. Here is the layout of all three types of capabilities in the configure space:
1) pci capabilities . it has ID table:
Can we find some pci capabilities in lspci information?
Yes, For bridge:
Capabilities: [40] Subsystem: Intel Corporation Sky Lake-E PCI Express Root Port A
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee01000 Data: 4021
Masking: 00000002 Pending: 00000000
Capabilities: [e0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-2) pci express capabilities . it is in pci capabilities id table 0x10.
Here is its structure:
For bridge, it is:
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #5, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #6, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootCap: CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-3) pci express extended capabilities
it's struct is:
each extended capability has version number, so we can know them for bridge:
Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
Capabilities: [110 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [148 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
Capabilities: [250 v1] #19
Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>So how many kind of pci express extended capabilities? Good question! Count them in the spec, looks like 26. You can count them.
And now you should be able to find all 3 types of capabilities of the device.
2. setpci
It is a tool to help us to access the pci/pcie configure space regitsters.
Here list all the registers and registers base.
xxx@YYYY:~$ setpci --dumpregs
cap pos w name
00 W VENDOR_ID -- type 0 header --
02 W DEVICE_ID
04 W COMMAND
06 W STATUS
08 B REVISION
09 B CLASS_PROG
0a W CLASS_DEVICE
0c B CACHE_LINE_SIZE
0d B LATENCY_TIMER
0e B HEADER_TYPE
0f B BIST
10 L BASE_ADDRESS_0
14 L BASE_ADDRESS_1
18 L BASE_ADDRESS_2
1c L BASE_ADDRESS_3
20 L BASE_ADDRESS_4
24 L BASE_ADDRESS_5
28 L CARDBUS_CIS
2c W SUBSYSTEM_VENDOR_ID
2e W SUBSYSTEM_ID
30 L ROM_ADDRESS
3c B INTERRUPT_LINE
3d B INTERRUPT_PIN
3e B MIN_GNT
3f B MAX_LAT
18 B PRIMARY_BUS -- type 1 header --
19 B SECONDARY_BUS
1a B SUBORDINATE_BUS
1b B SEC_LATENCY_TIMER
1c B IO_BASE
1d B IO_LIMIT
1e W SEC_STATUS
20 W MEMORY_BASE
22 W MEMORY_LIMIT
24 W PREF_MEMORY_BASE
26 W PREF_MEMORY_LIMIT
28 L PREF_BASE_UPPER32
2c L PREF_LIMIT_UPPER32
30 W IO_BASE_UPPER16
32 W IO_LIMIT_UPPER16
38 L BRIDGE_ROM_ADDRESS
3e W BRIDGE_CONTROL
10 L CB_CARDBUS_BASE --card bus --
14 W CB_CAPABILITIES
16 W CB_SEC_STATUS
18 B CB_BUS_NUMBER
19 B CB_CARDBUS_NUMBER
1a B CB_SUBORDINATE_BUS
1b B CB_CARDBUS_LATENCY
1c L CB_MEMORY_BASE_0
20 L CB_MEMORY_LIMIT_0
24 L CB_MEMORY_BASE_1
28 L CB_MEMORY_LIMIT_1
2c W CB_IO_BASE_0
2e W CB_IO_BASE_0_HI
30 W CB_IO_LIMIT_0
32 W CB_IO_LIMIT_0_HI
34 W CB_IO_BASE_1
36 W CB_IO_BASE_1_HI
38 W CB_IO_LIMIT_1
3a W CB_IO_LIMIT_1_HI
40 W CB_SUBSYSTEM_VENDOR_ID
42 W CB_SUBSYSTEM_ID
44 L CB_LEGACY_MODE_BASE
01 00 - CAP_PM -- pci express caps --
02 00 - CAP_AGP
03 00 - CAP_VPD
04 00 - CAP_SLOTID
05 00 - CAP_MSI
06 00 - CAP_CHSWP
07 00 - CAP_PCIX
08 00 - CAP_HT
09 00 - CAP_VNDR
0a 00 - CAP_DBG
0b 00 - CAP_CCRC
0c 00 - CAP_HOTPLUG
0d 00 - CAP_SSVID
0e 00 - CAP_AGP3
0f 00 - CAP_SECURE
10 00 - CAP_EXP
11 00 - CAP_MSIX
12 00 - CAP_SATA
13 00 - CAP_AF
0001 00 - ECAP_AER -- pci express extended caps , not 26 types ??--
0002 00 - ECAP_VC
0003 00 - ECAP_DSN
0004 00 - ECAP_PB
0005 00 - ECAP_RCLINK
0006 00 - ECAP_RCILINK
0007 00 - ECAP_RCECOLL
0008 00 - ECAP_MFVC
000a 00 - ECAP_RBCB
000b 00 - ECAP_VNDR
000d 00 - ECAP_ACS
000e 00 - ECAP_ARI
000f 00 - ECAP_ATS
0010 00 - ECAP_SRIOV
001d 00 - ECAP_DPCYou can check all the registers refer to the configure header and capabilities struct.
To read a register:
xxx@YYYY:~$ sudo setpci -s 0000:64:00.0 CAP_EXP+0x02.w
01420000:64:00.0 is the pcie address (domain:bus:device.function) of bridge. 0142 is the register value.
To write a register:
xxx@YYYY:~$ sudo setpci -s 0000:64:00.0 CAP_EXP+0x18.w=0x03c8
xxx@YYYY:~$ sudo setpci -s 0000:64:00.0 CAP_EXP+0x18.w
03c8This enables slot control register bit 3 -- presence detect changed enabled.
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-PresDet+ is bit 3.
The end.