Introduction to SMMU under Linux system
In the computer system architecture, similar to the traditional MMU used for the management of CPU access to memory, the IOMMU (Input Output Memory Management Unit) converts the request before passing the DMA request from the system I/O device to the system interconnect. address, and manage and restrict memory access transactions of system I/O devices. IOMMU maps device-visible virtual addresses (IOVA) to physical memory addresses. Different hardware architectures have different IOMMU implementations. The IOMMU of the ARM platform is SMMU (System Memory Management).
The SMMU provides translation services only for memory access transactions from system I/O devices, not for transactions to system I/O devices. Transactions from the system or CPU to system I/O devices are managed by other means, such as the MMU. The following figure shows the role of SMMU in the system.
Memory access transactions from system I/O devices refer to system I/O devices reading and writing memory. Transactions to system I/O devices usually refer to CPU accesses to system I/O devices internally mapped to physical memory addresses. Space memory or register. For a more detailed introduction to SMMU, please refer to IOMMU and Arm SMMU Introduction and SMMU Software Guide< a i=4>. For a detailed description of the SMMU's registers, data structures and behavior, please refer to ARM System Memory Management Unit Architecture Specification Version 3. For the specific implementation of SMMU, you can refer to relevant implementation documents, such as MMU-600's Arm CoreLink MMU-600 System Memory Management Unit Technical Reference Manual and MMU -700Arm® CoreLink™ MMU-700 System Memory Management Unit Technical Reference Manual.
SMMU distinguishes different system I/O devices through StreamID, etc. When system I/O devices access memory through SMMU, they need to bring StreamID and other information to SMMU. From the perspective of system I/O devices, the more detailed structure of the system including SMMU is as follows:
The system I/O device accesses the memory through DMA. After the DMA request is sent, it first passes through a device called DAA (for other implementations, it may be other devices) before being sent to the SMMU and system interconnection. The DAA does the first address. After conversion, the memory access request information, including the configured StreamID, etc. are sent to the SMMU for further processing.
In a Linux system, to enable SMMU for a certain system I/O device, you generally need to go through the following steps:
- Initialization of SMMU driver. This mainly includes reading the SMMU device node in the dts file, detecting the hardware device characteristics of the SMMU, initializing global resources and data structures, such as command queues, event queues, interrupts, and flow tables, etc., and registering the SMMU devices into the Linux kernel. IOMMU subsystem.
- During the system I/O device detection, discovery, and driver binding initialization process, the device is bound to the IOMMU. For devices that use DMA to access memory, this is typically done by calling the
of_dma_configure()/of_dma_configure_id()function. The process of device detection, discovery, and driver binding initialization requires access to the IOMMU configuration-related fields in the device node definition in the device tree dts file. As in arch/arm64/boot/dts/renesas/r8a77961.dtsi file:
iommus = <&ipmmu_vc0 19>;
- System I/O device driver configuration regarding IOMMU. This part usually varies depending on the specific hardware system implementation. This mainly includes calling the
dma_coerce_mask_and_coherent()/dma_set_mask_and_coherent()function to set the DMA mask and consistent DMA mask to the same value, and configuring something like the DAA mentioned earlier device of. - The system I/O device driver allocates memory. The system I/O device driver allocates memory through interfaces such as
dma_alloc_coherent(). In addition to allocating memory, this process also creates an address translation table through the operation function of the SMMU device driver, and completes SMMU CD, etc. Data structure settings. In the Linux kernel, different subsystems actually call different methods to allocate DMA memory, but ultimately they all need to call thedma_alloc_coherent()function, so that the allocated memory will go through the SMMU when accessed through DMA. . - accesses allocated memory. The address of the memory allocated through the
dma_alloc_coherent()function can be provided to the DMA configuration related logic of the system I/O device. Subsequent system I/O devices access the memory through DMA and will complete the address conversion through SMMU. When accessing memory through DMA, it will undergo address translation by SMMU.
The address translation of SMMU is completed with the help of related data structures, which mainly include the flow table and its flow table entries STE, the context descriptor table and its table entries CD, and the address translation table and its table entries. STE stores context information for the stream, 64 bytes per STE. CDs store all settings related to stage 1 conversion, 64 bytes per CD. The address translation table is used to describe the mapping relationship between virtual addresses and physical memory addresses. The structure of the flow table can be divided into linear flow table and 2-level flow table Two kinds. The linear flow table structure is as follows:
The example structure of the 2-level stream table is as follows:
The structure of the context descriptor table can be divided intosingle CD, single-level CD table a> and Level 2 CD table three situations. The example structure of a single CD is as follows:
The example structure of a single-stage CD table is as follows:
The example structure of level 2 CD table is as follows:
When SMMU does address translation, it finds the flow table according to the flow table base address register of SMMU, and finds STE in the flow table throughStreamID . Then find the context descriptor according to the configuration of STE and SubstreamID/PASID Table and corresponding CD. Then find the address translation table based on the information in the CD, and complete the final address translation through the address translation table.
The source code related to the IOMMU subsystem of the Linux kernel is located atdrivers/iommu, and the ARM SMMU driver implementation is located at drivers/iommu/arm/arm-smmu-v3. In the SMMU driver implementation of the Linux kernel, the data structures used for address translation are created in the different steps mentioned above:
- Flow tables are created during the initialization process of the SMMU driver. If the structure of the flow table is a linear flow table, all STEs in the linear flow table are configured as bypass SMMU, that is, the corresponding flows do not undergo SMMU address translation; if the structure of the flow table is a level 2 flow table, then the flow table Invalid L1 flow table descriptor.
- During the process of system I/O device discovery, detection, and driver binding initialization, a context descriptor table is created when the device is bound to the IOMMU. If the structure of the flow table is a level 2 flow table, this process will first create the level 2 flow table, and the STEs in the level 2 flow table are configured as bypass SMMUs. When creating a context descriptor table, it also depends on whether a level 2 context descriptor table is required. After the context descriptor table is created, its address is written to STE.
- System I/O device drivers create address translation tables during the process of allocating memory. During this process, the SMMU driver's callback will be called to put the address of the address translation table into the CD.
Data structure of SMMU in Linux kernel
The IOMMU subsystem of the Linux kernel uses the struct iommu_device structure to represent an IOMMU hardware device instance, and uses the struct iommu_ops structure to describe the operations supported by the IOMMU hardware device instance. and capabilities, these two structures are defined (located in the include/linux/iommu.h file) as follows:
/**
* struct iommu_ops - iommu ops and capabilities
* @capable: check capability
* @domain_alloc: allocate iommu domain
* @domain_free: free iommu domain
* @attach_dev: attach device to an iommu domain
* @detach_dev: detach device from an iommu domain
* @map: map a physically contiguous memory region to an iommu domain
* @unmap: unmap a physically contiguous memory region from an iommu domain
* @flush_iotlb_all: Synchronously flush all hardware TLBs for this domain
* @iotlb_sync_map: Sync mappings created recently using @map to the hardware
* @iotlb_sync: Flush all queued ranges from the hardware TLBs and empty flush
* queue
* @iova_to_phys: translate iova to physical address
* @probe_device: Add device to iommu driver handling
* @release_device: Remove device from iommu driver handling
* @probe_finalize: Do final setup work after the device is added to an IOMMU
* group and attached to the groups domain
* @device_group: find iommu group for a particular device
* @domain_get_attr: Query domain attributes
* @domain_set_attr: Change domain attributes
* @support_dirty_log: Check whether domain supports dirty log tracking
* @switch_dirty_log: Perform actions to start|stop dirty log tracking
* @sync_dirty_log: Sync dirty log from IOMMU into a dirty bitmap
* @clear_dirty_log: Clear dirty log of IOMMU by a mask bitmap
* @get_resv_regions: Request list of reserved regions for a device
* @put_resv_regions: Free list of reserved regions for a device
* @apply_resv_region: Temporary helper call-back for iova reserved ranges
* @domain_window_enable: Configure and enable a particular window for a domain
* @domain_window_disable: Disable a particular window for a domain
* @of_xlate: add OF master IDs to iommu grouping
* @is_attach_deferred: Check if domain attach should be deferred from iommu
* driver init to device driver init (default no)
* @dev_has/enable/disable_feat: per device entries to check/enable/disable
* iommu specific features.
* @dev_feat_enabled: check enabled feature
* @aux_attach/detach_dev: aux-domain specific attach/detach entries.
* @aux_get_pasid: get the pasid given an aux-domain
* @sva_bind: Bind process address space to device
* @sva_unbind: Unbind process address space from device
* @sva_get_pasid: Get PASID associated to a SVA handle
* @page_response: handle page request response
* @cache_invalidate: invalidate translation caches
* @sva_bind_gpasid: bind guest pasid and mm
* @sva_unbind_gpasid: unbind guest pasid and mm
* @def_domain_type: device default domain type, return value:
* - IOMMU_DOMAIN_IDENTITY: must use an identity domain
* - IOMMU_DOMAIN_DMA: must use a dma domain
* - 0: use the default setting
* @attach_pasid_table: attach a pasid table
* @detach_pasid_table: detach the pasid table
* @pgsize_bitmap: bitmap of all possible supported page sizes
* @owner: Driver module providing these ops
*/
struct iommu_ops {
bool (*capable)(enum iommu_cap);
/* Domain allocation and freeing by the iommu driver */
struct iommu_domain *(*domain_alloc)(unsigned iommu_domain_type);
void (*domain_free)(struct iommu_domain *);
int (*attach_dev)(struct iommu_domain *domain, struct device *dev);
void (*detach_dev)(struct iommu_domain *domain, struct device *dev);
int (*map)(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp);
size_t (*unmap)(struct iommu_domain *domain, unsigned long iova,
size_t size, struct iommu_iotlb_gather *iotlb_gather);
void (*flush_iotlb_all)(struct iommu_domain *domain);
void (*iotlb_sync_map)(struct iommu_domain *domain, unsigned long iova,
size_t size);
void (*iotlb_sync)(struct iommu_domain *domain,
struct iommu_iotlb_gather *iotlb_gather);
phys_addr_t (*iova_to_phys)(struct iommu_domain *domain, dma_addr_t iova);
struct iommu_device *(*probe_device)(struct device *dev);
void (*release_device)(struct device *dev);
void (*probe_finalize)(struct device *dev);
struct iommu_group *(*device_group)(struct device *dev);
int (*domain_get_attr)(struct iommu_domain *domain,
enum iommu_attr attr, void *data);
int (*domain_set_attr)(struct iommu_domain *domain,
enum iommu_attr attr, void *data);
/*
* Track dirty log. Note: Don't concurrently call these interfaces with
* other ops that access underlying page table.
*/
bool (*support_dirty_log)(struct iommu_domain *domain);
int (*switch_dirty_log)(struct iommu_domain *domain, bool enable,
unsigned long iova, size_t size, int prot);
int (*sync_dirty_log)(struct iommu_domain *domain,
unsigned long iova, size_t size,
unsigned long *bitmap, unsigned long base_iova,
unsigned long bitmap_pgshift);
int (*clear_dirty_log)(struct iommu_domain *domain,
unsigned long iova, size_t size,
unsigned long *bitmap, unsigned long base_iova,
unsigned long bitmap_pgshift);
/* Request/Free a list of reserved regions for a device */
void (*get_resv_regions)(struct device *dev, struct list_head *list);
void (*put_resv_regions)(struct device *dev, struct list_head *list);
void (*apply_resv_region)(struct device *dev,
struct iommu_domain *domain,
struct iommu_resv_region *region);
/* Window handling functions */
int (*domain_window_enable)(struct iommu_domain *domain, u32 wnd_nr,
phys_addr_t paddr, u64 size, int prot);
void (*domain_window_disable)(struct iommu_domain *domain, u32 wnd_nr);
int (*of_xlate)(struct device *dev, struct of_phandle_args *args);
bool (*is_attach_deferred)(struct iommu_domain *domain, struct device *dev);
/* Per device IOMMU features */
bool (*dev_has_feat)(struct device *dev, enum iommu_dev_features f);
bool (*dev_feat_enabled)(struct device *dev, enum iommu_dev_features f);
int (*dev_enable_feat)(struct device *dev, enum iommu_dev_features f);
int (*dev_disable_feat)(struct device *dev, enum iommu_dev_features f);
/* Aux-domain specific attach/detach entries */
int (*aux_attach_dev)(struct iommu_domain *domain, struct device *dev);
void (*aux_detach_dev)(struct iommu_domain *domain, struct device *dev);
int (*aux_get_pasid)(struct iommu_domain *domain, struct device *dev);
struct iommu_sva *(*sva_bind)(struct device *dev, struct mm_struct *mm,
void *drvdata);
void (*sva_unbind)(struct iommu_sva *handle);
u32 (*sva_get_pasid)(struct iommu_sva *handle);
int (*page_response)(struct device *dev,
struct iommu_fault_event *evt,
struct iommu_page_response *msg);
int (*cache_invalidate)(struct iommu_domain *domain, struct device *dev,
struct iommu_cache_invalidate_info *inv_info);
int (*sva_bind_gpasid)(struct iommu_domain *domain,
struct device *dev, struct iommu_gpasid_bind_data *data);
int (*sva_unbind_gpasid)(struct device *dev, u32 pasid);
int (*attach_pasid_table)(struct iommu_domain *domain,
struct iommu_pasid_table_config *cfg);
void (*detach_pasid_table)(struct iommu_domain *domain);
int (*def_domain_type)(struct device *dev);
int (*dev_get_config)(struct device *dev, int type, void *data);
int (*dev_set_config)(struct device *dev, int type, void *data);
unsigned long pgsize_bitmap;
struct module *owner;
};
/**
* struct iommu_device - IOMMU core representation of one IOMMU hardware
* instance
* @list: Used by the iommu-core to keep a list of registered iommus
* @ops: iommu-ops for talking to this iommu
* @dev: struct device for sysfs handling
*/
struct iommu_device {
struct list_head list;
const struct iommu_ops *ops;
struct fwnode_handle *fwnode;
struct device *dev;
};
The SMMU driver creates instances of the struct iommu_device and struct iommu_ops structures and registers them with the IOMMU subsystem.
The IOMMU subsystem of the Linux kernel uses the struct dev_iommu structure to represent a system I/O device connected to the IOMMU, and struct iommu_fwspec to represent the system I/O The IOMMU device connected to the device, these structure definitions (located in the include/linux/iommu.h file) are as follows:
struct fwnode_handle {
struct fwnode_handle *secondary;
const struct fwnode_operations *ops;
struct device *dev;
};
. . . . . .
/**
* struct dev_iommu - Collection of per-device IOMMU data
*
* @fault_param: IOMMU detected device fault reporting data
* @iopf_param: I/O Page Fault queue and data
* @fwspec: IOMMU fwspec data
* @iommu_dev: IOMMU device this device is linked to
* @priv: IOMMU Driver private data
*
* TODO: migrate other per device data pointers under iommu_dev_data, e.g.
* struct iommu_group *iommu_group;
*/
struct dev_iommu {
struct mutex lock;
struct iommu_fault_param *fault_param;
struct iopf_device_param *iopf_param;
struct iommu_fwspec *fwspec;
struct iommu_device *iommu_dev;
void *priv;
};
. . . . . .
/**
* struct iommu_fwspec - per-device IOMMU instance data
* @ops: ops for this device's IOMMU
* @iommu_fwnode: firmware handle for this device's IOMMU
* @iommu_priv: IOMMU driver private data for this device
* @num_ids: number of associated device IDs
* @ids: IDs which this device may present to the IOMMU
*/
struct iommu_fwspec {
const struct iommu_ops *ops;
struct fwnode_handle *iommu_fwnode;
u32 flags;
unsigned int num_ids;
u32 ids[];
};
In IOMMU, each domain represents an IOMMU mapped address space, that is, a page table. A group logically needs to be bound to a domain, that is, all devices in a group are located in a domain. In the IOMMU subsystem of the Linux kernel, domain is represented by the struct iommu_domain structure, which is defined (located in include/linux/iommu.h file) as follows:
struct iommu_domain {
unsigned type;
const struct iommu_ops *ops;
unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */
iommu_fault_handler_t handler;
void *handler_token;
struct iommu_domain_geometry geometry;
void *iova_cookie;
struct mutex switch_log_lock;
};
The IOMMU subsystem of the Linux kernel uses the struct iommu_group structure to represent a device group located in the same domain, and uses the struct group_device structure to represent a device group. equipment. The two structure definitions (located in the drivers/iommu/iommu.c file) are as follows:
struct iommu_group {
struct kobject kobj;
struct kobject *devices_kobj;
struct list_head devices;
struct mutex mutex;
struct blocking_notifier_head notifier;
void *iommu_data;
void (*iommu_data_release)(void *iommu_data);
char *name;
int id;
struct iommu_domain *default_domain;
struct iommu_domain *domain;
struct list_head entry;
};
struct group_device {
struct list_head list;
struct device *dev;
char *name;
};
From the object-oriented programming method, it can be considered that in the ARM SMMUv3 driver, the struct iommu_device and struct iommu_domain structures have their specific implementations. That is, the struct arm_smmu_device and struct arm_smmu_domain structures inherit the struct iommu_device and struct iommu_domain structures. These two structures The body definition (located in the drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h file) is as follows:
/* An SMMUv3 instance */
struct arm_smmu_device {
struct device *dev;
void __iomem *base;
void __iomem *page1;
#define ARM_SMMU_FEAT_2_LVL_STRTAB (1 << 0)
#define ARM_SMMU_FEAT_2_LVL_CDTAB (1 << 1)
#define ARM_SMMU_FEAT_TT_LE (1 << 2)
#define ARM_SMMU_FEAT_TT_BE (1 << 3)
#define ARM_SMMU_FEAT_PRI (1 << 4)
#define ARM_SMMU_FEAT_ATS (1 << 5)
#define ARM_SMMU_FEAT_SEV (1 << 6)
#define ARM_SMMU_FEAT_MSI (1 << 7)
#define ARM_SMMU_FEAT_COHERENCY (1 << 8)
#define ARM_SMMU_FEAT_TRANS_S1 (1 << 9)
#define ARM_SMMU_FEAT_TRANS_S2 (1 << 10)
#define ARM_SMMU_FEAT_STALLS (1 << 11)
#define ARM_SMMU_FEAT_HYP (1 << 12)
#define ARM_SMMU_FEAT_STALL_FORCE (1 << 13)
#define ARM_SMMU_FEAT_VAX (1 << 14)
#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
#define ARM_SMMU_FEAT_BTM (1 << 16)
#define ARM_SMMU_FEAT_SVA (1 << 17)
#define ARM_SMMU_FEAT_E2H (1 << 18)
#define ARM_SMMU_FEAT_HA (1 << 19)
#define ARM_SMMU_FEAT_HD (1 << 20)
#define ARM_SMMU_FEAT_BBML1 (1 << 21)
#define ARM_SMMU_FEAT_BBML2 (1 << 22)
#define ARM_SMMU_FEAT_ECMDQ (1 << 23)
#define ARM_SMMU_FEAT_MPAM (1 << 24)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
#define ARM_SMMU_OPT_PAGE0_REGS_ONLY (1 << 1)
#define ARM_SMMU_OPT_MSIPOLL (1 << 2)
u32 options;
union {
u32 nr_ecmdq;
u32 ecmdq_enabled;
};
struct arm_smmu_ecmdq *__percpu *ecmdq;
struct arm_smmu_cmdq cmdq;
struct arm_smmu_evtq evtq;
struct arm_smmu_priq priq;
int gerr_irq;
int combined_irq;
unsigned long ias; /* IPA */
unsigned long oas; /* PA */
unsigned long pgsize_bitmap;
#define ARM_SMMU_MAX_ASIDS (1 << 16)
unsigned int asid_bits;
#define ARM_SMMU_MAX_VMIDS (1 << 16)
unsigned int vmid_bits;
DECLARE_BITMAP(vmid_map, ARM_SMMU_MAX_VMIDS);
unsigned int ssid_bits;
unsigned int sid_bits;
struct arm_smmu_strtab_cfg strtab_cfg;
/* IOMMU core code handle */
struct iommu_device iommu;
struct rb_root streams;
struct mutex streams_mutex;
unsigned int mpam_partid_max;
unsigned int mpam_pmg_max;
bool bypass;
};
. . . . . .
struct arm_smmu_domain {
struct arm_smmu_device *smmu;
struct mutex init_mutex; /* Protects smmu pointer */
struct io_pgtable_ops *pgtbl_ops;
bool stall_enabled;
bool non_strict;
atomic_t nr_ats_masters;
enum arm_smmu_domain_stage stage;
union {
struct arm_smmu_s1_cfg s1_cfg;
struct arm_smmu_s2_cfg s2_cfg;
};
struct iommu_domain domain;
/* Unused in aux domains */
struct list_head devices;
spinlock_t devices_lock;
struct list_head mmu_notifiers;
/* Auxiliary domain stuff */
struct arm_smmu_domain *parent;
ioasid_t ssid;
unsigned long aux_nr_devs;
};
In the ARM SMMUv3 driver, use the struct arm_smmu_master structure to describe the SMMU private data of the system I/O device connected to the SMMU. This structure definition (located in < a i=2>drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h file) as follows:
struct arm_smmu_stream {
u32 id;
struct arm_smmu_master *master;
struct rb_node node;
};
/* SMMU private data for each master */
struct arm_smmu_master {
struct arm_smmu_device *smmu;
struct device *dev;
struct arm_smmu_domain *domain;
struct list_head domain_head;
struct arm_smmu_stream *streams;
unsigned int num_streams;
bool ats_enabled;
bool stall_enabled;
bool pri_supported;
bool prg_resp_needs_ssid;
bool sva_enabled;
bool iopf_enabled;
bool auxd_enabled;
struct list_head bonds;
unsigned int ssid_bits;
};
From an object-oriented programming perspective, it can be considered that the struct arm_smmu_master structure inherits the struct dev_iommu structure.
The data structure of SMMU in the Linux kernel generally has the following structural relationship:
The above data structures basically contain pointers to struct device objects, and struct device contain pointers to several key IOMMU objects. struct device Objects are intermediaries between various parts, and related subsystems often find the operations or data they need through struct device objects. struct device The fields related to IOMMU in the structure mainly include the following:
struct device {
#ifdef CONFIG_DMA_OPS
const struct dma_map_ops *dma_ops;
#endif
. . . . . .
#ifdef CONFIG_DMA_DECLARE_COHERENT
struct dma_coherent_mem *dma_mem; /* internal for coherent mem
override */
#endif
. . . . . .
struct iommu_group *iommu_group;
struct dev_iommu *iommu;
. . . . . .
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
bool dma_coherent:1;
#endif
#ifdef CONFIG_DMA_OPS_BYPASS
bool dma_ops_bypass : 1;
#endif
};
In addition to these data structures of the IOMMU subsystem, a number of hardware-specific data structures are defined in the lower-level SMMU driver implementation, such as:
- Command queue item
struct arm_smmu_cmdq_ent, - Instruction sequence
struct arm_smmu_cmdq, - Extended command queue
struct arm_smmu_ecmdq, - Event Queue
struct arm_smmu_evtq, - PRI column
struct arm_smmu_priq, - L1 flow table descriptor in level 2 flow table
struct arm_smmu_strtab_l1_desc, - context descriptor
struct arm_smmu_ctx_desc - L1 table descriptor in level 2 context descriptor table
struct arm_smmu_l1_ctx_desc - Context descriptor configuration
struct arm_smmu_ctx_desc_cfg - Phase 1 conversion configuration
struct arm_smmu_s1_cfg - Phase 2 conversion configuration
struct arm_smmu_s2_cfg - Flow table configuration
struct arm_smmu_strtab_cfg
These hardware-specific data structures are basically the same as ARM's official hardware documentationSMMU Software Guide andThe data structures mentioned in ARM system memory management unit architecture specification version 3 have a strict one-to-one correspondence.
SMMU-related operations and processes, as well as access to SMMU, are implemented based on the above data structures. The hierarchical structure of this implementation is roughly as shown in the figure below:
The system I/O device discovery, detection, and binding initialization process with the driver and the system I/O device driver usually call the interface provided by the platform device subsystem and DMA subsystem, such as the platform device subsystem of_dma_configure()/of_dma_configure_id() and dma_alloc_coherent() functions of the DMA subsystem. The implementation of these functions is completed with the help of lower-level modules.
SMMUv3 device driver initialization
In the early stage of Linux kernel startup, IOMMU initialization will be performed. This mainly executes the iommu_init() function, which creates and adds iommu_groups kset. This function definition (located in drivers/iommu/iommu.c file) as follows:
static int __init iommu_init(void)
{
iommu_group_kset = kset_create_and_add("iommu_groups",
NULL, kernel_kobj);
BUG_ON(!iommu_group_kset);
iommu_debugfs_setup();
return 0;
}
core_initcall(iommu_init);
When the Linux kernel starts, you can pass in some command line parameters to configure IOMMU, including configuring the default domain typeiommu.passthrough and configuring DMA setup iommu.strict and iommu.prq_timeout used to configure the corresponding timeout for pages waiting for pending page requests. During the early startup of the Linux kernel, the IOMMU subsystem will be initialized. If IOMMU is not configured through the command line parameters of the Linux kernel, the default domain type will be set. The relevant code (located in drivers/iommu/iommu. c file) as follows:
static unsigned int iommu_def_domain_type __read_mostly;
static bool iommu_dma_strict __read_mostly;
static u32 iommu_cmd_line __read_mostly;
/*
* Timeout to wait for page response of a pending page request. This is
* intended as a basic safety net in case a pending page request is not
* responded for an exceptionally long time. Device may also implement
* its own protection mechanism against this exception.
* Units are in jiffies with a range between 1 - 100 seconds equivalent.
* Default to 10 seconds.
* Setting 0 means no timeout tracking.
*/
#define IOMMU_PAGE_RESPONSE_MAX_TIMEOUT (HZ * 100)
#define IOMMU_PAGE_RESPONSE_DEF_TIMEOUT (HZ * 10)
static unsigned long prq_timeout = IOMMU_PAGE_RESPONSE_DEF_TIMEOUT;
. . . . . .
#define IOMMU_CMD_LINE_DMA_API BIT(0)
static void iommu_set_cmd_line_dma_api(void)
{
iommu_cmd_line |= IOMMU_CMD_LINE_DMA_API;
}
static bool iommu_cmd_line_dma_api(void)
{
return !!(iommu_cmd_line & IOMMU_CMD_LINE_DMA_API);
}
. . . . . .
/*
* Use a function instead of an array here because the domain-type is a
* bit-field, so an array would waste memory.
*/
static const char *iommu_domain_type_str(unsigned int t)
{
switch (t) {
case IOMMU_DOMAIN_BLOCKED:
return "Blocked";
case IOMMU_DOMAIN_IDENTITY:
return "Passthrough";
case IOMMU_DOMAIN_UNMANAGED:
return "Unmanaged";
case IOMMU_DOMAIN_DMA:
return "Translated";
default:
return "Unknown";
}
}
static int __init iommu_subsys_init(void)
{
bool cmd_line = iommu_cmd_line_dma_api();
if (!cmd_line) {
if (IS_ENABLED(CONFIG_IOMMU_DEFAULT_PASSTHROUGH))
iommu_set_default_passthrough(false);
else
iommu_set_default_translated(false);
if (iommu_default_passthrough() && mem_encrypt_active()) {
pr_info("Memory encryption detected - Disabling default IOMMU Passthrough\n");
iommu_set_default_translated(false);
}
}
pr_info("Default domain type: %s %s\n",
iommu_domain_type_str(iommu_def_domain_type),
cmd_line ? "(set via kernel command line)" : "");
return 0;
}
subsys_initcall(iommu_subsys_init);
. . . . . .
static int __init iommu_set_def_domain_type(char *str)
{
bool pt;
int ret;
ret = kstrtobool(str, &pt);
if (ret)
return ret;
if (pt)
iommu_set_default_passthrough(true);
else
iommu_set_default_translated(true);
return 0;
}
early_param("iommu.passthrough", iommu_set_def_domain_type);
static int __init iommu_dma_setup(char *str)
{
return kstrtobool(str, &iommu_dma_strict);
}
early_param("iommu.strict", iommu_dma_setup);
static int __init iommu_set_prq_timeout(char *str)
{
int ret;
unsigned long timeout;
if (!str)
return -EINVAL;
ret = kstrtoul(str, 10, &timeout);
if (ret)
return ret;
timeout = timeout * HZ;
if (timeout > IOMMU_PAGE_RESPONSE_MAX_TIMEOUT)
return -EINVAL;
prq_timeout = timeout;
return 0;
}
early_param("iommu.prq_timeout", iommu_set_prq_timeout);
. . . . . .
void iommu_set_default_passthrough(bool cmd_line)
{
if (cmd_line)
iommu_set_cmd_line_dma_api();
iommu_def_domain_type = IOMMU_DOMAIN_IDENTITY;
}
void iommu_set_default_translated(bool cmd_line)
{
if (cmd_line)
iommu_set_cmd_line_dma_api();
iommu_def_domain_type = IOMMU_DOMAIN_DMA;
}
core_initcallThe function of is executed earlier than the function of subsys_initcall.
After the IOMMU subsystem is initialized, it is the SMMU device driver's turn. SMMUv3 itself is a platform device. Its hardware device information, including register mapping address range, interrupt number and other resources used, are described in the device treedts/dtsi file . A sample device node for an SMMUv3 device in the device tree file (located in arch/arm64/boot/dts/arm/fvp-base-revc.dts ) as follows:
smmu: iommu@2b400000 {
compatible = "arm,smmu-v3";
reg = <0x0 0x2b400000 0x0 0x100000>;
interrupts = <GIC_SPI 74 IRQ_TYPE_EDGE_RISING>,
<GIC_SPI 79 IRQ_TYPE_EDGE_RISING>,
<GIC_SPI 75 IRQ_TYPE_EDGE_RISING>,
<GIC_SPI 77 IRQ_TYPE_EDGE_RISING>;
interrupt-names = "eventq", "gerror", "priq", "cmdq-sync";
dma-coherent;
#iommu-cells = <1>;
msi-parent = <&its 0x10000>;
};
The entry point for loading the SMMUv3 device driver is thearm_smmu_device_probe() function, which is defined (located indrivers/iommu/arm/arm-smmu-v3/ arm-smmu-v3.c file) as follows:
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
{ 0, NULL},
};
static void parse_driver_options(struct arm_smmu_device *smmu)
{
int i = 0;
do {
if (of_property_read_bool(smmu->dev->of_node,
arm_smmu_options[i].prop)) {
smmu->options |= arm_smmu_options[i].opt;
dev_notice(smmu->dev, "option %s\n",
arm_smmu_options[i].prop);
}
} while (arm_smmu_options[++i].opt);
}
. . . . . .
static int arm_smmu_device_dt_probe(struct platform_device *pdev,
struct arm_smmu_device *smmu)
{
struct device *dev = &pdev->dev;
u32 cells;
int ret = -EINVAL;
if (of_property_read_u32(dev->of_node, "#iommu-cells", &cells))
dev_err(dev, "missing #iommu-cells property\n");
else if (cells != 1)
dev_err(dev, "invalid #iommu-cells value (%d)\n", cells);
else
ret = 0;
parse_driver_options(smmu);
if (of_dma_is_coherent(dev->of_node))
smmu->features |= ARM_SMMU_FEAT_COHERENCY;
return ret;
}
static unsigned long arm_smmu_resource_size(struct arm_smmu_device *smmu)
{
if (smmu->options & ARM_SMMU_OPT_PAGE0_REGS_ONLY)
return SZ_64K;
else
return SZ_128K;
}
. . . . . .
static void __iomem *arm_smmu_ioremap(struct device *dev, resource_size_t start,
resource_size_t size)
{
struct resource res = DEFINE_RES_MEM(start, size);
return devm_ioremap_resource(dev, &res);
}
. . . . . .
static int arm_smmu_device_probe(struct platform_device *pdev)
{
int irq, ret;
struct resource *res;
resource_size_t ioaddr;
struct arm_smmu_device *smmu;
struct device *dev = &pdev->dev;
smmu = devm_kzalloc(dev, sizeof(*smmu), GFP_KERNEL);
if (!smmu) {
dev_err(dev, "failed to allocate arm_smmu_device\n");
return -ENOMEM;
}
smmu->dev = dev;
if (dev->of_node) {
ret = arm_smmu_device_dt_probe(pdev, smmu);
} else {
ret = arm_smmu_device_acpi_probe(pdev, smmu);
if (ret == -ENODEV)
return ret;
}
/* Set bypass mode according to firmware probing result */
smmu->bypass = !!ret;
/* Base address */
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res)
return -EINVAL;
if (resource_size(res) < arm_smmu_resource_size(smmu)) {
dev_err(dev, "MMIO region too small (%pr)\n", res);
return -EINVAL;
}
ioaddr = res->start;
/*
* Don't map the IMPLEMENTATION DEFINED regions, since they may contain
* the PMCG registers which are reserved by the PMU driver.
*/
smmu->base = arm_smmu_ioremap(dev, ioaddr, ARM_SMMU_REG_SZ);
if (IS_ERR(smmu->base))
return PTR_ERR(smmu->base);
if (arm_smmu_resource_size(smmu) > SZ_64K) {
smmu->page1 = arm_smmu_ioremap(dev, ioaddr + SZ_64K,
ARM_SMMU_REG_SZ);
if (IS_ERR(smmu->page1))
return PTR_ERR(smmu->page1);
} else {
smmu->page1 = smmu->base;
}
/* Interrupt lines */
irq = platform_get_irq_byname_optional(pdev, "combined");
if (irq > 0)
smmu->combined_irq = irq;
else {
irq = platform_get_irq_byname_optional(pdev, "eventq");
if (irq > 0)
smmu->evtq.q.irq = irq;
irq = platform_get_irq_byname_optional(pdev, "priq");
if (irq > 0)
smmu->priq.q.irq = irq;
irq = platform_get_irq_byname_optional(pdev, "gerror");
if (irq > 0)
smmu->gerr_irq = irq;
}
/* Probe the h/w */
ret = arm_smmu_device_hw_probe(smmu);
if (ret)
return ret;
/* Initialise in-memory data structures */
ret = arm_smmu_init_structures(smmu);
if (ret)
return ret;
/* Record our private device structure */
platform_set_drvdata(pdev, smmu);
/* Reset the device */
ret = arm_smmu_device_reset(smmu, false);
if (ret)
return ret;
/* And we're up. Go go go! */
ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL,
"smmu3.%pa", &ioaddr);
if (ret)
return ret;
iommu_device_set_ops(&smmu->iommu, &arm_smmu_ops);
iommu_device_set_fwnode(&smmu->iommu, dev->fwnode);
ret = iommu_device_register(&smmu->iommu);
if (ret) {
dev_err(dev, "Failed to register iommu\n");
return ret;
}
return arm_smmu_set_bus_ops(&arm_smmu_ops);
}
arm_smmu_device_probe()The function mainly does the following things:
- Allocates
struct arm_smmu_deviceobject, which is used to describe the SMMUv3 device in the IOMMU subsystem. - Get the information contained in the SMMUv3 device node in the device tree file
dts/dtsi, and the referenced resources, which mainly include:- Information about SMMUv3 devices, such as
iommu-cells, its value must be 1; options, such as whether there is only register page 0, etc.; whether SMMU supports coherent, which is mainly determined by the device tree filedma-coherentattribute representation of the device node; - The register map of the SMMUv3 device,
arm_smmu_device_probe()The function will check whether the range size of the register map matches the expected value according to the value of options, and remap the register map of the SMMUv3 device; - Interrupt resources referenced by SMMUv3 devices, including those for the command queue, event queue, and global errors.
- Information about SMMUv3 devices, such as
- Detect the hardware characteristics of SMMUv3 devices, which mainly follow the registers defined inARM System Memory Management Unit Architecture Specification Version 3 SMMU_IDR0, SMMU_IDR1, SMMU_IDR3 , andSMMU_IDR5 (Other read-only registers used to provide information contain information that has little to do with the characteristics of SMMUv3 hardware devices. SMMU_IDR2 Contains information about features implemented by SMMU for non-secure programming interfaces, SMMU_IDR4 is an SMMU implementation-defined register, SMMU_IIDR The register contains SMMU implementation and implementer information, as well as supported architecture version information defined by the implementation ,SMMU_AIDR The register contains various fields of the SMMU architecture version number information that the SMMU implementation complies with), confirming the features supported by the actual SMMUv3 hardware device, which is mainly done by calling
arm_smmu_device_hw_probe()The function is completed. - Initialize the data structure, which mainly includes several queues and flow tables. The queues include command queue, event queue and PRIQ queue. For the initialization of the flow table, there are two situations. If the structure of the flow table is a linear flow table, all STEs in the linear flow table are configured as bypass SMMUs; if the structure of the flow table is a 2-level flow table, the flow There are invalid L1 flow table descriptors in the table. This is mainly done by calling the
arm_smmu_init_structures()function. - Record the object in the private fields of the device structure
struct platform_deviceobject.struct arm_smmu_device - Reset the SMMUv3 device, which mainly includes resetting the hardware device through registers such as SMMU_CR0, setting the flow table base address register, etc.; and setting interrupts, Including requesting an interrupt from the system and registering an interrupt handler; initializing the data structure to establish each data structure in the memory, and resetting the SMMUv3 device will write the base address and various configurations of each data structure into the corresponding device register, which is mainly done by calling < /span>
arm_smmu_device_reset()The function is completed. - Register the SMMUv3 device with the IOMMU subsystem, which includes setting up and for
struct iommu_deviceand setting < /span> function. is used to match SMMUv3 devices and system I/O devices, which is mainly done by calling the The object is registered into the IOMMU subsystem.struct iommu_opsstruct fwnode_handlestruct iommu_devicestruct fwnode_handleiommu_device_register() - Settings for each bus type
struct iommu_ops, the loading order of SMMUv3 device drivers and system I/O devices to use IOMMU may be undefined; normally, it should be SMMUv3 The device driver is loaded first, and the system I/O device that uses IOMMU is loaded later; here we will handle the situation where the system I/O device that uses IOMMU is loaded before the SMMUv3 device driver, which is mainly done by callingarm_smmu_set_bus_ops()The function is completed.
Probing the hardware characteristics of SMMUv3 devices
arm_smmu_device_probe() function callarm_smmu_device_hw_probe() The function detects the hardware characteristics of the SMMUv3 device, which is defined (located indrivers/iommu/arm/arm-smmu-v3/ arm-smmu-v3.c file) as follows:
static int arm_smmu_ecmdq_probe(struct arm_smmu_device *smmu)
{
int ret, cpu;
u32 i, nump, numq, gap;
u32 reg, shift_increment;
u64 addr, smmu_dma_base;
void __iomem *cp_regs, *cp_base;
/* IDR6 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR6);
smmu_reg_dump(smmu);
nump = 1 << FIELD_GET(IDR6_LOG2NUMP, reg);
numq = 1 << FIELD_GET(IDR6_LOG2NUMQ, reg);
smmu->nr_ecmdq = nump * numq;
gap = ECMDQ_CP_RRESET_SIZE >> FIELD_GET(IDR6_LOG2NUMQ, reg);
smmu_dma_base = (vmalloc_to_pfn(smmu->base) << PAGE_SHIFT);
cp_regs = ioremap(smmu_dma_base + ARM_SMMU_ECMDQ_CP_BASE, PAGE_SIZE);
if (!cp_regs)
return -ENOMEM;
for (i = 0; i < nump; i++) {
u64 val, pre_addr;
val = readq_relaxed(cp_regs + 32 * i);
if (!(val & ECMDQ_CP_PRESET)) {
iounmap(cp_regs);
dev_err(smmu->dev, "ecmdq control page %u is memory mode\n", i);
return -EFAULT;
}
if (i && ((val & ECMDQ_CP_ADDR) != (pre_addr + ECMDQ_CP_RRESET_SIZE))) {
iounmap(cp_regs);
dev_err(smmu->dev, "ecmdq_cp memory region is not contiguous\n");
return -EFAULT;
}
pre_addr = val & ECMDQ_CP_ADDR;
}
addr = readl_relaxed(cp_regs) & ECMDQ_CP_ADDR;
iounmap(cp_regs);
cp_base = devm_ioremap(smmu->dev, smmu_dma_base + addr, ECMDQ_CP_RRESET_SIZE * nump);
if (!cp_base)
return -ENOMEM;
smmu->ecmdq = devm_alloc_percpu(smmu->dev, struct arm_smmu_ecmdq *);
if (!smmu->ecmdq)
return -ENOMEM;
ret = arm_smmu_ecmdq_layout(smmu);
if (ret)
return ret;
shift_increment = order_base_2(num_possible_cpus() / smmu->nr_ecmdq);
addr = 0;
for_each_possible_cpu(cpu) {
struct arm_smmu_ecmdq *ecmdq;
struct arm_smmu_queue *q;
ecmdq = *per_cpu_ptr(smmu->ecmdq, cpu);
q = &ecmdq->cmdq.q;
/*
* The boot option "maxcpus=" can limit the number of online
* CPUs. The CPUs that are not selected are not showed in
* cpumask_of_node(node), their 'ecmdq' may be NULL.
*
* (q->ecmdq_prod & ECMDQ_PROD_EN) indicates that the ECMDQ is
* shared by multiple cores and has been initialized.
*/
if (!ecmdq || (q->ecmdq_prod & ECMDQ_PROD_EN))
continue;
ecmdq->base = cp_base + addr;
q->llq.max_n_shift = ECMDQ_MAX_SZ_SHIFT + shift_increment;
ret = arm_smmu_init_one_queue(smmu, q, ecmdq->base, ARM_SMMU_ECMDQ_PROD,
ARM_SMMU_ECMDQ_CONS, CMDQ_ENT_DWORDS, "ecmdq");
if (ret)
return ret;
q->ecmdq_prod = ECMDQ_PROD_EN;
rwlock_init(&q->ecmdq_lock);
ret = arm_smmu_ecmdq_init(&ecmdq->cmdq);
if (ret) {
dev_err(smmu->dev, "ecmdq[%d] init failed\n", i);
return ret;
}
addr += gap;
}
return 0;
}
static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg)
{
u32 fw_features = smmu->features & (ARM_SMMU_FEAT_HA | ARM_SMMU_FEAT_HD);
u32 features = 0;
switch (FIELD_GET(IDR0_HTTU, reg)) {
case IDR0_HTTU_ACCESS_DIRTY:
features |= ARM_SMMU_FEAT_HD;
fallthrough;
case IDR0_HTTU_ACCESS:
features |= ARM_SMMU_FEAT_HA;
}
if (smmu->dev->of_node)
smmu->features |= features;
else if (features != fw_features)
/* ACPI IORT sets the HTTU bits */
dev_warn(smmu->dev,
"IDR0.HTTU overridden by FW configuration (0x%x)\n",
fw_features);
}
static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
{
u32 reg;
bool coherent = smmu->features & ARM_SMMU_FEAT_COHERENCY;
bool vhe = cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN);
/* IDR0 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR0);
/* 2-level structures */
if (FIELD_GET(IDR0_ST_LVL, reg) == IDR0_ST_LVL_2LVL)
smmu->features |= ARM_SMMU_FEAT_2_LVL_STRTAB;
if (reg & IDR0_CD2L)
smmu->features |= ARM_SMMU_FEAT_2_LVL_CDTAB;
/*
* Translation table endianness.
* We currently require the same endianness as the CPU, but this
* could be changed later by adding a new IO_PGTABLE_QUIRK.
*/
switch (FIELD_GET(IDR0_TTENDIAN, reg)) {
case IDR0_TTENDIAN_MIXED:
smmu->features |= ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE;
break;
#ifdef __BIG_ENDIAN
case IDR0_TTENDIAN_BE:
smmu->features |= ARM_SMMU_FEAT_TT_BE;
break;
#else
case IDR0_TTENDIAN_LE:
smmu->features |= ARM_SMMU_FEAT_TT_LE;
break;
#endif
default:
dev_err(smmu->dev, "unknown/unsupported TT endianness!\n");
return -ENXIO;
}
/* Boolean feature flags */
if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI)
smmu->features |= ARM_SMMU_FEAT_PRI;
if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS)
smmu->features |= ARM_SMMU_FEAT_ATS;
if (reg & IDR0_SEV)
smmu->features |= ARM_SMMU_FEAT_SEV;
if (reg & IDR0_MSI) {
smmu->features |= ARM_SMMU_FEAT_MSI;
if (coherent && !disable_msipolling)
smmu->options |= ARM_SMMU_OPT_MSIPOLL;
}
if (reg & IDR0_HYP) {
smmu->features |= ARM_SMMU_FEAT_HYP;
if (vhe)
smmu->features |= ARM_SMMU_FEAT_E2H;
}
arm_smmu_get_httu(smmu, reg);
/*
* If the CPU is using VHE, but the SMMU doesn't support it, the SMMU
* will create TLB entries for NH-EL1 world and will miss the
* broadcasted TLB invalidations that target EL2-E2H world. Don't enable
* BTM in that case.
*/
if (reg & IDR0_BTM && (!vhe || reg & IDR0_HYP))
smmu->features |= ARM_SMMU_FEAT_BTM;
/*
* The coherency feature as set by FW is used in preference to the ID
* register, but warn on mismatch.
*/
if (!!(reg & IDR0_COHACC) != coherent)
dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n",
coherent ? "true" : "false");
switch (FIELD_GET(IDR0_STALL_MODEL, reg)) {
case IDR0_STALL_MODEL_FORCE:
smmu->features |= ARM_SMMU_FEAT_STALL_FORCE;
fallthrough;
case IDR0_STALL_MODEL_STALL:
smmu->features |= ARM_SMMU_FEAT_STALLS;
}
if (reg & IDR0_S1P)
smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
if (reg & IDR0_S2P)
smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
if (!(reg & (IDR0_S1P | IDR0_S2P))) {
dev_err(smmu->dev, "no translation support!\n");
return -ENXIO;
}
/* We only support the AArch64 table format at present */
switch (FIELD_GET(IDR0_TTF, reg)) {
case IDR0_TTF_AARCH32_64:
smmu->ias = 40;
fallthrough;
case IDR0_TTF_AARCH64:
break;
default:
dev_err(smmu->dev, "AArch64 table format not supported!\n");
return -ENXIO;
}
/* ASID/VMID sizes */
smmu->asid_bits = reg & IDR0_ASID16 ? 16 : 8;
smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
/* IDR1 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR1);
if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) {
dev_err(smmu->dev, "embedded implementation not supported\n");
return -ENXIO;
}
if (reg & IDR1_ECMDQ)
smmu->features |= ARM_SMMU_FEAT_ECMDQ;
/* Queue sizes, capped to ensure natural alignment */
smmu->cmdq.q.llq.max_n_shift = min_t(u32, CMDQ_MAX_SZ_SHIFT,
FIELD_GET(IDR1_CMDQS, reg));
if (smmu->cmdq.q.llq.max_n_shift <= ilog2(CMDQ_BATCH_ENTRIES)) {
/*
* We don't support splitting up batches, so one batch of
* commands plus an extra sync needs to fit inside the command
* queue. There's also no way we can handle the weird alignment
* restrictions on the base pointer for a unit-length queue.
*/
dev_err(smmu->dev, "command queue size <= %d entries not supported\n",
CMDQ_BATCH_ENTRIES);
return -ENXIO;
}
smmu->evtq.q.llq.max_n_shift = min_t(u32, EVTQ_MAX_SZ_SHIFT,
FIELD_GET(IDR1_EVTQS, reg));
smmu->priq.q.llq.max_n_shift = min_t(u32, PRIQ_MAX_SZ_SHIFT,
FIELD_GET(IDR1_PRIQS, reg));
/* SID/SSID sizes */
smmu->ssid_bits = FIELD_GET(IDR1_SSIDSIZE, reg);
smmu->sid_bits = FIELD_GET(IDR1_SIDSIZE, reg);
/*
* If the SMMU supports fewer bits than would fill a single L2 stream
* table, use a linear table instead.
*/
if (smmu->sid_bits <= STRTAB_SPLIT)
smmu->features &= ~ARM_SMMU_FEAT_2_LVL_STRTAB;
/* IDR3 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR3);
switch (FIELD_GET(IDR3_BBML, reg)) {
case IDR3_BBML0:
break;
case IDR3_BBML1:
smmu->features |= ARM_SMMU_FEAT_BBML1;
break;
case IDR3_BBML2:
smmu->features |= ARM_SMMU_FEAT_BBML2;
break;
default:
dev_err(smmu->dev, "unknown/unsupported BBM behavior level\n");
return -ENXIO;
}
if (FIELD_GET(IDR3_RIL, reg))
smmu->features |= ARM_SMMU_FEAT_RANGE_INV;
if (reg & IDR3_MPAM) {
reg = readl_relaxed(smmu->base + ARM_SMMU_MPAMIDR);
smmu->mpam_partid_max = FIELD_GET(MPAMIDR_PARTID_MAX, reg);
smmu->mpam_pmg_max = FIELD_GET(MPAMIDR_PMG_MAX, reg);
if (smmu->mpam_partid_max || smmu->mpam_pmg_max)
smmu->features |= ARM_SMMU_FEAT_MPAM;
}
/* IDR5 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);
/* Maximum number of outstanding stalls */
smmu->evtq.max_stalls = FIELD_GET(IDR5_STALL_MAX, reg);
/* Page sizes */
if (reg & IDR5_GRAN64K)
smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
if (reg & IDR5_GRAN16K)
smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
if (reg & IDR5_GRAN4K)
smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
/* Input address size */
if (FIELD_GET(IDR5_VAX, reg) == IDR5_VAX_52_BIT)
smmu->features |= ARM_SMMU_FEAT_VAX;
/* Output address size */
switch (FIELD_GET(IDR5_OAS, reg)) {
case IDR5_OAS_32_BIT:
smmu->oas = 32;
break;
case IDR5_OAS_36_BIT:
smmu->oas = 36;
break;
case IDR5_OAS_40_BIT:
smmu->oas = 40;
break;
case IDR5_OAS_42_BIT:
smmu->oas = 42;
break;
case IDR5_OAS_44_BIT:
smmu->oas = 44;
break;
case IDR5_OAS_52_BIT:
smmu->oas = 52;
smmu->pgsize_bitmap |= 1ULL << 42; /* 4TB */
break;
default:
dev_info(smmu->dev,
"unknown output address size. Truncating to 48-bit\n");
fallthrough;
case IDR5_OAS_48_BIT:
smmu->oas = 48;
}
if (arm_smmu_ops.pgsize_bitmap == -1UL)
arm_smmu_ops.pgsize_bitmap = smmu->pgsize_bitmap;
else
arm_smmu_ops.pgsize_bitmap |= smmu->pgsize_bitmap;
/* Set the DMA mask for our table walker */
if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(smmu->oas)))
dev_warn(smmu->dev,
"failed to set DMA mask for table walker\n");
smmu->ias = max(smmu->ias, smmu->oas);
if (arm_smmu_sva_supported(smmu))
smmu->features |= ARM_SMMU_FEAT_SVA;
dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
smmu->ias, smmu->oas, smmu->features);
if (smmu->features & ARM_SMMU_FEAT_ECMDQ) {
int err;
err = arm_smmu_ecmdq_probe(smmu);
if (err) {
dev_err(smmu->dev, "suppress ecmdq feature, errno=%d\n", err);
smmu->ecmdq_enabled = 0;
}
}
return 0;
}
In the struct arm_smmu_device structure, the SMMUv3 driver uses a 32-bit value to describe the supported hardware features, where each feature is represented by one bit. Function arm_smmu_device_hw_probe() obtains the hardware characteristics of the SMMU by reading its registers.
SMMU_IDR0 Parasitic organ:
- Whether to support two-level flow table
- Whether two-level context descriptor (CD) tables are supported
- Endianness of supported conversion tables
- Whether to support PRI
- Whether to support ATS
- Whether to support SEV
- Whether to support MSI
- Whether to support HYP
- HTTU characteristics
- Whether to support BTM
- Whether to support COHACC
- Whether to support STALL
- Whether stage 1 conversion is supported
- Whether stage 2 conversion is supported
- The value of IAS (input address size)
- ASID bits
- VMID bits
SMMU_IDR1 register (some fields are ignored, such as ATTR_TYPE_OVR and ATTR_PERMS_OVR):
- Are the flow table base address and flow table configuration fixed?
- Are the base addresses of the command queue, event queue and PRI queue fixed?
- When the base address is fixed, does the base address register contain an absolute address or a relative address? The SMMUv3 device driver requires that the flow table base address and flow table configuration are not fixed. The base addresses of the command queue, event queue and PRI queue are not fixed.
- Whether to support extended command queue
- Command queue, event queue, and PRI queue sizes
- StreamID SID size
- SubstreamID SSID size
SMMU_IDR3 register (some fields are ignored):
- Supported BBML
- Whether to support RIL
- Whether MPAM is supported. When MPAM is supported, the MPAM register will also be read to obtain more information.
SMMU_IDR5 Parasitic organ:
- The maximum number of outstanding stalled transactions supported by the SMMU and system.
- Supported page sizes
- Virtual address extension VAX, the supported virtual address size
- Output address size OAS
In addition, thearm_smmu_device_hw_probe() function will also detect whether SVA is supported. When it is detected that the extended command queue is supported, it will also readARM_SMMU_IDR6< /span> Register detects the characteristics of ECMDQ.
arm_smmu_device_hw_probe() The function is executed according to the meaning of each field of the register defined in ARM System Memory Management Unit Architecture Specification Version 3.
Initialize data structure
Initializing the data structure is mainly completed by calling the arm_smmu_init_structures() function. This function definition (located in drivers/iommu/arm/arm-smmu-v3/arm -smmu-v3.c file) as follows:
/* Stream table manipulation functions */
static void
arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
{
u64 val = 0;
val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span);
val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK;
/* See comment in arm_smmu_write_ctx_desc() */
WRITE_ONCE(*dst, cpu_to_le64(val));
}
static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
{
struct arm_smmu_cmdq_ent cmd = {
.opcode = CMDQ_OP_CFGI_STE,
.cfgi = {
.sid = sid,
.leaf = true,
},
};
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
}
static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
__le64 *dst)
{
/*
* This is hideously complicated, but we only really care about
* three cases at the moment:
*
* 1. Invalid (all zero) -> bypass/fault (init)
* 2. Bypass/fault -> translation/bypass (attach)
* 3. Translation/bypass -> bypass/fault (detach)
*
* Given that we can't update the STE atomically and the SMMU
* doesn't read the thing in a defined order, that leaves us
* with the following maintenance requirements:
*
* 1. Update Config, return (init time STEs aren't live)
* 2. Write everything apart from dword 0, sync, write dword 0, sync
* 3. Update Config, sync
*/
u64 val = le64_to_cpu(dst[0]);
bool ste_live = false;
struct arm_smmu_device *smmu = NULL;
struct arm_smmu_s1_cfg *s1_cfg = NULL;
struct arm_smmu_s2_cfg *s2_cfg = NULL;
struct arm_smmu_domain *smmu_domain = NULL;
struct arm_smmu_cmdq_ent prefetch_cmd = {
.opcode = CMDQ_OP_PREFETCH_CFG,
.prefetch = {
.sid = sid,
},
};
if (master) {
smmu_domain = master->domain;
smmu = master->smmu;
}
if (smmu_domain) {
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_S1:
s1_cfg = &smmu_domain->s1_cfg;
break;
case ARM_SMMU_DOMAIN_S2:
case ARM_SMMU_DOMAIN_NESTED:
s2_cfg = &smmu_domain->s2_cfg;
break;
default:
break;
}
}
if (val & STRTAB_STE_0_V) {
switch (FIELD_GET(STRTAB_STE_0_CFG, val)) {
case STRTAB_STE_0_CFG_BYPASS:
break;
case STRTAB_STE_0_CFG_S1_TRANS:
case STRTAB_STE_0_CFG_S2_TRANS:
ste_live = true;
break;
case STRTAB_STE_0_CFG_ABORT:
BUG_ON(!disable_bypass);
break;
default:
BUG(); /* STE corruption */
}
}
/* Nuke the existing STE_0 value, as we're going to rewrite it */
val = STRTAB_STE_0_V;
/* Bypass/fault */
if (!smmu_domain || !(s1_cfg || s2_cfg)) {
if (!smmu_domain && disable_bypass)
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT);
else
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_BYPASS);
dst[0] = cpu_to_le64(val);
dst[1] = cpu_to_le64(FIELD_PREP(STRTAB_STE_1_SHCFG,
STRTAB_STE_1_SHCFG_INCOMING));
dst[2] = 0; /* Nuke the VMID */
/*
* The SMMU can perform negative caching, so we must sync
* the STE regardless of whether the old value was live.
*/
if (smmu)
arm_smmu_sync_ste_for_sid(smmu, sid);
return;
}
if (s1_cfg) {
u64 strw = smmu->features & ARM_SMMU_FEAT_E2H ?
STRTAB_STE_1_STRW_EL2 : STRTAB_STE_1_STRW_NSEL1;
BUG_ON(ste_live);
dst[1] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
FIELD_PREP(STRTAB_STE_1_STRW, strw));
if (master->prg_resp_needs_ssid)
dst[1] |= cpu_to_le64(STRTAB_STE_1_PPAR);
if (smmu->features & ARM_SMMU_FEAT_STALLS &&
!master->stall_enabled)
dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD);
val |= (s1_cfg->cdcfg.cdtab_dma & STRTAB_STE_0_S1CTXPTR_MASK) |
FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) |
FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) |
FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt);
}
if (s2_cfg) {
BUG_ON(ste_live);
dst[2] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_2_S2VMID, s2_cfg->vmid) |
FIELD_PREP(STRTAB_STE_2_VTCR, s2_cfg->vtcr) |
#ifdef __BIG_ENDIAN
STRTAB_STE_2_S2ENDI |
#endif
STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2AA64 |
STRTAB_STE_2_S2R);
dst[3] = cpu_to_le64(s2_cfg->vttbr & STRTAB_STE_3_S2TTB_MASK);
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S2_TRANS);
}
if (master->ats_enabled)
dst[1] |= cpu_to_le64(FIELD_PREP(STRTAB_STE_1_EATS,
STRTAB_STE_1_EATS_TRANS));
pr_info("arm_smmu_write_strtab_ent[%d], val[0]=0x%llx, val[1]=0x%llx, val[2]=0x%llx, val[3]=0x%llx\n",
sid, val, dst[1], dst[2], dst[3]);
arm_smmu_sync_ste_for_sid(smmu, sid);
/* See comment in arm_smmu_write_ctx_desc() */
WRITE_ONCE(dst[0], cpu_to_le64(val));
arm_smmu_sync_ste_for_sid(smmu, sid);
/* It's likely that we'll want to use the new STE soon */
if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
}
static void arm_smmu_init_bypass_stes(__le64 *strtab, unsigned int nent)
{
unsigned int i;
for (i = 0; i < nent; ++i) {
arm_smmu_write_strtab_ent(NULL, -1, strtab);
strtab += STRTAB_STE_DWORDS;
}
}
. . . . . .
/* Probing and initialisation functions */
static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
struct arm_smmu_queue *q,
void __iomem *page,
unsigned long prod_off,
unsigned long cons_off,
size_t dwords, const char *name)
{
size_t qsz;
do {
qsz = ((1 << q->llq.max_n_shift) * dwords) << 3;
q->base = dmam_alloc_coherent(smmu->dev, qsz, &q->base_dma,
GFP_KERNEL);
if (q->base || qsz < PAGE_SIZE)
break;
q->llq.max_n_shift--;
} while (1);
if (!q->base) {
dev_err(smmu->dev,
"failed to allocate queue (0x%zx bytes) for %s\n",
qsz, name);
return -ENOMEM;
}
if (!WARN_ON(q->base_dma & (qsz - 1))) {
dev_info(smmu->dev, "allocated %u entries for %s\n",
1 << q->llq.max_n_shift, name);
}
q->prod_reg = page + prod_off;
q->cons_reg = page + cons_off;
q->ent_dwords = dwords;
q->q_base = Q_BASE_RWA;
q->q_base |= q->base_dma & Q_BASE_ADDR_MASK;
q->q_base |= FIELD_PREP(Q_BASE_LOG2SIZE, q->llq.max_n_shift);
q->llq.prod = q->llq.cons = 0;
return 0;
}
static void arm_smmu_cmdq_free_bitmap(void *data)
{
unsigned long *bitmap = data;
bitmap_free(bitmap);
}
static int arm_smmu_cmdq_init(struct arm_smmu_device *smmu)
{
int ret = 0;
struct arm_smmu_cmdq *cmdq = &smmu->cmdq;
unsigned int nents = 1 << cmdq->q.llq.max_n_shift;
atomic_long_t *bitmap;
cmdq->shared = 1;
atomic_set(&cmdq->owner_prod, 0);
atomic_set(&cmdq->lock, 0);
bitmap = (atomic_long_t *)bitmap_zalloc(nents, GFP_KERNEL);
if (!bitmap) {
dev_err(smmu->dev, "failed to allocate cmdq bitmap\n");
ret = -ENOMEM;
} else {
cmdq->valid_map = bitmap;
devm_add_action(smmu->dev, arm_smmu_cmdq_free_bitmap, bitmap);
}
return ret;
}
static int arm_smmu_ecmdq_init(struct arm_smmu_cmdq *cmdq)
{
unsigned int nents = 1 << cmdq->q.llq.max_n_shift;
atomic_set(&cmdq->owner_prod, 0);
atomic_set(&cmdq->lock, 0);
cmdq->valid_map = (atomic_long_t *)bitmap_zalloc(nents, GFP_KERNEL);
if (!cmdq->valid_map)
return -ENOMEM;
return 0;
}
static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
{
int ret;
/* cmdq */
ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, smmu->base,
ARM_SMMU_CMDQ_PROD, ARM_SMMU_CMDQ_CONS,
CMDQ_ENT_DWORDS, "cmdq");
if (ret)
return ret;
ret = arm_smmu_cmdq_init(smmu);
if (ret)
return ret;
/* evtq */
ret = arm_smmu_init_one_queue(smmu, &smmu->evtq.q, smmu->page1,
ARM_SMMU_EVTQ_PROD, ARM_SMMU_EVTQ_CONS,
EVTQ_ENT_DWORDS, "evtq");
if (ret)
return ret;
if ((smmu->features & ARM_SMMU_FEAT_SVA) &&
(smmu->features & ARM_SMMU_FEAT_STALLS)) {
smmu->evtq.iopf = iopf_queue_alloc(dev_name(smmu->dev));
if (!smmu->evtq.iopf)
return -ENOMEM;
}
/* priq */
if (!(smmu->features & ARM_SMMU_FEAT_PRI))
return 0;
if (smmu->features & ARM_SMMU_FEAT_SVA) {
smmu->priq.iopf = iopf_queue_alloc(dev_name(smmu->dev));
if (!smmu->priq.iopf)
return -ENOMEM;
}
init_waitqueue_head(&smmu->priq.wq);
smmu->priq.batch = 0;
return arm_smmu_init_one_queue(smmu, &smmu->priq.q, smmu->page1,
ARM_SMMU_PRIQ_PROD, ARM_SMMU_PRIQ_CONS,
PRIQ_ENT_DWORDS, "priq");
}
static int arm_smmu_init_l1_strtab(struct arm_smmu_device *smmu)
{
unsigned int i;
struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
size_t size = sizeof(*cfg->l1_desc) * cfg->num_l1_ents;
void *strtab = smmu->strtab_cfg.strtab;
cfg->l1_desc = devm_kzalloc(smmu->dev, size, GFP_KERNEL);
if (!cfg->l1_desc) {
dev_err(smmu->dev, "failed to allocate l1 stream table desc\n");
return -ENOMEM;
}
for (i = 0; i < cfg->num_l1_ents; ++i) {
arm_smmu_write_strtab_l1_desc(strtab, &cfg->l1_desc[i]);
strtab += STRTAB_L1_DESC_DWORDS << 3;
}
return 0;
}
#ifdef CONFIG_SMMU_BYPASS_DEV
static void arm_smmu_install_bypass_ste_for_dev(struct arm_smmu_device *smmu,
u32 sid)
{
u64 val;
__le64 *step = arm_smmu_get_step_for_sid(smmu, sid);
if (!step)
return;
val = STRTAB_STE_0_V;
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_BYPASS);
step[0] = cpu_to_le64(val);
step[1] = cpu_to_le64(FIELD_PREP(STRTAB_STE_1_SHCFG,
STRTAB_STE_1_SHCFG_INCOMING));
step[2] = 0;
}
static int arm_smmu_prepare_init_l2_strtab(struct device *dev, void *data)
{
u32 sid;
int ret;
struct pci_dev *pdev;
struct arm_smmu_device *smmu = (struct arm_smmu_device *)data;
if (!arm_smmu_device_domain_type(dev))
return 0;
pdev = to_pci_dev(dev);
sid = PCI_DEVID(pdev->bus->number, pdev->devfn);
if (!arm_smmu_sid_in_range(smmu, sid))
return -ERANGE;
ret = arm_smmu_init_l2_strtab(smmu, sid);
if (ret)
return ret;
arm_smmu_install_bypass_ste_for_dev(smmu, sid);
return 0;
}
#endif
static int arm_smmu_init_strtab_2lvl(struct arm_smmu_device *smmu)
{
void *strtab;
u64 reg;
u32 size, l1size;
struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
#ifdef CONFIG_SMMU_BYPASS_DEV
int ret;
#endif
/* Calculate the L1 size, capped to the SIDSIZE. */
size = STRTAB_L1_SZ_SHIFT - (ilog2(STRTAB_L1_DESC_DWORDS) + 3);
size = min(size, smmu->sid_bits - STRTAB_SPLIT);
cfg->num_l1_ents = 1 << size;
size += STRTAB_SPLIT;
if (size < smmu->sid_bits)
dev_warn(smmu->dev,
"2-level strtab only covers %u/%u bits of SID\n",
size, smmu->sid_bits);
l1size = cfg->num_l1_ents * (STRTAB_L1_DESC_DWORDS << 3);
strtab = dmam_alloc_coherent(smmu->dev, l1size, &cfg->strtab_dma,
GFP_KERNEL);
if (!strtab) {
dev_err(smmu->dev,
"failed to allocate l1 stream table (%u bytes)\n",
l1size);
return -ENOMEM;
}
cfg->strtab = strtab;
/* Configure strtab_base_cfg for 2 levels */
reg = FIELD_PREP(STRTAB_BASE_CFG_FMT, STRTAB_BASE_CFG_FMT_2LVL);
reg |= FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, size);
reg |= FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT);
cfg->strtab_base_cfg = reg;
#ifdef CONFIG_SMMU_BYPASS_DEV
ret = arm_smmu_init_l1_strtab(smmu);
if (ret)
return ret;
if (smmu_bypass_devices_num) {
ret = bus_for_each_dev(&pci_bus_type, NULL, (void *)smmu,
arm_smmu_prepare_init_l2_strtab);
}
return ret;
#else
return arm_smmu_init_l1_strtab(smmu);
#endif
}
static int arm_smmu_init_strtab_linear(struct arm_smmu_device *smmu)
{
void *strtab;
u64 reg;
u32 size;
struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
size = (1 << smmu->sid_bits) * (STRTAB_STE_DWORDS << 3);
strtab = dmam_alloc_coherent(smmu->dev, size, &cfg->strtab_dma,
GFP_KERNEL);
if (!strtab) {
dev_err(smmu->dev,
"failed to allocate linear stream table (%u bytes)\n",
size);
return -ENOMEM;
}
cfg->strtab = strtab;
cfg->num_l1_ents = 1 << smmu->sid_bits;
/* Configure strtab_base_cfg for a linear table covering all SIDs */
reg = FIELD_PREP(STRTAB_BASE_CFG_FMT, STRTAB_BASE_CFG_FMT_LINEAR);
reg |= FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits);
cfg->strtab_base_cfg = reg;
arm_smmu_init_bypass_stes(strtab, cfg->num_l1_ents);
return 0;
}
static int arm_smmu_init_strtab(struct arm_smmu_device *smmu)
{
u64 reg;
int ret;
if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)
ret = arm_smmu_init_strtab_2lvl(smmu);
else
ret = arm_smmu_init_strtab_linear(smmu);
if (ret)
return ret;
/* Set the strtab base address */
reg = smmu->strtab_cfg.strtab_dma & STRTAB_BASE_ADDR_MASK;
reg |= STRTAB_BASE_RA;
smmu->strtab_cfg.strtab_base = reg;
/* Allocate the first VMID for stage-2 bypass STEs */
set_bit(0, smmu->vmid_map);
return 0;
}
static int arm_smmu_init_structures(struct arm_smmu_device *smmu)
{
int ret;
mutex_init(&smmu->streams_mutex);
smmu->streams = RB_ROOT;
ret = arm_smmu_init_queues(smmu);
if (ret)
return ret;
return arm_smmu_init_strtab(smmu);
}
arm_smmu_init_structures()The data structures initialized by the function mainly include command queue and event queue. When the SMMUv3 hardware device supports PRI, the PRI queue and flow table will be initialized.
The initialization of the queue is mainly completed through the arm_smmu_init_queues()/arm_smmu_init_one_queue() function. The process is roughly as follows:
- Allocate memory for the queue,SMMU_IDR1 There are fields in the register that describe the maximum size of the supported command queue, event queue and PRIQ queue. This Size is the base-2 logarithm of the maximum number of items contained in a supported queue. When allocating memory for the queue, it will try to start by allocating the maximum amount of memory and gradually reduce it by half until the memory allocation is successful or the queue size is less than one memory page in bytes, which means the memory allocation failed. , then an error will be reported and exited.
- Initializes the producer and consumer register addresses of the queue, and the size of the queue entries in 64 bits.
- Construct the value of the queue base address register based on the address of the memory allocated for the queue and the queue size. This value will be written laterSMMU_CMDQ_BASE, SMMU_EVENTQ_BASE and SMMU_PRIQ_BASE and other registers.
The initialization of the flow table is mainly completed through arm_smmu_init_strtab() and the functions it calls arm_smmu_init_strtab_2lvl()/arm_smmu_init_strtab_linear(). This process is roughly As follows:
- If the SMMUv3 hardware device supports level 2 flow tables, create a level 2 flow table:
- SMMU_STRTAB_BASE_CFG There are several bits in the register that can be used to configure the split point of StreamID when using multi-level flow tables, that is, how many bits are used to index the first-level flow table, and how many Bits are used to index the level 2 flow table, and several bits can be used to configure the bit length of StreamID. In addition, SMMUv3 can be obtained from the SMMU_IDR1 register The longest StreamID bit length supported by the hardware device;
- SMMUv3 device driver takes the bit length of the second-level flow table asSTRTAB_SPLIT bits, that is, 8 bits, and takes the first-level flow table The maximum occupancy of 1 MB of memory space is used to calculate the bit length of the first-level flow table, the number of items in the first-level flow table, and the required memory space in bytes;
- Allocate memory for the level 1 flow table;
- Initialize the flow table configuration structure
struct arm_smmu_strtab_cfg's flow table base address, number of flow table items, and flow table configuration values. The flow table configuration values will be used later. WriteSMMU_STRTAB_BASE_CFG register; - Call
arm_smmu_init_l1_strtab()function initializes the level 1 flow table. The SMMUv3 device driver maintains two L1 flow table descriptor tables. One is mainly accessed by the SMMUv3 driver and the other is for SMMUv3 hardware. Access, the former is represented bystruct arm_smmu_strtab_l1_descstructure array. Inarm_smmu_init_l1_strtab()function,struct arm_smmu_strtab_l1_descstructure array will be created and initialized to invalid L1 flow table descriptors and writes the contents of these objects to the level 1 flow table.
- SMMUv3 hardware devices do not support level 2 flow tables. Create a linear flow table:
- According to the bit length of StreamID previously obtained fromSMMU_IDR1 register, calculate the memory space required for the linear flow table in bytes size;
- Allocate memory for the linear flow table;
- Initialize the flow table configuration structure
struct arm_smmu_strtab_cfg's flow table base address, number of flow table items, and flow table configuration values. The flow table configuration values will be used later. WriteSMMU_STRTAB_BASE_CFG register; - Call the
arm_smmu_init_bypass_stes()function to initialize all STEs in the linear flow table as bypass SMMUs.
- Construct the value of the flow table base address register based on the base address of the flow table created in memory. This value will be written laterSMMU_STRTAB_BASERegister.
arm_smmu_init_structures() Functions are executed according to the data structures and their relationships defined in ARM System Memory Management Unit Architecture Specification Version 3.
Reset SMMUv3 device
Resetting the SMMUv3 device is mainly completed by calling the arm_smmu_device_reset() function. This function is defined (located in drivers/iommu/arm/arm-smmu-v3/arm -smmu-v3.c file) as follows:
static int arm_smmu_write_reg_sync(struct arm_smmu_device *smmu, u32 val,
unsigned int reg_off, unsigned int ack_off)
{
u32 reg;
writel_relaxed(val, smmu->base + reg_off);
return readl_relaxed_poll_timeout(smmu->base + ack_off, reg, reg == val,
1, ARM_SMMU_POLL_TIMEOUT_US);
}
/* GBPA is "special" */
static int arm_smmu_update_gbpa(struct arm_smmu_device *smmu, u32 set, u32 clr)
{
int ret;
u32 reg, __iomem *gbpa = smmu->base + ARM_SMMU_GBPA;
ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
1, ARM_SMMU_POLL_TIMEOUT_US);
if (ret)
return ret;
reg &= ~clr;
reg |= set;
writel_relaxed(reg | GBPA_UPDATE, gbpa);
ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
1, ARM_SMMU_POLL_TIMEOUT_US);
if (ret)
dev_err(smmu->dev, "GBPA not responding to update\n");
return ret;
}
static void arm_smmu_free_msis(void *data)
{
struct device *dev = data;
platform_msi_domain_free_irqs(dev);
}
static void arm_smmu_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
{
phys_addr_t doorbell;
struct device *dev = msi_desc_to_dev(desc);
struct arm_smmu_device *smmu = dev_get_drvdata(dev);
phys_addr_t *cfg = arm_smmu_msi_cfg[desc->platform.msi_index];
doorbell = (((u64)msg->address_hi) << 32) | msg->address_lo;
doorbell &= MSI_CFG0_ADDR_MASK;
#ifdef CONFIG_PM_SLEEP
/* Saves the msg (base addr of msi irq) and restores it during resume */
desc->msg.address_lo = msg->address_lo;
desc->msg.address_hi = msg->address_hi;
desc->msg.data = msg->data;
#endif
writeq_relaxed(doorbell, smmu->base + cfg[0]);
writel_relaxed(msg->data, smmu->base + cfg[1]);
writel_relaxed(ARM_SMMU_MEMATTR_DEVICE_nGnRE, smmu->base + cfg[2]);
}
static void arm_smmu_setup_msis(struct arm_smmu_device *smmu)
{
struct msi_desc *desc;
int ret, nvec = ARM_SMMU_MAX_MSIS;
struct device *dev = smmu->dev;
/* Clear the MSI address regs */
writeq_relaxed(0, smmu->base + ARM_SMMU_GERROR_IRQ_CFG0);
writeq_relaxed(0, smmu->base + ARM_SMMU_EVTQ_IRQ_CFG0);
if (smmu->features & ARM_SMMU_FEAT_PRI)
writeq_relaxed(0, smmu->base + ARM_SMMU_PRIQ_IRQ_CFG0);
else
nvec--;
if (!(smmu->features & ARM_SMMU_FEAT_MSI))
return;
if (!dev->msi_domain) {
dev_info(smmu->dev, "msi_domain absent - falling back to wired irqs\n");
return;
}
/* Allocate MSIs for evtq, gerror and priq. Ignore cmdq */
ret = platform_msi_domain_alloc_irqs(dev, nvec, arm_smmu_write_msi_msg);
if (ret) {
dev_warn(dev, "failed to allocate MSIs - falling back to wired irqs\n");
return;
}
for_each_msi_entry(desc, dev) {
switch (desc->platform.msi_index) {
case EVTQ_MSI_INDEX:
smmu->evtq.q.irq = desc->irq;
break;
case GERROR_MSI_INDEX:
smmu->gerr_irq = desc->irq;
break;
case PRIQ_MSI_INDEX:
smmu->priq.q.irq = desc->irq;
break;
default: /* Unknown */
continue;
}
}
/* Add callback to free MSIs on teardown */
devm_add_action(dev, arm_smmu_free_msis, dev);
}
#ifdef CONFIG_PM_SLEEP
static void arm_smmu_resume_msis(struct arm_smmu_device *smmu)
{
struct msi_desc *desc;
struct device *dev = smmu->dev;
for_each_msi_entry(desc, dev) {
switch (desc->platform.msi_index) {
case EVTQ_MSI_INDEX:
case GERROR_MSI_INDEX:
case PRIQ_MSI_INDEX: {
phys_addr_t *cfg = arm_smmu_msi_cfg[desc->platform.msi_index];
struct msi_msg *msg = &desc->msg;
phys_addr_t doorbell = (((u64)msg->address_hi) << 32) | msg->address_lo;
doorbell &= MSI_CFG0_ADDR_MASK;
writeq_relaxed(doorbell, smmu->base + cfg[0]);
writel_relaxed(msg->data, smmu->base + cfg[1]);
writel_relaxed(ARM_SMMU_MEMATTR_DEVICE_nGnRE,
smmu->base + cfg[2]);
break;
}
default:
continue;
}
}
}
#else
static void arm_smmu_resume_msis(struct arm_smmu_device *smmu)
{
}
#endif
static void arm_smmu_setup_unique_irqs(struct arm_smmu_device *smmu, bool resume)
{
int irq, ret;
if (!resume)
arm_smmu_setup_msis(smmu);
else {
/* The irq doesn't need to be re-requested during resume */
arm_smmu_resume_msis(smmu);
return;
}
/* Request interrupt lines */
irq = smmu->evtq.q.irq;
if (irq) {
ret = devm_request_threaded_irq(smmu->dev, irq, NULL,
arm_smmu_evtq_thread,
IRQF_ONESHOT,
"arm-smmu-v3-evtq", smmu);
if (ret < 0)
dev_warn(smmu->dev, "failed to enable evtq irq\n");
} else {
dev_warn(smmu->dev, "no evtq irq - events will not be reported!\n");
}
irq = smmu->gerr_irq;
if (irq) {
ret = devm_request_irq(smmu->dev, irq, arm_smmu_gerror_handler,
0, "arm-smmu-v3-gerror", smmu);
if (ret < 0)
dev_warn(smmu->dev, "failed to enable gerror irq\n");
} else {
dev_warn(smmu->dev, "no gerr irq - errors will not be reported!\n");
}
if (smmu->features & ARM_SMMU_FEAT_PRI) {
irq = smmu->priq.q.irq;
if (irq) {
ret = devm_request_threaded_irq(smmu->dev, irq, NULL,
arm_smmu_priq_thread,
IRQF_ONESHOT,
"arm-smmu-v3-priq",
smmu);
if (ret < 0)
dev_warn(smmu->dev,
"failed to enable priq irq\n");
} else {
dev_warn(smmu->dev, "no priq irq - PRI will be broken\n");
}
}
}
static int arm_smmu_setup_irqs(struct arm_smmu_device *smmu, bool resume)
{
int ret, irq;
u32 irqen_flags = IRQ_CTRL_EVTQ_IRQEN | IRQ_CTRL_GERROR_IRQEN;
/* Disable IRQs first */
ret = arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_IRQ_CTRL,
ARM_SMMU_IRQ_CTRLACK);
if (ret) {
dev_err(smmu->dev, "failed to disable irqs\n");
return ret;
}
irq = smmu->combined_irq;
if (irq) {
/*
* Cavium ThunderX2 implementation doesn't support unique irq
* lines. Use a single irq line for all the SMMUv3 interrupts.
*/
ret = devm_request_threaded_irq(smmu->dev, irq,
arm_smmu_combined_irq_handler,
arm_smmu_combined_irq_thread,
IRQF_ONESHOT,
"arm-smmu-v3-combined-irq", smmu);
if (ret < 0)
dev_warn(smmu->dev, "failed to enable combined irq\n");
} else
arm_smmu_setup_unique_irqs(smmu, resume);
if (smmu->features & ARM_SMMU_FEAT_PRI)
irqen_flags |= IRQ_CTRL_PRIQ_IRQEN;
/* Enable interrupt generation on the SMMU */
ret = arm_smmu_write_reg_sync(smmu, irqen_flags,
ARM_SMMU_IRQ_CTRL, ARM_SMMU_IRQ_CTRLACK);
if (ret)
dev_warn(smmu->dev, "failed to enable irqs\n");
return 0;
}
static int arm_smmu_device_disable(struct arm_smmu_device *smmu)
{
int ret;
ret = arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_CR0, ARM_SMMU_CR0ACK);
if (ret)
dev_err(smmu->dev, "failed to clear cr0\n");
return ret;
}
static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool resume)
{
int i;
int ret;
u32 reg, enables;
struct arm_smmu_cmdq_ent cmd;
/* Clear CR0 and sync (disables SMMU and queue processing) */
reg = readl_relaxed(smmu->base + ARM_SMMU_CR0);
if (reg & CR0_SMMUEN) {
dev_warn(smmu->dev, "SMMU currently enabled! Resetting...\n");
WARN_ON(is_kdump_kernel() && !disable_bypass);
arm_smmu_update_gbpa(smmu, GBPA_ABORT, 0);
}
ret = arm_smmu_device_disable(smmu);
if (ret)
return ret;
/* CR1 (table and queue memory attributes) */
reg = FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) |
FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) |
FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) |
FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) |
FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) |
FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB);
writel_relaxed(reg, smmu->base + ARM_SMMU_CR1);
/* CR2 (random crap) */
reg = CR2_RECINVSID;
if (smmu->features & ARM_SMMU_FEAT_E2H)
reg |= CR2_E2H;
if (!(smmu->features & ARM_SMMU_FEAT_BTM))
reg |= CR2_PTM;
writel_relaxed(reg, smmu->base + ARM_SMMU_CR2);
/* Stream table */
writeq_relaxed(smmu->strtab_cfg.strtab_base,
smmu->base + ARM_SMMU_STRTAB_BASE);
writel_relaxed(smmu->strtab_cfg.strtab_base_cfg,
smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
/* Command queue */
writeq_relaxed(smmu->cmdq.q.q_base, smmu->base + ARM_SMMU_CMDQ_BASE);
writel_relaxed(smmu->cmdq.q.llq.prod, smmu->base + ARM_SMMU_CMDQ_PROD);
writel_relaxed(smmu->cmdq.q.llq.cons, smmu->base + ARM_SMMU_CMDQ_CONS);
for (i = 0; i < smmu->nr_ecmdq; i++) {
struct arm_smmu_ecmdq *ecmdq;
struct arm_smmu_queue *q;
ecmdq = *per_cpu_ptr(smmu->ecmdq, i);
q = &ecmdq->cmdq.q;
if (WARN_ON(q->llq.prod != q->llq.cons)) {
q->llq.prod = 0;
q->llq.cons = 0;
}
writeq_relaxed(q->q_base, ecmdq->base + ARM_SMMU_ECMDQ_BASE);
writel_relaxed(q->llq.prod, ecmdq->base + ARM_SMMU_ECMDQ_PROD);
writel_relaxed(q->llq.cons, ecmdq->base + ARM_SMMU_ECMDQ_CONS);
/* enable ecmdq */
writel(ECMDQ_PROD_EN | q->llq.prod, q->prod_reg);
ret = readl_relaxed_poll_timeout(q->cons_reg, reg, reg & ECMDQ_CONS_ENACK,
1, ARM_SMMU_POLL_TIMEOUT_US);
if (ret) {
dev_err(smmu->dev, "ecmdq[%d] enable failed\n", i);
smmu->ecmdq_enabled = 0;
break;
}
}
enables = CR0_CMDQEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
ARM_SMMU_CR0ACK);
if (ret) {
dev_err(smmu->dev, "failed to enable command queue\n");
return ret;
}
/* Invalidate any cached configuration */
cmd.opcode = CMDQ_OP_CFGI_ALL;
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
/* Invalidate any stale TLB entries */
if (smmu->features & ARM_SMMU_FEAT_HYP) {
cmd.opcode = CMDQ_OP_TLBI_EL2_ALL;
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
}
cmd.opcode = CMDQ_OP_TLBI_NSNH_ALL;
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
/* Event queue */
writeq_relaxed(smmu->evtq.q.q_base, smmu->base + ARM_SMMU_EVTQ_BASE);
writel_relaxed(smmu->evtq.q.llq.prod, smmu->page1 + ARM_SMMU_EVTQ_PROD);
writel_relaxed(smmu->evtq.q.llq.cons, smmu->page1 + ARM_SMMU_EVTQ_CONS);
enables |= CR0_EVTQEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
ARM_SMMU_CR0ACK);
if (ret) {
dev_err(smmu->dev, "failed to enable event queue\n");
return ret;
}
/* PRI queue */
if (smmu->features & ARM_SMMU_FEAT_PRI) {
writeq_relaxed(smmu->priq.q.q_base,
smmu->base + ARM_SMMU_PRIQ_BASE);
writel_relaxed(smmu->priq.q.llq.prod,
smmu->page1 + ARM_SMMU_PRIQ_PROD);
writel_relaxed(smmu->priq.q.llq.cons,
smmu->page1 + ARM_SMMU_PRIQ_CONS);
enables |= CR0_PRIQEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
ARM_SMMU_CR0ACK);
if (ret) {
dev_err(smmu->dev, "failed to enable PRI queue\n");
return ret;
}
}
if (smmu->features & ARM_SMMU_FEAT_ATS) {
enables |= CR0_ATSCHK;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
ARM_SMMU_CR0ACK);
if (ret) {
dev_err(smmu->dev, "failed to enable ATS check\n");
return ret;
}
}
ret = arm_smmu_setup_irqs(smmu, resume);
if (ret) {
dev_err(smmu->dev, "failed to setup irqs\n");
return ret;
}
if (is_kdump_kernel())
enables &= ~(CR0_EVTQEN | CR0_PRIQEN);
/* Enable the SMMU interface, or ensure bypass */
if (!smmu->bypass || disable_bypass) {
enables |= CR0_SMMUEN;
} else {
ret = arm_smmu_update_gbpa(smmu, 0, GBPA_ABORT);
if (ret)
return ret;
}
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
ARM_SMMU_CR0ACK);
if (ret) {
dev_err(smmu->dev, "failed to enable SMMU interface\n");
return ret;
}
return 0;
}
Reset the SMMUv3 device to complete the enabling of the hardware SMMUv3 device. The process is roughly as follows:
- CheckSMMU_CR0 register, if SMMU is enabled, updateSMMU_GBPA< a i=4> register, stops all incoming transactions.
- Disable all functions of the SMMU device, including command queue, event queue, PRI queue, etc. Here you can see a unique register writing mode of SMMU, SMMU_CR0< The /span>SMMU_CR0 register to confirm that the write is effective , there are several other SMMU registers with similar writing patterns. SMMU_CR0ACK register, it will wait for the update of the corresponding bit of the SMMU_CR0 The corresponding bit of the register will be updated, which is written hereSMMU_CR0ACK When the value written in the register is confirmed to be effective, , when directed to SMMU_CR0ACK register has a corresponding acknowledgment register
- WriteSMMU_CR1 register, configuration table and queue memory attributes, flow table, command queue, event queue and PRI queue, etc. Cacheability, shareability.
- TransferSMMU_CR2 Parasitic organ, arrangement RECINVSID, < /span>. BTM Sum E2H
- Configure the flow table base address created previously in the initialization data structure, and write the flow table base address value into the corresponding register.
- Write the command queue base address value, command queue producer pointer, command queue consumer pointer, and extended command queue related configurations created in the initialization data structure to the corresponding registers, and write. Enable command queue registerSMMU_CR0
- Send several commands to the command queue, invalidate any cache configuration, stale TLB entries, etc.
- Write the event queue base address value, event queue producer pointer, and event queue consumer pointer created previously in the initialization data structure to the corresponding registers, and writeSMMU_CR0 RegisterEnable event queue.
- When the SMMUv3 hardware device supports PRI, write the PRI queue base address value, PRI queue producer pointer, and PRI queue consumer pointer created previously in the initialization data structure into the corresponding registers and write. Enable PRI queue registerSMMU_CR0
- If the SMMUv3 hardware device supports ATS checking, writeSMMU_CR0 registerenable ATS checking a>.
- Configuration interrupted.
- If the bypass SMMU is not configured or the SMMU is disabled, write the SMMU_CR0 register to enable the SMMU.
The process of setting up interrupts is as follows:
- Copying SMMU_IRQ_CTRL Parasitic organ prohibition interrupted.
- If the use of joint interrupts is configured, apply for interrupt resources from the system and register the interrupt handler.
- No combined interrupts are used:
- Configure MSI;
- Request an interrupt line for the event queue and register an interrupt handler;
- Request an interrupt line for global errors and register an interrupt handler;
- When the SMMUv3 hardware device supports PRI, request an interrupt line for the PRI queue and register an interrupt handler.
- Copying SMMU_IRQ_CTRL Interruption of parasitic equipment.
arm_smmu_device_reset()The function resets the SMMUv3 device, centrally sets various registers of the SMMUv3 device, and registers related to flow tables, queues, and interrupts, and finally enables the SMMU hardware device.
Register SMMUv3 devices with the IOMMU subsystem
arm_smmu_device_probe() function calliommu_device_register() function registers the SMMUv3 device into the IOMMU subsystem, iommu_device_register() function definition (located in drivers/ iommu/iommu.c file) as follows:
static LIST_HEAD(iommu_device_list);
static DEFINE_SPINLOCK(iommu_device_lock);
. . . . . .
int iommu_device_register(struct iommu_device *iommu)
{
spin_lock(&iommu_device_lock);
list_add_tail(&iommu->list, &iommu_device_list);
spin_unlock(&iommu_device_lock);
return 0;
}
EXPORT_SYMBOL_GPL(iommu_device_register);
The IOMMU subsystem uses a linked list to maintain the IOMMU devices in the system. Registering the SMMUv3 device into the IOMMU subsystem means placing the struct iommu_device object of the SMMUv3 device into the IOMMU device of the IOMMU subsystem. in the linked list.
Set IOMMU callbacks for each bus type
arm_smmu_device_probe() function callarm_smmu_set_bus_ops() function sets the IOMMU callback for each supported bus type, arm_smmu_set_bus_ops() function definition (located in drivers /iommu/arm/arm-smmu-v3/arm-smmu-v3.c file) as follows:
static int arm_smmu_set_bus_ops(struct iommu_ops *ops)
{
int err;
#ifdef CONFIG_PCI
if (pci_bus_type.iommu_ops != ops) {
err = bus_set_iommu(&pci_bus_type, ops);
if (err)
return err;
}
#endif
#ifdef CONFIG_ARM_AMBA
if (amba_bustype.iommu_ops != ops) {
err = bus_set_iommu(&amba_bustype, ops);
if (err)
goto err_reset_pci_ops;
}
#endif
if (platform_bus_type.iommu_ops != ops) {
err = bus_set_iommu(&platform_bus_type, ops);
if (err)
goto err_reset_amba_ops;
}
return 0;
err_reset_amba_ops:
#ifdef CONFIG_ARM_AMBA
bus_set_iommu(&amba_bustype, NULL);
#endif
err_reset_pci_ops: __maybe_unused;
#ifdef CONFIG_PCI
bus_set_iommu(&pci_bus_type, NULL);
#endif
return err;
}
arm_smmu_set_bus_ops() function callbus_set_iommu() function sets IOMMU for platform_bus_type, pci_bus_type, and amba_bustype etc. callback. bus_set_iommu() The function definition (located in the drivers/iommu/iommu.c file) is as follows:
static int probe_get_default_domain_type(struct device *dev, void *data)
{
const struct iommu_ops *ops = dev->bus->iommu_ops;
struct __group_domain_type *gtype = data;
unsigned int type = 0;
if (ops->def_domain_type)
type = ops->def_domain_type(dev);
if (type) {
if (gtype->type && gtype->type != type) {
dev_warn(dev, "Device needs domain type %s, but device %s in the same iommu group requires type %s - using default\n",
iommu_domain_type_str(type),
dev_name(gtype->dev),
iommu_domain_type_str(gtype->type));
gtype->type = 0;
}
if (!gtype->dev) {
gtype->dev = dev;
gtype->type = type;
}
}
return 0;
}
static void probe_alloc_default_domain(struct bus_type *bus,
struct iommu_group *group)
{
struct __group_domain_type gtype;
memset(>ype, 0, sizeof(gtype));
/* Ask for default domain requirements of all devices in the group */
__iommu_group_for_each_dev(group, >ype,
probe_get_default_domain_type);
if (!gtype.type)
gtype.type = iommu_def_domain_type;
iommu_group_alloc_default_domain(bus, group, gtype.type);
}
static int iommu_group_do_dma_attach(struct device *dev, void *data)
{
struct iommu_domain *domain = data;
int ret = 0;
if (!iommu_is_attach_deferred(domain, dev))
ret = __iommu_attach_device(domain, dev);
return ret;
}
static int __iommu_group_dma_attach(struct iommu_group *group)
{
return __iommu_group_for_each_dev(group, group->default_domain,
iommu_group_do_dma_attach);
}
static int iommu_group_do_probe_finalize(struct device *dev, void *data)
{
struct iommu_domain *domain = data;
if (domain->ops->probe_finalize)
domain->ops->probe_finalize(dev);
return 0;
}
static void __iommu_group_dma_finalize(struct iommu_group *group)
{
__iommu_group_for_each_dev(group, group->default_domain,
iommu_group_do_probe_finalize);
}
static void __iommu_group_dma_finalize(struct iommu_group *group)
{
__iommu_group_for_each_dev(group, group->default_domain,
iommu_group_do_probe_finalize);
}
static int iommu_group_create_direct_mappings(struct iommu_group *group)
{
return __iommu_group_for_each_dev(group, group,
iommu_do_create_direct_mappings);
}
static int probe_iommu_group(struct device *dev, void *data)
{
struct list_head *group_list = data;
struct iommu_group *group;
int ret;
/* Device is probed already if in a group */
group = iommu_group_get(dev);
if (group) {
iommu_group_put(group);
return 0;
}
ret = __iommu_probe_device(dev, group_list);
if (ret == -ENODEV)
ret = 0;
return ret;
}
static int remove_iommu_group(struct device *dev, void *data)
{
iommu_release_device(dev);
return 0;
}
static int iommu_bus_notifier(struct notifier_block *nb,
unsigned long action, void *data)
{
unsigned long group_action = 0;
struct device *dev = data;
struct iommu_group *group;
/*
* ADD/DEL call into iommu driver ops if provided, which may
* result in ADD/DEL notifiers to group->notifier
*/
if (action == BUS_NOTIFY_ADD_DEVICE) {
int ret;
ret = iommu_probe_device(dev);
return (ret) ? NOTIFY_DONE : NOTIFY_OK;
} else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
iommu_release_device(dev);
return NOTIFY_OK;
}
/*
* Remaining BUS_NOTIFYs get filtered and republished to the
* group, if anyone is listening
*/
group = iommu_group_get(dev);
if (!group)
return 0;
switch (action) {
case BUS_NOTIFY_BIND_DRIVER:
group_action = IOMMU_GROUP_NOTIFY_BIND_DRIVER;
break;
case BUS_NOTIFY_BOUND_DRIVER:
group_action = IOMMU_GROUP_NOTIFY_BOUND_DRIVER;
break;
case BUS_NOTIFY_UNBIND_DRIVER:
group_action = IOMMU_GROUP_NOTIFY_UNBIND_DRIVER;
break;
case BUS_NOTIFY_UNBOUND_DRIVER:
group_action = IOMMU_GROUP_NOTIFY_UNBOUND_DRIVER;
break;
}
if (group_action)
blocking_notifier_call_chain(&group->notifier,
group_action, dev);
iommu_group_put(group);
return 0;
}
. . . . . .
int bus_iommu_probe(struct bus_type *bus)
{
struct iommu_group *group, *next;
LIST_HEAD(group_list);
int ret;
/*
* This code-path does not allocate the default domain when
* creating the iommu group, so do it after the groups are
* created.
*/
ret = bus_for_each_dev(bus, NULL, &group_list, probe_iommu_group);
if (ret)
return ret;
list_for_each_entry_safe(group, next, &group_list, entry) {
/* Remove item from the list */
list_del_init(&group->entry);
mutex_lock(&group->mutex);
/* Try to allocate default domain */
probe_alloc_default_domain(bus, group);
if (!group->default_domain) {
mutex_unlock(&group->mutex);
continue;
}
iommu_group_create_direct_mappings(group);
ret = __iommu_group_dma_attach(group);
mutex_unlock(&group->mutex);
if (ret)
break;
__iommu_group_dma_finalize(group);
}
return ret;
}
static int iommu_bus_init(struct bus_type *bus, const struct iommu_ops *ops)
{
struct notifier_block *nb;
int err;
nb = kzalloc(sizeof(struct notifier_block), GFP_KERNEL);
if (!nb)
return -ENOMEM;
nb->notifier_call = iommu_bus_notifier;
err = bus_register_notifier(bus, nb);
if (err)
goto out_free;
err = bus_iommu_probe(bus);
if (err)
goto out_err;
return 0;
out_err:
/* Clean up */
bus_for_each_dev(bus, NULL, NULL, remove_iommu_group);
bus_unregister_notifier(bus, nb);
out_free:
kfree(nb);
return err;
}
/**
* bus_set_iommu - set iommu-callbacks for the bus
* @bus: bus.
* @ops: the callbacks provided by the iommu-driver
*
* This function is called by an iommu driver to set the iommu methods
* used for a particular bus. Drivers for devices on that bus can use
* the iommu-api after these ops are registered.
* This special function is needed because IOMMUs are usually devices on
* the bus itself, so the iommu drivers are not initialized when the bus
* is set up. With this function the iommu-driver can set the iommu-ops
* afterwards.
*/
int bus_set_iommu(struct bus_type *bus, const struct iommu_ops *ops)
{
int err;
if (ops == NULL) {
bus->iommu_ops = NULL;
return 0;
}
if (bus->iommu_ops != NULL)
return -EBUSY;
bus->iommu_ops = ops;
/* Do IOMMU specific setup for this bus-type */
err = iommu_bus_init(bus, ops);
if (err)
bus->iommu_ops = NULL;
return err;
}
EXPORT_SYMBOL_GPL(bus_set_iommu);
bus_set_iommu()The function execution process is as follows:
- Set the IOMMU callback for the bus.
- Perform bus type IOMMU-specific settings:
- Register a callback with the bus to be notified when a new device is added;
- Detect devices that have been added to the bus.
The process for detecting devices that have been added to the bus is as follows:
- Traverse all the devices that have been added on the bus and detect each device. If the device is connected to a registered IOMMU device, obtain the device's
iommu_group. Theseiommu_groupare put into a linked list. - traverses the obtained
iommu_group, for eachiommu_group:- General
iommu_groupExport table; - Try to allocate the default domain, first confirm the default domain type, and then allocate the domain object. When confirming the domain type, first try to obtain it through the callback registered by the SMMU device
def_domain_type. If it fails, then Take the default domain type of the IOMMO subsystem; - Create direct mappings for individual devices;
- Perform attach for each device;
- Execute probe end callback for each device.
- General
Set IOMMU callbacks for each bus type, handling the situation where system I/O devices connected to the IOMMU are detected before the IOMMU device.
Reference documentation
Current status and development of IOMMU
Introduction to IOMMU and Arm SMMU
IOMMU/SMMUV3 code analysis (5) Association between IO devices and SMMU 2