PCIe Driver Development¶

This guide covers practical techniques for developing PCIe drivers across all major operating systems. You'll learn to build drivers that unlock the full performance potential of PCIe hardware while maintaining stability, security, and compliance with OS vendor requirements.

Linux PCI Driver Development¶

PCI Subsystem Overview¶

The Linux PCI subsystem reflects actual hardware configuration and interconnections. Every bus and device is assigned a unique number following this pattern:

<bus id>:<device id>:<function id>

Every PCI device contains factory-programmed Vendor ID (VID) and Device ID (PID). The Linux kernel uses these IDs to identify devices and load the appropriate driver.

Viewing PCI Devices¶

Use lspci to explore PCI devices:

# List all PCI devices with IDs
lspci -nn

# Detailed view of specific device
lspci -v -s 00:1f.0

# View kernel driver in use
lspci -k

Device information is also available via sysfs:

ls -la /sys/bus/pci/devices/

PCI Configuration Space¶

Every PCI device implements a standard configuration register set. The first 64 bytes are mandatory:

Register	Offset	Size	Description
Vendor ID	0x00	2 bytes	Hardware vendor identifier
Device ID	0x02	2 bytes	Device model identifier
Command	0x04	2 bytes	Device control flags
Status	0x06	2 bytes	Device status flags
Revision ID	0x08	1 byte	Device revision
Class Code	0x09	3 bytes	Device class/subclass
BAR0-BAR5	0x10-0x24	4 bytes each	Base Address Registers
Subsystem Vendor ID	0x2C	2 bytes	Subsystem vendor
Subsystem ID	0x2E	2 bytes	Subsystem identifier

Byte Order

PCI configuration space is always little-endian. This is important when working on big-endian systems.

Linux PCI Driver Structure¶

Key Header Files¶

#include <linux/init.h>
#include <linux/module.h>
#include <linux/pci.h>

Device ID Table¶

Define supported Vendor/Product ID pairs:

static struct pci_device_id my_driver_id_table[] = {
    { PCI_DEVICE(0x010F, 0x0F0E) },  /* VID=0x010F, PID=0x0F0E */
    { PCI_DEVICE(0x010F, 0x0F0F) },  /* Additional device */
    {0,}  /* Terminating entry */
};

MODULE_DEVICE_TABLE(pci, my_driver_id_table);

The MODULE_DEVICE_TABLE macro:

Integrates device IDs into the global device table (built-in drivers)
Enables depmod to extract IDs for automatic module loading (loadable modules)

Driver Registration Structure¶

static struct pci_driver my_driver = {
    .name = "my_pci_driver",
    .id_table = my_driver_id_table,
    .probe = my_driver_probe,
    .remove = my_driver_remove,
    .suspend = my_driver_suspend,  /* Optional: power management */
    .resume = my_driver_resume,    /* Optional: power management */
};

Field	Description
`.name`	Unique driver name (appears in `/sys/bus/pci/drivers`)
`.id_table`	Supported device ID array
`.probe`	Called when matching device is found
`.remove`	Called when driver is unloaded
`.suspend`	Called on system suspend
`.resume`	Called on system resume

Module Init/Exit¶

static int __init my_driver_init(void)
{
    return pci_register_driver(&my_driver);
}

static void __exit my_driver_exit(void)
{
    pci_unregister_driver(&my_driver);
}

module_init(my_driver_init);
module_exit(my_driver_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("Sample PCI Driver");

Device Access¶

Reading Configuration Registers¶

#include <linux/pci.h>

u16 vendor, device;
u8 revision;

/* Read 16-bit values */
pci_read_config_word(pdev, PCI_VENDOR_ID, &vendor);
pci_read_config_word(pdev, PCI_DEVICE_ID, &device);

/* Read 8-bit value */
pci_read_config_byte(pdev, PCI_REVISION_ID, &revision);

printk(KERN_INFO "Device: VID=0x%X, PID=0x%X, Rev=%d\n",
       vendor, device, revision);

Available functions:

int pci_read_config_byte(const struct pci_dev *dev, int where, u8 *val);
int pci_read_config_word(const struct pci_dev *dev, int where, u16 *val);
int pci_read_config_dword(const struct pci_dev *dev, int where, u32 *val);
int pci_write_config_byte(const struct pci_dev *dev, int where, u8 val);
int pci_write_config_word(const struct pci_dev *dev, int where, u16 val);
int pci_write_config_dword(const struct pci_dev *dev, int where, u32 val);

Register offsets are defined in linux/pci_regs.h.

Memory-Mapped I/O (BAR Access)¶

Device memory is accessed through Base Address Registers (BARs):

int bar;
unsigned long mmio_start, mmio_len;
u8 __iomem *hwmem;

/* Select memory BAR */
bar = pci_select_bars(pdev, IORESOURCE_MEM);

/* Enable device memory */
pci_enable_device_mem(pdev);

/* Request the memory region */
pci_request_region(pdev, bar, "my_driver");

/* Get memory start and length */
mmio_start = pci_resource_start(pdev, 0);
mmio_len = pci_resource_len(pdev, 0);

/* Map to kernel virtual address */
hwmem = ioremap(mmio_start, mmio_len);

I/O Read/Write Functions¶

Use these kernel functions for memory-mapped I/O:

/* Write operations */
void iowrite8(u8 val, void __iomem *addr);
void iowrite16(u16 val, void __iomem *addr);
void iowrite32(u32 val, void __iomem *addr);

/* Read operations */
unsigned int ioread8(void __iomem *addr);
unsigned int ioread16(void __iomem *addr);
unsigned int ioread32(void __iomem *addr);

Example usage:

/* Write 32-bit value to device */
iowrite32(0xDEADBEEF, hwmem + DEVICE_CTRL_REG);

/* Read 32-bit value from device */
u32 status = ioread32(hwmem + DEVICE_STATUS_REG);

Portability

Always use ioread*/iowrite* functions instead of direct pointer access for better portability across architectures.

DMA (Direct Memory Access)¶

Enabling Bus Mastering¶

High-performance devices use DMA for direct memory transfers:

/* Enable bus mastering */
pci_set_master(pdev);

/* Disable bus mastering (cleanup) */
pci_clear_master(pdev);

DMA Memory Allocation¶

#include <linux/dma-mapping.h>

void *cpu_addr;
dma_addr_t dma_handle;

/* Allocate coherent DMA memory */
cpu_addr = dma_alloc_coherent(&pdev->dev, size, &dma_handle, GFP_KERNEL);

/* Free DMA memory */
dma_free_coherent(&pdev->dev, size, cpu_addr, dma_handle);

Interrupt Handling¶

Interrupt Types¶

Type	Description
INTx (Legacy)	Pin-based, can be shared between devices
MSI	Message Signaled Interrupts, dedicated per device
MSI-X	Extended MSI, supports multiple interrupt vectors

Best Practice

Use MSI/MSI-X when possible. They avoid sharing issues and provide better performance since interrupt delivery is guaranteed after DMA completion.

MSI/MSI-X Setup¶

/* Allocate MSI vectors */
int ret = pci_alloc_irq_vectors(pdev, min_vecs, max_vecs,
                                 PCI_IRQ_MSI | PCI_IRQ_MSIX);
if (ret < 0) {
    /* Fall back to legacy interrupts or handle error */
}

/* Get the IRQ number for a vector */
int irq = pci_irq_vector(pdev, vector_num);

/* Request the interrupt */
ret = request_threaded_irq(irq, irq_handler, irq_thread_fn,
                           0, "my_driver", pdev);

/* Free on cleanup */
free_irq(irq, pdev);
pci_free_irq_vectors(pdev);

Interrupt Handler¶

static irqreturn_t my_irq_handler(int irq, void *dev_id)
{
    struct pci_dev *pdev = dev_id;
    struct my_driver_priv *priv = pci_get_drvdata(pdev);

    /* Check if this interrupt is from our device */
    u32 status = ioread32(priv->hwmem + IRQ_STATUS_REG);
    if (!(status & IRQ_PENDING)) {
        return IRQ_NONE;  /* Not our interrupt */
    }

    /* Acknowledge interrupt */
    iowrite32(status, priv->hwmem + IRQ_ACK_REG);

    /* Handle the interrupt */
    /* ... */

    return IRQ_HANDLED;
}

Complete Driver Example¶

/* Sample Linux PCI Device Driver */

#include <linux/init.h>
#include <linux/module.h>
#include <linux/pci.h>

#define MY_DRIVER "my_pci_driver"

/* Supported devices */
static struct pci_device_id my_driver_id_table[] = {
    { PCI_DEVICE(0x010F, 0x0F0E) },
    {0,}
};

MODULE_DEVICE_TABLE(pci, my_driver_id_table);

/* Private driver data */
struct my_driver_priv {
    u8 __iomem *hwmem;
};

/* Forward declarations */
static int my_driver_probe(struct pci_dev *pdev,
                           const struct pci_device_id *ent);
static void my_driver_remove(struct pci_dev *pdev);

/* Driver registration */
static struct pci_driver my_driver = {
    .name = MY_DRIVER,
    .id_table = my_driver_id_table,
    .probe = my_driver_probe,
    .remove = my_driver_remove
};

/* Probe function - called when device is found */
static int my_driver_probe(struct pci_dev *pdev,
                           const struct pci_device_id *ent)
{
    int bar, err;
    unsigned long mmio_start, mmio_len;
    struct my_driver_priv *drv_priv;

    /* Enable device memory */
    err = pci_enable_device_mem(pdev);
    if (err)
        return err;

    /* Request BAR */
    bar = pci_select_bars(pdev, IORESOURCE_MEM);
    err = pci_request_region(pdev, bar, MY_DRIVER);
    if (err) {
        pci_disable_device(pdev);
        return err;
    }

    /* Get memory info */
    mmio_start = pci_resource_start(pdev, 0);
    mmio_len = pci_resource_len(pdev, 0);

    /* Allocate private data */
    drv_priv = kzalloc(sizeof(*drv_priv), GFP_KERNEL);
    if (!drv_priv) {
        pci_release_region(pdev, bar);
        pci_disable_device(pdev);
        return -ENOMEM;
    }

    /* Map device memory */
    drv_priv->hwmem = ioremap(mmio_start, mmio_len);
    if (!drv_priv->hwmem) {
        kfree(drv_priv);
        pci_release_region(pdev, bar);
        pci_disable_device(pdev);
        return -EIO;
    }

    /* Store private data */
    pci_set_drvdata(pdev, drv_priv);

    /* Enable bus mastering for DMA */
    pci_set_master(pdev);

    printk(KERN_INFO "%s: Device initialized\n", MY_DRIVER);
    return 0;
}

/* Remove function - cleanup */
static void my_driver_remove(struct pci_dev *pdev)
{
    struct my_driver_priv *drv_priv = pci_get_drvdata(pdev);

    if (drv_priv) {
        if (drv_priv->hwmem)
            iounmap(drv_priv->hwmem);
        kfree(drv_priv);
    }

    pci_clear_master(pdev);
    pci_release_region(pdev, pci_select_bars(pdev, IORESOURCE_MEM));
    pci_disable_device(pdev);

    printk(KERN_INFO "%s: Device removed\n", MY_DRIVER);
}

/* Module init/exit */
static int __init my_driver_init(void)
{
    return pci_register_driver(&my_driver);
}

static void __exit my_driver_exit(void)
{
    pci_unregister_driver(&my_driver);
}

module_init(my_driver_init);
module_exit(my_driver_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("HSIO Lab");
MODULE_DESCRIPTION("Sample PCI Driver");
MODULE_VERSION("1.0");

Makefile¶

BINARY     := my_pci_driver
KERNEL     := /lib/modules/$(shell uname -r)/build
ARCH       := x86
KMOD_DIR   := $(shell pwd)

obj-m += $(BINARY).o

all:
    make -C $(KERNEL) M=$(KMOD_DIR) modules

clean:
    make -C $(KERNEL) M=$(KMOD_DIR) clean

Building and Loading¶

# Build the module
make

# Load the module
sudo insmod my_pci_driver.ko

# Check kernel messages
dmesg | tail

# Verify driver is loaded
lsmod | grep my_pci

# Unload the module
sudo rmmod my_pci_driver

Windows Driver Development¶

KMDF vs UMDF¶

Framework	Use Case	Performance
KMDF	High-performance, DMA, interrupts	Highest
UMDF	Security-sensitive, simpler devices	Good

Key Concepts¶

WHQL Certification - Required for Windows logo compliance
INF Files - Installation configuration
Digital Signing - Required for driver loading
HLK Tests - Hardware Lab Kit for certification testing

Embedded and RTOS Drivers¶

VxWorks¶

PCI library functions for device access
VxBus driver model for modern VxWorks versions

QNX Neutrino¶

Resource manager model
PCI server for device enumeration

Bare-Metal¶

Direct PCI configuration space access
No OS overhead for real-time requirements

Best Practices¶

Practice	Description
Error Handling	Always check return values and clean up on failure
Resource Management	Release all resources in reverse order of acquisition
Locking	Use appropriate synchronization for shared resources
Power Management	Implement suspend/resume for laptop compatibility
Documentation	Document hardware registers and driver interfaces

PCIe Architecture - Protocol fundamentals
PCIe Software Development - User-space tools
PCIe High-Speed Validation - Signal integrity testing

References¶

Linux Kernel Documentation: Documentation/PCI/
PCI-SIG Specifications
Linux Device Drivers, 3^rd Edition (LDD3)