flameiq.providers — Plugin System

Base class

FlameIQ metric provider base class.

Providers are the plugin layer that translates raw benchmark output from any tool or format into a PerformanceSnapshot.

All providers must be:

Stateless — no mutable state between calls.
Deterministic — same input always produces the same snapshot.
Side-effect free — read-only access to the source file.

Implementing a custom provider

Subclass MetricProvider, implement the three abstract methods, and register in flameiq.providers.registry:

from flameiq.providers.base import MetricProvider
from flameiq.providers.registry import PROVIDER_REGISTRY

class MyProvider(MetricProvider):

    @property
    def name(self) -> str:
        return "my-tool"

    def collect(self, source: str) -> dict:
        ...

    def validate(self, raw: dict) -> bool:
        ...

    def normalize(self, raw: dict) -> PerformanceSnapshot:
        ...

PROVIDER_REGISTRY["my-tool"] = MyProvider

Then use it:

flameiq run --metrics output.json --provider my-tool

class flameiq.providers.base.MetricProvider[source]

Bases: ABC

Abstract base class for all FlameIQ metric providers.

abstract property name: str: Unique provider identifier, e.g. 'pytest-benchmark'.

abstractmethod collect(source)[source]

Read raw benchmark data from a source path.

Parameters:

source (str) – File path containing benchmark output.

Returns:

Raw data as a Python dict.

Raises:

MetricsFileNotFoundError – If the file does not exist.
ProviderError – On any read or parse failure.

Return type:

dict[str, Any]

abstractmethod validate(raw)[source]

Check whether raw can be processed by this provider.

Parameters:: raw (dict[str, Any]) – The dict returned by collect().
Returns:: True if the data is valid for this provider.
Return type:: bool

abstractmethod normalize(raw)[source]

Transform validated raw data into a PerformanceSnapshot.

Full type: PerformanceSnapshot.

Parameters:: raw (dict[str, Any]) – Validated dict from collect().
Returns:: A valid PerformanceSnapshot.
Raises:: ProviderError – On normalisation failure.
Return type:: PerformanceSnapshot

load(source)[source]

Convenience pipeline: collect() → validate() → normalize().

Parameters:: source (str) – Path to the metrics file.
Returns:: A validated, normalised snapshot.
Raises:: ProviderError – If validation fails.
Return type:: PerformanceSnapshot

__abstractmethods__ = frozenset({'collect', 'name', 'normalize', 'validate'})

__dict__ = mappingproxy({'__module__': 'flameiq.providers.base', '__doc__': 'Abstract base class for all FlameIQ metric providers.', 'name': <property object>, 'collect': <function MetricProvider.collect>, 'validate': <function MetricProvider.validate>, 'normalize': <function MetricProvider.normalize>, 'load': <function MetricProvider.load>, '__dict__': <attribute '__dict__' of 'MetricProvider' objects>, '__weakref__': <attribute '__weakref__' of 'MetricProvider' objects>, '__abstractmethods__': frozenset({'normalize', 'validate', 'collect', 'name'}), '_abc_impl': <_abc._abc_data object>, '__annotations__': {}})

__module__ = 'flameiq.providers.base'

__weakref__: list of weak references to the object

JSON Provider

FlameIQ JSON provider.

The default built-in provider. Accepts any JSON file that already conforms to the FlameIQ v1 schema. This is the simplest integration path.

Usage:

flameiq run --metrics output.json
flameiq run --metrics output.json --provider json  # explicit

class flameiq.providers.json_provider.JsonProvider[source]

Bases: MetricProvider

Provider for files already in FlameIQ v1 JSON schema format.

property name: str: Return the unique provider name.

collect(source)[source]

Collect raw metrics data from a JSON file.

Parameters:: source (str)
Return type:: dict[str, Any]

validate(raw)[source]

Check if raw data conforms to the FlameIQ v1 schema.

Parameters:: raw (dict[str, Any])
Return type:: bool

normalize(raw)[source]

Parse raw data into a PerformanceSnapshot.

Parameters:: raw (dict[str, Any])
Return type:: PerformanceSnapshot

pytest-benchmark Provider

FlameIQ pytest-benchmark provider.

Adapts output from pytest-benchmark into a FlameIQ v1 PerformanceSnapshot.

Generate compatible output with:

pytest --benchmark-json=benchmark_output.json

Then run:

flameiq run --metrics benchmark_output.json --provider pytest-benchmark

Mapping

pytest-benchmark field	FlameIQ field
`stats.mean` (s)	`latency.mean` (ms)
`stats.median` (s)	`latency.p50` (ms)
`stats.ops`	`throughput` (ops/s)
`commit_info.id`	`metadata.commit`
`commit_info.branch`	`metadata.branch`

When multiple benchmarks are present in a single JSON file, each benchmark is stored as a separate custom.<benchmark_name> metric (mean latency in ms), and the first benchmark’s latency percentiles are used for the primary signals.

class flameiq.providers.pytest_provider.PytestBenchmarkProvider[source]

Bases: MetricProvider

Provider for pytest-benchmark JSON output files.

property name: str: Return the unique provider name.

collect(source)[source]

Collect raw metrics data from a pytest-benchmark JSON file.

Parameters:: source (str)
Return type:: dict[str, Any]

validate(raw)[source]

Check if raw data has the expected pytest-benchmark structure.

Parameters:: raw (dict[str, Any])
Return type:: bool

normalize(raw)[source]

Parse raw pytest-benchmark data into a PerformanceSnapshot.

Full type: PerformanceSnapshot.

Parameters:: raw (dict[str, Any])
Return type:: PerformanceSnapshot

Registry

FlameIQ provider registry.

Maps provider name strings to provider classes. Add new providers here to make them available via --provider <name>.

flameiq.providers.registry.PROVIDER_REGISTRY: dict[str, type[MetricProvider]] = {'json': <class 'flameiq.providers.json_provider.JsonProvider'>, 'pytest-benchmark': <class 'flameiq.providers.pytest_provider.PytestBenchmarkProvider'>}: Mutable registry — add custom providers here.

flameiq.providers.registry.get_provider(name)[source]

Instantiate a registered provider by name.

Parameters:: name (str) – Provider identifier, e.g. 'json' or 'pytest-benchmark'.
Returns:: An instantiated MetricProvider.
Raises:: ProviderNotFoundError – If name is not in PROVIDER_REGISTRY.
Return type:: MetricProvider

flameiq.providers.registry.list_providers()[source]

Return all registered provider names, sorted alphabetically.

Return type:: list[str]