FlameIQ

Getting Started

  • Installation
    • Requirements
    • Install from PyPI
    • Install in a virtual environment (recommended)
    • Install with optional dependencies
    • Install from source
    • Offline / air-gap installation
    • Upgrading
    • Uninstalling
  • Quick Start
    • Step 1 — Initialise FlameIQ
    • Step 2 — Prepare benchmark output
    • Step 3 — Set a baseline
    • Step 4 — Compare on every pull request
    • Step 5 — Generate an HTML report
    • Step 6 — Advance the baseline
    • That’s it
  • CI Integration
    • GitHub Actions
      • Minimal example
      • Full matrix example (multiple environments)
    • GitLab CI
    • Jenkins
      • Declarative Pipeline
    • CircleCI
    • Storing baselines
    • Exit codes

User Guides

  • Configuration Reference
    • Complete example
    • Thresholds
      • Direction semantics
      • Per-metric threshold examples
    • Baseline strategies
      • last_successful (default)
      • rolling_median
      • tagged
    • Statistical mode
    • Noise handling
    • Config file location
  • Baseline Strategies
    • Overview
    • last_successful
    • rolling_median
    • tagged
    • Switching strategies
  • Building a Custom Provider
    • Provider contract
    • Minimal example
    • Register the provider
    • Use the provider
    • Testing your provider
    • Provider checklist

CLI Reference

  • CLI Reference
    • Global options
    • flameiq init
    • flameiq run
    • flameiq compare
    • flameiq baseline
      • flameiq baseline set
      • flameiq baseline show
      • flameiq baseline promote
      • flameiq baseline clear
    • flameiq report
    • flameiq validate

Architecture

  • Architecture Overview
    • Module layers
    • Dependency rules
    • Data flow
    • Design decisions
  • Module Layer Reference
    • flameiq/schema/
    • flameiq/core/
    • flameiq/engine/
    • flameiq/storage/
    • flameiq/providers/
    • flameiq/reporting/
    • flameiq/cli/
  • Schema Design
    • Versioning philosophy
    • Why dataclasses?
    • The flat() method

Specifications

  • FlameIQ Performance Schema — Version 1
    • Overview
    • JSON structure
    • Field reference
      • schema_version
      • metadata
      • metrics
        • metrics.latency
        • metrics.throughput
        • metrics.memory_mb
        • metrics.cpu_percent
        • metrics.custom
    • Flat key format
    • Validation rules
    • Serialisation contract
    • Changelog
      • Version 1.0 (2026-03-01)
  • Statistical Methodology Specification
    • Overview
    • Mann-Whitney U Test
      • Background
      • Hypothesis
      • Significance decision
      • Implementation
      • Minimum samples
    • Effect Size — Cohen’s d
      • Formula
      • Verbal labels (Cohen 1988 conventions)
    • Noise-resistant Median
      • Formula
      • Use cases
    • Determinism guarantees
  • Threshold Algorithm Specification
    • Overview
    • Step 1 — Compute change percent
    • Step 2 — Evaluate against threshold
      • Higher-is-worse metrics
      • Lower-is-worse metrics
      • Unknown / custom metrics
    • Step 3 — Warning detection
    • Threshold format
    • Default threshold
    • Overall status
  • Exit Code Specification
    • Using exit codes in CI
    • Using JSON output with exit codes

API Reference

  • flameiq.schema — Data Models
    • Environment
      • Environment.CI
      • Environment.LOCAL
      • Environment.STAGING
      • Environment.CUSTOM
    • LatencyMetrics
      • LatencyMetrics.mean
      • LatencyMetrics.p50
      • LatencyMetrics.p95
      • LatencyMetrics.p99
      • LatencyMetrics.__post_init__()
    • Metrics
      • Metrics.latency
      • Metrics.throughput
      • Metrics.memory_mb
      • Metrics.cpu_percent
      • Metrics.custom
      • Metrics.__post_init__()
      • Metrics.flat()
    • SnapshotMetadata
      • SnapshotMetadata.run_id
      • SnapshotMetadata.commit
      • SnapshotMetadata.branch
      • SnapshotMetadata.timestamp
      • SnapshotMetadata.environment
      • SnapshotMetadata.tags
    • PerformanceSnapshot
      • PerformanceSnapshot.metrics
      • PerformanceSnapshot.schema_version
      • PerformanceSnapshot.metadata
      • PerformanceSnapshot.__post_init__()
      • PerformanceSnapshot.to_dict()
      • PerformanceSnapshot.from_dict()
  • flameiq.core — Comparison Engine
    • Comparator
      • Determinism guarantee
      • Floating-point policy
      • compute_change_percent()
      • compare_snapshots()
    • Models
      • RegressionStatus
        • RegressionStatus.PASS
        • RegressionStatus.REGRESSION
        • RegressionStatus.WARNING
        • RegressionStatus.INSUFFICIENT_DATA
      • MetricDiff
        • MetricDiff.metric_key
        • MetricDiff.baseline_value
        • MetricDiff.current_value
        • MetricDiff.change_percent
        • MetricDiff.threshold_percent
        • MetricDiff.is_regression
        • MetricDiff.is_warning
        • MetricDiff.p_value
        • MetricDiff.effect_size
        • MetricDiff.direction
        • MetricDiff.abs_change_percent
        • MetricDiff.__init__()
        • MetricDiff.status_label
      • ComparisonResult
        • ComparisonResult.__init__()
        • ComparisonResult.status
        • ComparisonResult.diffs
        • ComparisonResult.baseline_commit
        • ComparisonResult.current_commit
        • ComparisonResult.statistical_mode
        • ComparisonResult.summary
        • ComparisonResult.regressions
        • ComparisonResult.warnings
        • ComparisonResult.passed
        • ComparisonResult.exit_code
        • ComparisonResult.to_dict()
    • Thresholds
      • DEFAULT_THRESHOLD_PERCENT
      • parse_threshold()
      • evaluate_threshold()
      • build_threshold_map()
    • Errors
      • FlameIQError
      • ValidationError
      • SchemaVersionError
        • SchemaVersionError.__init__()
      • ConfigurationError
      • ThresholdConfigError
        • ThresholdConfigError.__init__()
      • BaselineError
      • BaselineNotFoundError
        • BaselineNotFoundError.__init__()
      • BaselineCorruptedError
        • BaselineCorruptedError.__init__()
      • ProviderError
      • ProviderNotFoundError
        • ProviderNotFoundError.__init__()
      • MetricsFileNotFoundError
        • MetricsFileNotFoundError.__init__()
      • ComparisonError
      • InsufficientSamplesError
        • InsufficientSamplesError.__init__()
      • StorageError
      • MigrationError
  • flameiq.engine — Statistical Engine
    • Statistics
      • Supported methods (v1.0)
      • Mathematical specification
      • MINIMUM_SAMPLES
      • StatisticalResult
        • StatisticalResult.is_significant
        • StatisticalResult.p_value
        • StatisticalResult.effect_size
        • StatisticalResult.test_name
        • StatisticalResult.confidence_level
        • StatisticalResult.__init__()
        • StatisticalResult.alpha
        • StatisticalResult.effect_label
      • mann_whitney_compare()
      • noise_filter_median()
    • Baseline Strategies
      • BaselineStrategy
        • BaselineStrategy.LAST_SUCCESSFUL
        • BaselineStrategy.ROLLING_MEDIAN
        • BaselineStrategy.TAGGED
      • select_baseline()
  • flameiq.storage — Local Storage
    • BaselineStore
      • BaselineStore.__init__()
      • BaselineStore.baseline_path
      • BaselineStore.history_path
      • BaselineStore.has_baseline()
      • BaselineStore.load_baseline()
      • BaselineStore.save_baseline()
      • BaselineStore.load_history()
      • BaselineStore.clear()
  • flameiq.providers — Plugin System
    • Base class
      • Implementing a custom provider
      • MetricProvider
        • MetricProvider.name
        • MetricProvider.collect()
        • MetricProvider.validate()
        • MetricProvider.normalize()
        • MetricProvider.load()
        • MetricProvider.__abstractmethods__
        • MetricProvider.__dict__
        • MetricProvider.__module__
        • MetricProvider.__weakref__
    • JSON Provider
      • JsonProvider
        • JsonProvider.name
        • JsonProvider.collect()
        • JsonProvider.validate()
        • JsonProvider.normalize()
    • pytest-benchmark Provider
      • Mapping
      • PytestBenchmarkProvider
        • PytestBenchmarkProvider.name
        • PytestBenchmarkProvider.collect()
        • PytestBenchmarkProvider.validate()
        • PytestBenchmarkProvider.normalize()
    • Registry
      • PROVIDER_REGISTRY
      • get_provider()
      • list_providers()
  • flameiq.reporting — HTML Reports
    • generate_report()

Contributing

  • Development Setup
    • Prerequisites
    • Setup steps
    • Running the tests
    • Linting and type checking
    • Commit conventions
    • Branch naming
  • RFC Process
    • When is an RFC required?
    • RFC lifecycle
    • RFC template
  • Testing Standards
    • Test organisation
    • Coverage requirements
    • Determinism tests
    • Statistical tests
    • Fixture conventions

Project

  • Changelog
    • 1.0.0 — 2026-03-01
      • Added
    • Unreleased
  • Security Policy
    • Supported versions
    • Reporting a vulnerability
    • Security model
    • Supply chain security
FlameIQ
  • Overview: module code

All modules for which code is available

  • flameiq.core.comparator
  • flameiq.core.errors
  • flameiq.core.models
  • flameiq.core.thresholds
  • flameiq.engine.baseline
  • flameiq.engine.statistics
  • flameiq.providers.base
  • flameiq.providers.json_provider
  • flameiq.providers.pytest_provider
  • flameiq.providers.registry
  • flameiq.reporting.html_generator
  • flameiq.schema.v1.models
  • flameiq.storage.baseline_store

© Copyright 2026, FlameIQ Contributors.

Built with Sphinx using a theme provided by Read the Docs.