ProbabilisticMetrics#

class ProbabilisticMetrics[source]#

Define and customize probalistic metrics.

Notes

Probabilistic metrics compare a value against a distribution of predicted values, such as ensemble forecasts. Available probabilistic metrics include:

  • CRPS (Continuous Ranked Probability Score)

  • Brier Score

Skill score metrics (CRPSS and BSS) are enabled by setting the reference_configuration attribute to a valid reference forecast, such as “climatology” or “persistence”.

See teehr.evaluation.generate.GeneratedTimeseries for more information on generating reference forecasts.

Methods

class BrierScore(*, return_type: str | ~pyspark.sql.types.ArrayType | ~pyspark.sql.types.MapType = None, unpack_results: bool = False, unpack_function: ~typing.Callable = None, reference_configuration: str = None, transform: ~typing.Any = None, backend: str = 'numba', summary_func: ~typing.Callable | None = None, input_field_names: str | ~teehr.models.str_enum.StrEnum | ~typing.List[str | ~teehr.models.str_enum.StrEnum] = ['primary_value', 'secondary_value', 'member'], threshold: float = 0.75, output_field_name: str = 'mean_brier_score', func: ~typing.Callable = <function ensemble_brier_score>, attrs: ~typing.Dict = {'category': MetricCategories.Probabilistic, 'display_name': 'Brier Score - Ensemble', 'optimal_value': 0.0, 'requires_threshold_field': False, 'short_name': 'brier_score_ensemble', 'value_range': [0.0, 1.0]})#

Brier Score for ensemble probabilistic forecasts.

Parameters:

  • threshold (float) – The threshold to use for binary event definition.

  • output_field_name (str) – The output field name, by default “mean_brier_score”.

  • func (Callable) – The function to apply to the data, by default probabilistic_funcs.ensemble_brier_score().

  • attrs (Dict) – The static attributes for the metric.

  • reference_configuration (str) – The reference configuration for skill score calculation, by default None. If specified skill score will be included in the results.

class CRPS(*, return_type: str | ~pyspark.sql.types.ArrayType | ~pyspark.sql.types.MapType = None, unpack_results: bool = False, unpack_function: ~typing.Callable = None, reference_configuration: str = None, transform: ~typing.Any = None, backend: str = 'numba', summary_func: ~typing.Callable | None = None, input_field_names: str | ~teehr.models.str_enum.StrEnum | ~typing.List[str | ~teehr.models.str_enum.StrEnum] = ['primary_value', 'secondary_value', 'member'], estimator: ~teehr.models.metrics.basemodels.CRPSEstimators = 'pwm', output_field_name: str = 'mean_crps_ensemble', func: ~typing.Callable = <function ensemble_crps>, attrs: ~typing.Dict = {'category': MetricCategories.Probabilistic, 'display_name': 'Continuous Ranked Probability Score - Ensemble', 'optimal_value': 0.0, 'requires_threshold_field': False, 'short_name': 'crps_ensemble', 'value_range': [None, None]})#

Continous Ranked Probability Score - Ensemble.

Parameters:

  • estimator (str) – CRPS estimator, can be (“pwm”, “nrg”, or “fair”). Default is “pwm”.

  • output_field_name (str) – The output field name, by default “mean_crps_ensemble”.

  • func (Callable) – The function to apply to the data, by default probabilistic_funcs.ensemble_crps().

  • attrs (Dict) – The static attributes for the metric.

  • reference_configuration (str) – The reference configuration for skill score calculation, by default None. If specified skill score will be included in the results.