ax.plot¶

Base¶

class ax.plot.base.AxPlotConfig[source]¶

Bases: ax.plot.base._AxPlotConfigBase

Config for plots

class ax.plot.base.AxPlotTypes[source]¶

Bases: enum.Enum

Enum of Ax plot types.

BANDIT_ROLLOUT = 4¶

CONTOUR = 0¶

GENERIC = 1¶

HTML = 6¶

INTERACT_CONTOUR = 3¶

INTERACT_SLICE = 5¶

SLICE = 2¶

class ax.plot.base.PlotData[source]¶

Bases: tuple

Struct for plot data, including both in-sample and out-of-sample arms

property in_sample¶

Alias for field number 1

property metrics¶

Alias for field number 0

property out_of_sample¶

Alias for field number 2

property status_quo_name¶

Alias for field number 3

class ax.plot.base.PlotInSampleArm[source]¶

Bases: tuple

Struct for in-sample arms (both observed and predicted data)

property context_stratum¶

Alias for field number 6

property name¶

Alias for field number 0

property parameters¶

Alias for field number 1

property se¶

Alias for field number 4

property se_hat¶

Alias for field number 5

property y¶

Alias for field number 2

property y_hat¶

Alias for field number 3

class ax.plot.base.PlotMetric[source]¶

Bases: tuple

Struct for metric

property metric¶

Alias for field number 0

property pred¶

Alias for field number 1

property rel¶

Alias for field number 2

class ax.plot.base.PlotOutOfSampleArm[source]¶

Bases: tuple

Struct for out-of-sample arms (only predicted data)

property context_stratum¶

Alias for field number 4

property name¶

Alias for field number 0

property parameters¶

Alias for field number 1

property se_hat¶

Alias for field number 3

property y_hat¶

Alias for field number 2

Bandit Rollout¶

ax.plot.bandit_rollout.plot_bandit_rollout(experiment)[source]¶

Plot bandit rollout from ane experiement.

Return type

AxPlotConfig

Contour Plot¶

ax.plot.contour.interact_contour(model, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False, fixed_features=None, trial_index=None)[source]¶

Create interactive plot with predictions for a 2-d slice of the parameter space.

Parameters

• model (ModelBridge) – ModelBridge that contains model for predictions

• metric_name (str) – Name of metric to plot

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative (bool) – Predictions relative to status quo

• density (int) – Number of points along slice to evaluate predictions.

• slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better (bool) – Lower values for metric are better.

• fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.contour.plot_contour(model, param_x, param_y, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, lower_is_better=False, fixed_features=None, trial_index=None)[source]¶

Plot predictions for a 2-d slice of the parameter space.

Parameters

• model (ModelBridge) – ModelBridge that contains model for predictions

• param_x (str) – Name of parameter that will be sliced on x-axis

• param_y (str) – Name of parameter that will be sliced on y-axis

• metric_name (str) – Name of metric to plot

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative (bool) – Predictions relative to status quo

• density (int) – Number of points along slice to evaluate predictions.

• slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters.

• lower_is_better (bool) – Lower values for metric are better.

• fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.contour.short_name(param_name)[source]¶

Return type

str

Feature Importances¶

ax.plot.feature_importances.plot_feature_importance(df, title)[source]¶

Return type

AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_feature(model, relative=True)[source]¶

One plot per metric, showing importances by feature.

Return type

AxPlotConfig

ax.plot.feature_importances.plot_feature_importance_by_metric(model)[source]¶

One plot per feature, showing importances by metric.

Return type

AxPlotConfig

ax.plot.feature_importances.plot_relative_feature_importance(model)[source]¶

Create a stacked bar chart of feature importances per metric

Return type

AxPlotConfig

Marginal Effects¶

ax.plot.marginal_effects.plot_marginal_effects(model, metric)[source]¶

Calculates and plots the marginal effects – the effect of changing one factor away from the randomized distribution of the experiment and fixing it at a particular level.

Parameters

• model (ModelBridge) – Model to use for estimating effects

• metric (str) – The metric for which to plot marginal effects.

Return type

AxPlotConfig

Returns

AxPlotConfig of the marginal effects

Model Diagnostics¶

ax.plot.diagnostic.interact_batch_comparison(observations, experiment, batch_x, batch_y, rel=False, status_quo_name=None)[source]¶

Compare repeated arms from two trials; select metric via dropdown.

Parameters

• observations (List[Observation]) – List of observations to compute comparison.

• batch_x (int) – Index of batch for x-axis.

• batch_y (int) – Index of bach for y-axis.

• rel (bool) – Whether to relativize data against status_quo arm.

• status_quo_name (Optional[str]) – Name of the status_quo arm.

Return type

AxPlotConfig

ax.plot.diagnostic.interact_cross_validation(cv_results, show_context=True)[source]¶

Interactive cross-validation (CV) plotting; select metric via dropdown.

Note: uses the Plotly version of dropdown (which means that all data is stored within the notebook).

Parameters

• cv_results (List[CVResult]) – cross-validation results.

• show_context (bool) – if True, show context on hover.

Return type

AxPlotConfig

ax.plot.diagnostic.interact_empirical_model_validation(batch, data)[source]¶

Compare the model predictions for the batch arms against observed data.

Relies on the model predictions stored on the generator_runs of batch.

Parameters

• batch (BatchTrial) – Batch on which to perform analysis.

• data (Data) – Observed data for the batch.

Return type

AxPlotConfig

Returns

AxPlotConfig for the plot.

ax.plot.diagnostic.tile_cross_validation(cv_results, show_arm_details_on_hover=True, show_context=True)[source]¶

Tile version of CV plots; sorted by ‘best fitting’ outcomes.

Plots are sorted in decreasing order using the p-value of a Fisher exact test statistic.

Parameters

• cv_results (List[CVResult]) – cross-validation results.

• include_measurement_error – if True, include measurement_error metrics in plot.

• show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.

• show_context (bool) – if True (default), display context on hover.

Return type

AxPlotConfig

Pareto Plots¶

ax.plot.pareto_frontier.interact_pareto_frontier(frontier_list, CI_level=0.9, show_parameterization_on_hover=True)[source]¶

Plot a pareto frontier from a list of objects

Return type

AxPlotConfig

ax.plot.pareto_frontier.plot_pareto_frontier(frontier, CI_level=0.9, show_parameterization_on_hover=True)[source]¶

Plot a Pareto frontier from a ParetoFrontierResults object.

Parameters

• frontier (ParetoFrontierResults) – The results of the Pareto frontier computation.

• CI_level (float, optional) – The confidence level, i.e. 0.95 (95%)

• show_parameterization_on_hover (bool, optional) – If True, show the parameterization of the points on the frontier on hover.

Returns

The resulting Plotly plot definition.

Return type

AEPlotConfig

class ax.plot.pareto_utils.COLORS[source]¶

Bases: enum.Enum

An enumeration.

CORAL = (251, 128, 114)¶

LIGHT_PURPLE = (190, 186, 218)¶

ORANGE = (253, 180, 98)¶

PINK = (188, 128, 189)¶

STEELBLUE = (128, 177, 211)¶

TEAL = (141, 211, 199)¶

class ax.plot.pareto_utils.ParetoFrontierResults[source]¶

Bases: tuple

Container for results from Pareto frontier computation.

property absolute_metrics¶

Alias for field number 5

property means¶

Alias for field number 1

property outcome_constraints¶

Alias for field number 6

property param_dicts¶

Alias for field number 0

property primary_metric¶

Alias for field number 3

property secondary_metric¶

Alias for field number 4

property sems¶

Alias for field number 2

ax.plot.pareto_utils.compute_pareto_frontier(experiment, primary_objective, secondary_objective, data=None, outcome_constraints=None, absolute_metrics=None, num_points=10, trial_index=None, chebyshev=True)[source]¶

Compute the Pareto frontier between two objectives. For experiments with batch trials, a trial index must be provided.

Parameters

• experiment (Experiment) – The experiment to compute a pareto frontier for.

• primary_objective (Metric) – The primary objective to optimize.

• secondary_objective (Metric) – The secondary objective against which to trade off the primary objective.

• outcome_constraints (Optional[List[OutcomeConstraint]]) – Outcome constraints to be respected by the optimization. Can only contain constraints on metrics that are not primary or secondary objectives.

• absolute_metrics (Optional[List[str]]) – List of outcome metrics that should NOT be relativized w.r.t. the status quo (all other outcomes will be in % relative to status_quo).

• num_points (int) – The number of points to compute on the Pareto frontier.

• chebyshev (bool) – Whether to use augmented_chebyshev_scalarization when computing Pareto Frontier points.

Returns

A NamedTuple with the following fields:

• param_dicts: The parameter dicts of the

  points generated on the Pareto Frontier.

• means: The posterior mean predictions of

  the model for each metric (same order as the param dicts).

• sems: The posterior sem predictions of

  the model for each metric (same order as the param dicts).

• primary_metric: The name of the primary metric.

• secondary_metric: The name of the secondary metric.

• absolute_metrics: List of outcome metrics that

  are NOT be relativized w.r.t. the status quo (all other metrics are in % relative to status_quo).

Return type

ParetoFrontierResults

ax.plot.pareto_utils.rgba(rgb_tuple, alpha=1)[source]¶

Convert RGB tuple to an RGBA string.

Return type

str

Scatter Plots¶

ax.plot.scatter.interact_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=True, show_CI=True, arm_noun='arm', metrics=None, fixed_features=None)[source]¶

Interactive fitted outcome plots for each arm used in fitting the model.

Choose the outcome to plot using a dropdown.

Parameters

• model (ModelBridge) – model to use for predictions.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

• show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is True.

• show_CI (bool) – if True, render confidence intervals.

• arm_noun (str) – noun to use instead of “arm” (e.g. group)

• metrics (Optional[List[str]]) – List of metric names to restrict to when plotting.

• fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

ax.plot.scatter.lattice_multiple_metrics(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False)[source]¶

Plot raw values or predictions of combinations of two metrics for arms.

Parameters

• model (ModelBridge) – model to draw predictions from.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

• show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.

Return type

AxPlotConfig

ax.plot.scatter.plot_fitted(model, metric, generator_runs_dict=None, rel=True, custom_arm_order=None, custom_arm_order_name='Custom', show_CI=True)[source]¶

Plot fitted metrics.

Parameters

• model (ModelBridge) – model to use for predictions.

• metric (str) – metric to plot predictions for.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

• custom_arm_order (Optional[List[str]]) – a list of arm names in the order corresponding to how they should be plotted on the x-axis. If not None, this is the default ordering.

• custom_arm_order_name (str) – name for custom ordering to show in the ordering dropdown. Default is ‘Custom’.

• show_CI (bool) – if True, render confidence intervals.

Return type

AxPlotConfig

ax.plot.scatter.plot_multiple_metrics(model, metric_x, metric_y, generator_runs_dict=None, rel=True)[source]¶

Plot raw values or predictions of two metrics for arms.

All arms used in the model are included in the plot. Additional arms can be passed through the generator_runs_dict argument.

Parameters

• model (ModelBridge) – model to draw predictions from.

• metric_x (str) – metric to plot on the x-axis.

• metric_y (str) – metric to plot on the y-axis.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

Return type

AxPlotConfig

ax.plot.scatter.plot_objective_vs_constraints(model, objective, subset_metrics=None, generator_runs_dict=None, rel=True, infer_relative_constraints=False, fixed_features=None)[source]¶

Plot the tradeoff between an objetive and all other metrics in a model.

All arms used in the model are included in the plot. Additional arms can be passed through via the generator_runs_dict argument.

Fixed features input can be used to override fields of the insample arms when making model predictions.

Parameters

• model (ModelBridge) – model to draw predictions from.

• objective (str) – metric to optimize. Plotted on the x-axis.

• subset_metrics (Optional[List[str]]) – list of metrics to plot on the y-axes if need a subset of all metrics in the model.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

• infer_relative_constraints (Optional[bool]) – if True, read relative spec from model’s optimization config. Absolute constraints will not be relativized; relative ones will be. Objectives will respect the rel parameter. Metrics that are not constraints will be relativized.

• fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

ax.plot.scatter.tile_fitted(model, generator_runs_dict=None, rel=True, show_arm_details_on_hover=False, show_CI=True, arm_noun='arm', metrics=None, fixed_features=None)[source]¶

Tile version of fitted outcome plots.

Parameters

• model (ModelBridge) – model to use for predictions.

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – a mapping from generator run name to generator run.

• rel (bool) – if True, use relative effects. Default is True.

• show_arm_details_on_hover (bool) – if True, display parameterizations of arms on hover. Default is False.

• show_CI (bool) – if True, render confidence intervals.

• arm_noun (str) – noun to use instead of “arm” (e.g. group)

• metrics (Optional[List[str]]) – List of metric names to restrict to when plotting.

• fixed_features (Optional[ObservationFeatures]) – Fixed features to use when making model predictions.

Return type

AxPlotConfig

Slice Plot¶

ax.plot.slice.interact_slice(model, param_name, metric_name='', generator_runs_dict=None, relative=False, density=50, slice_values=None, fixed_features=None, trial_index=None)[source]¶

Create interactive plot with predictions for a 1-d slice of the parameter space.

Parameters

• model (ModelBridge) – ModelBridge that contains model for predictions

• param_name (str) – Name of parameter that will be sliced

• metric_name (str) – Name of metric to plot

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative (bool) – Predictions relative to status quo

• density (int) – Number of points along slice to evaluate predictions.

• slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

ax.plot.slice.plot_slice(model, param_name, metric_name, generator_runs_dict=None, relative=False, density=50, slice_values=None, fixed_features=None, trial_index=None)[source]¶

Plot predictions for a 1-d slice of the parameter space.

Parameters

• model (ModelBridge) – ModelBridge that contains model for predictions

• param_name (str) – Name of parameter that will be sliced

• metric_name (str) – Name of metric to plot

• generator_runs_dict (Optional[Dict[str, GeneratorRun]]) – A dictionary {name: generator run} of generator runs whose arms will be plotted, if they lie in the slice.

• relative (bool) – Predictions relative to status quo

• density (int) – Number of points along slice to evaluate predictions.

• slice_values (Optional[Dict[str, Any]]) – A dictionary {name: val} for the fixed values of the other parameters. If not provided, then the status quo values will be used if there is a status quo, otherwise the mean of numeric parameters or the mode of choice parameters. Ignored if fixed_features is specified.

• fixed_features (Optional[ObservationFeatures]) – An ObservationFeatures object containing the values of features (including non-parameter features like context) to be set in the slice.

Return type

AxPlotConfig

Table¶

ax.plot.table_view.get_color(x, ci, rel, reverse)[source]¶

Determine the color of the table cell.

ax.plot.table_view.table_view_plot(experiment, data, use_empirical_bayes=True, only_data_frame=False, arm_noun='arm')[source]¶

Table of means and confidence intervals.

Table is of the form:

arm	metric_1	metric_2
0_0	mean +- CI	…
0_1	…	…

Trace Plots¶

ax.plot.trace.mean_trace_scatter(y, trace_color=(128, 177, 211), legend_label='mean', hover_labels=None)[source]¶

Creates a graph object for trace of the mean of the given series across runs.

Parameters

• y (ndarray) – (r x t) array with results from r runs and t trials.

• trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.

• legend_label (str) – label for this trace.

• hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

Returns

plotly graph object

Return type

go.Scatter

ax.plot.trace.model_transitions_scatter(model_transitions, y_range, generator_change_color=(141, 211, 199))[source]¶

Creates a graph object for the line(s) representing generator changes.

Parameters

• model_transitions (List[int]) – iterations, before which generators changed

• y_range (List[float]) – upper and lower values of the y-range of the plot

• generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plotly graph objects for the lines representing generator

changes

Return type

go.Scatter

ax.plot.trace.optimization_times(fit_times, gen_times, title='')[source]¶

Plots wall times for each method as a bar chart.

Parameters

• fit_times (Dict[str, List[float]]) – A map from method name to a list of the model fitting times.

• gen_times (Dict[str, List[float]]) – A map from method name to a list of the gen times.

• title (str) – Title for this plot.

Returns: AxPlotConfig with the plot

Return type

AxPlotConfig

ax.plot.trace.optimization_trace_all_methods(y_dict, optimum=None, title='', ylabel='', hover_labels=None, trace_colors=[(128, 177, 211), (251, 128, 114), (141, 211, 199), (188, 128, 189), (190, 186, 218), (253, 180, 98)], optimum_color=(253, 180, 98))[source]¶

Plots a comparison of optimization traces with 2-SEM bands for multiple methods on the same problem.

Parameters

• y – a mapping of method names to (r x t) arrays, where r is the number of runs in the test, and t is the number of trials.

• optimum (Optional[float]) – value of the optimal objective.

• title (str) – title for this plot.

• ylabel (str) – label for the Y-axis.

• hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

• trace_colors (List[Tuple[int]]) – tuples of 3 int values representing RGB colors to use for different methods shown in the combination plot. Defaults to Ax discrete color scale.

• optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plot of the comparison of optimization traces with IQR

Return type

AxPlotConfig

ax.plot.trace.optimization_trace_single_method(y, optimum=None, model_transitions=None, title='', ylabel='', hover_labels=None, trace_color=(128, 177, 211), optimum_color=(253, 180, 98), generator_change_color=(141, 211, 199))[source]¶

Plots an optimization trace with mean and 2 SEMs

Parameters

• y (ndarray) – (r x t) array; result to plot, with r runs and t trials

• optimum (Optional[float]) – value of the optimal objective

• model_transitions (Optional[List[int]]) – iterations, before which generators changed

• title (str) – title for this plot.

• ylabel (str) – label for the Y-axis.

• hover_labels (Optional[List[str]]) – optional, text to show on hover; list where the i-th value corresponds to the i-th value in the value of the y argument.

• trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

• optimum_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

• generator_change_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plot of the optimization trace with IQR

Return type

AxPlotConfig

ax.plot.trace.optimum_objective_scatter(optimum, num_iterations, optimum_color=(253, 180, 98))[source]¶

Creates a graph object for the line representing optimal objective.

Parameters

• optimum (float) – value of the optimal objective

• num_iterations (int) – how many trials were in the optimization (used to determine the width of the plot)

• trace_color – tuple of 3 int values representing an RGB color. Defaults to orange.

Returns

plotly graph objects for the optimal objective line

Return type

go.Scatter

ax.plot.trace.sem_range_scatter(y, trace_color=(128, 177, 211), legend_label='')[source]¶

Creates a graph object for trace of mean +/- 2 SEMs for y, across runs.

Parameters

• y (ndarray) – (r x t) array with results from r runs and t trials.

• trace_color (Tuple[int]) – tuple of 3 int values representing an RGB color. Defaults to blue.

• legend_label (str) – Label for the legend group.

Returns

plotly graph objects for lower and upper bounds

Return type

Tuple[go.Scatter]