Interface

abstract type AbstractBenchmark

Abstract type interface for a benchmark problem.

The following methods are mandatory for benchmarks:

• generate_dataset
• generate_statistical_model
• generate_maximizer

The following methods are optional:

• plot_data
• objective_value
• compute_gap

struct DataSample{I, F<:AbstractArray, S<:Union{Nothing, AbstractArray}, C<:Union{Nothing, AbstractArray}}

Data sample data structure.

Fields

• x::AbstractArray: features

• θ_true::Union{Nothing, AbstractArray}: target cost parameters (optional)

• y_true::Union{Nothing, AbstractArray}: target solution (optional)

• instance::Any: instance object (optional)

average_tensor(x)

Average the tensor x along its third axis.

compute_gap(
    bench::AbstractBenchmark,
    dataset::AbstractVector{<:DataSample},
    statistical_model,
    maximizer
) -> Any
compute_gap(
    bench::AbstractBenchmark,
    dataset::AbstractVector{<:DataSample},
    statistical_model,
    maximizer,
    op
) -> Any

Default behaviour of compute_gap for a benchmark problem where features, solutions and costs are all defined.

compute_gap(::AbstractBenchmark, dataset::Vector{<:DataSample}, statistical_model, maximizer) -> Float64

Compute the average relative optimality gap of the pipeline on the dataset.

generate_dataset(::AbstractBenchmark, dataset_size::Int; kwargs...) -> Vector{<:DataSample}

Generate a Vector of DataSample of length dataset_size for given benchmark. Content of the dataset can be visualized using plot_data, when it applies.

generate_maximizer(::AbstractBenchmark; kwargs...)

Generates a maximizer function. Returns a callable f: (θ; kwargs...) -> y, where θ is a cost array and y is a solution.

generate_statistical_model(::AbstractBenchmark; kwargs...)

Initializes and return an untrained statistical model of the CO-ML pipeline. It's usually a Flux model, that takes a feature matrix x as input, and returns a cost array θ as output.

get_path(
    parents::AbstractVector{<:Integer},
    s::Integer,
    d::Integer
) -> Vector{T} where T<:Integer

Retrieve a path from the parents array and start sand endd`` of path.

grid_graph(
    costs::AbstractArray{R, 2};
    acyclic
) -> SimpleWeightedGraphs.SimpleWeightedDiGraph{Int64}

Convert a grid of cell costs into a weighted directed graph from SimpleWeightedGraphs.jl, where the vertices correspond to the cells and the edges are weighted by the cost of the arrival cell.

• If acyclic = false, a cell has edges to each one of its 8 neighbors.
• If acyclic = true, a cell has edges to its south, east and southeast neighbors only (ensures an acyclic graph where topological sort will work)

This can be used to model the Warcraft shortest paths problem of

Differentiation of Blackbox Combinatorial Solvers, Vlastelica et al. (2019)

highs_model() -> JuMP.Model

Initialize a HiGHS model (with disabled logging).

maximizer_kwargs(
    _::AbstractBenchmark,
    sample::DataSample
) -> NamedTuple{(:instance,), <:Tuple{Any}}

For benchmarks where there is an instance object, maximizer needs the instance object as a keyword argument.

maximizer_kwargs(
    _::AbstractBenchmark,
    sample::DataSample{Nothing, F, S, C}
) -> @NamedTuple{instance::Nothing}

For simple benchmarks where there is no instance object, maximizer does not need any keyword arguments.

neg_tensor(x)

Compute minus softplus element-wise on tensor x.

objective_value(
    _::AbstractBenchmark,
    θ::AbstractArray,
    y::AbstractArray
) -> Any

Default behaviour of objective_value.

objective_value(
    bench::AbstractBenchmark,
    sample::DataSample{I, F, S, C<:AbstractArray},
    y::AbstractArray
) -> Any

Compute the objective value of given solution y.

objective_value(
    bench::AbstractBenchmark,
    sample::DataSample{I, F, S<:AbstractArray, C}
) -> Any

Compute the objective value of the target in the sample (needs to exist).

path_to_matrix(
    path::Vector{<:Integer},
    h::Integer,
    w::Integer
) -> Matrix{Int64}

Transform path into a binary matrix of size (h, w) where each cell is 1 if the cell is part of the path, 0 otherwise.

plot_data(::AbstractBenchmark, ::DataSample; kwargs...)

Plot a data sample from the dataset created by generate_dataset. Check the specific benchmark documentation of plot_data for more details on the arguments.

scip_model() -> JuMP.Model

Initialize a SCIP model (with disabled logging).

squeeze_last_dims(x)

Squeeze two last dimensions on tensor x.

coord_to_index(
    i::Integer,
    j::Integer,
    h::Integer,
    w::Integer
) -> Any

Given a pair of row-column coordinates (i, j) on a grid of size (h, w), compute the corresponding vertex index in the graph generated by grid_graph.

count_edges(h::Integer, w::Integer; acyclic)

Compute the number of edges in a grid graph of size (h, w).

index_to_coord(
    v::Integer,
    h::Integer,
    w::Integer
) -> Tuple{Any, Any}

Given a vertex index in the graph generated by grid_graph, compute the corresponding row-column coordinates (i, j) on a grid of size (h, w).

is_minimization_problem(_::AbstractBenchmark) -> Bool

Check if the benchmark is a minimization problem.

plot_instance(::AbstractBenchmark, instance; kwargs...)

Plot the instance object of the sample.

plot_solution(::AbstractBenchmark, sample::DataSample, [solution]; kwargs...)

Plot solution if given, else plot the target solution in the sample.

fit(
    transform_type,
    dataset::AbstractVector{<:DataSample};
    kwargs...
)

Fit the given transform type (ZScoreTransform or UnitRangeTransform) on the dataset.

reconstruct!(t, dataset::AbstractVector{<:DataSample})

Reconstruct the features in the dataset in place.

reconstruct(t, dataset::AbstractVector{<:DataSample}) -> Any

Reconstruct the features in the dataset.

transform!(t, dataset::AbstractVector{<:DataSample})

Transform the features in the dataset in place.

transform(t, dataset::AbstractVector{<:DataSample}) -> Any

Transform the features in the dataset.