Advisor & Search Space¶

Advisor¶

Dr.Opt supports the following advisors:

Advisor	Description
Anneal	Begin with sampling from the prior and tends over time to sample from points closer and closer to the best ones observed.
Evolution	Randomly initialize a population based on the search space. It chooses better ones and does some mutation on them to get the next generation. Evolution may require many trials to work (ref).
Gaussian Process (coming soon)	A sequential model-based optimization (SMBO) approach that uses Gaussian Process as the surrogate (ref).
Grid Search	Perform an exhaustive search in a specified subset of the parameter space.
Random	Randomly select hyper-parameter combinations in the search space. Researches show that it might be surprisingly effective (ref).
TPE	A sequential model-based optimization (SMBO) approach that uses Tree-structured Parzen Estimators as the surrogate (ref).

Anneal¶

Arguments

optimize_mode: maximize (default) or minimize

Example

{
    "builtinTunerName": "Anneal",
    "classArgs": {"optimize_mode": "maximize"}
}

Evolution¶

Arguments

optimize_mode: maximize (default) or minimize
population_size: The initial size of the population

Example

{
    "builtinTunerName": "Evolution",
    "classArgs": {
        "optimize_mode": "maximize",
        "population_size": 100
    }
}

Grid Search¶

Arguments

None

Config example

{
    "builtinTunerName": "GridSearch"
}

Random¶

Arguments

None

Config example

{
    "builtinTunerName": "Random"
}

TPE¶

Arguments

optimize_mode: maximize (default) or minimize

Example

{
    "tuner": {
        "builtinTunerName": "TPE",
        "classArgs": {"optimize_mode": "maximize"}
    }
}

Search Space¶

Each parameter to search is assigned with certain space type. Dr.Opt currently supports the following search space types:

choice¶

Choose from a list of available options.

Format

A list of of numbers or strings, e.g., [0.1, 0.01, 0.001, 0.0001] or [“Adam”, “SGD”, “Adadelta”]

Example

{
    "learning_rate": {
        "_type": "choice",
        "_value": [0.1, 0.01, 0.001, 0.0001]
    }
}

randint¶

Choose a random integer within an interval.

Format

[lower_bound (inclusive), upper_bound (exclusive)]

Example

{
    "batch_size": {
        "_type": "randint",
        "_value": [8, 65]
    }
}

uniform¶

Choose a number randomly from a uniform distribution on an interval.

Format

[lower_bound (inclusive), upper_bound (exclusive)]

Example

{
    "droptout_rate": {
        "_type": "uniform",
        "_value": [0.1, 0.5]
    }
}

quniform¶

Choose a number randomly from an interval descretized by a fixed step size.

Format

[lower_bound (inclusive), upper_bound (exclusive), step]

Example

{
    "input_size": {
        "_type": "quniform",
        "_value": [224, 417, 32]
    }
}

Note: In this example, the possible values are: 224, 256, 288, 320, …, 384, 416.

normal¶

Choose a number randomly from a normal discribution with prescribed mean (\(\mu\)) and standard deviation (\(\sigma\)).

Format

[\(\mu\), \(\sigma\)]

Example

{
    "dropout_rate": {
        "_type": "normal",
        "_value": [0.5, 0.1]
    }
}