Documentation/Inspection ask/tell

[benjamc]

• document that ask and tell does not care about n_trials (but the initial design does)
• should there be no initial design preset for using ask/tell?
• what happens if you use the random facade /sobol and you ask for more than the specified trials? does sobol depend on the number of trials? if it does, raise a warning

[timruhkopf]

should there be no initial design preset for using ask/tell?

I was actually hoping that one could just set n_configs=0 in get_initial_design, but this line seems to silently overwrite user intention and still asks for at least one configuration from the initial design. I think this is bad conduct.
Btw. setting n_trials=0 in the scenario will raise, because of the n_configs=1 overwrite here

Either way the initial_design configuration is populated in the ConfigSelector._initial_design_configs and only executed/ask'ed for when ConfigSelector.iter is being called, which i think should only be done once the intensifier is being queried for a new configuration. So i suspect that an ask call will trigger this initial design config (checked this just now and it actually does)

Similarly passing None to the HyperparameterOptmizationFacade.__init__(initial_design=None) will just instantiate the default by get_initial_design which I think is confusing behaviour. But this is apparently how facades work.

[timruhkopf]

Changing the previously mentioned line to _n_configs=max(0,min(...)) will indeed produce no initial design configs, but in turn raises -- This line warns us about a potential malfunction of smac without initial design configs, but as it turns out an ask will eventually collect the first config by configspace.sample_configuration by means of ConfigSelector.__iter__ if there is no runhistory provided in advance. So it seems that the ConfigSelector can actually handle this edge case.

however the int(max(0, min())) has the side effect, that once we set n_configs=1 scenario.n_trials=1 this will yield a int(max(0,0.25)) and in turn not sample a configuration from the initial_design despite our best intentions

[timruhkopf]

what happens if you use the random facade /sobol and you ask for more than the specified trials? does sobol depend on the number of trials? if it does, raise a warning

is not an issue. Only dependency is through the above mentioned line. I.e. in determining how many configs should be used in the initial design based off the assumed share of max_ratio of the trials : min(self._n_configs, (max_ratio * scenario.n_trials). Other than that, once the initial configs that have been determined in the ConfigSelector and are depleted by ask in ConfigSelector.__iter__ the ask will continue to run the usual __iter__ loop including model training. At this point n_trials becomes irrelevant and we can continue to ask indefinetly.