Estimate Recommendation Time

[Scienfitz]

It would be nice to have a utility or some functionality to estimate the amount of time that is to be expected for getting a recommendation. The most straightforward interface would be something like .estimate_recommend_time(...) which would have the identical interface as .recommend()

Since it will generally complex and possibly impossible to give an accurate estimation its reasonable to factor out the aspects that enter the total time it takes via known approximate scaling laws. This has the advantage that a lot of internal settings that affect the time are encapsulated because the actual reocmmender etc is called. The utility would then measure time for a reduced setting and scale the estimate according to reasonable approimate scaling laws.

Example:

• actual recommend call is on 100 candidates, batch size 5, 20 training points
• perform .recommend with a reduced searchspace that has 10 candidates, batch_size1 and, 5 training points
• Calculate the estimation factors
  • $f_{c}=100/10=10$
  • $f_{b}=5/1=5$
  • $f_{t}=20/5=4$
• calculate the time estimate based on the measured time: $T_{est}=T * f_{c} * f_{b} * (f_{t})^3$

Further Settings That could Possibly Be Reduced and Incorporated Easily in Scaling Laws

• n_restarts (default 10)
• num_fantasies (default 128)
• max_n_subsets (default 10)

Notes:

• this is a rough idea first, one can probably do a lot more things by adding new factors or improving the above factors (like the training size factor could come also from the acqf inference scaling and not just model training)
• For purely continuous spaces $f_{c}$ does not playa role and the outcome ight also depend on sequential greediness, but a good approximation could perhaps be found with more thought and dedicated work
• it is fine enough if these estimates provide a ballpark idea of the expected computation cost, ie seconds, hours, days or weeks...
• Recomenders in general could get a helper methods that returns an evolved copy of themselves intended for this estimate, nicely encapsulating possible setting changes and scaling factors

[nihaase]

I like the idea of providing an estimated execution time to the user. This raises a related question: is compute usage tied to calculation time? For example, when calculating a recommendation on a node, it would be helpful to know upfront how much RAM is needed. A method like .estimate_recommendation_cost(...) that returns resource requirements before execution would be very useful.

I think a dataset with some examples could help to better understand how good such a approach is.