how to understand the model structure exported

[illsheep]

I thought I would see some formulas,but it seems that the file only contains the weights of various features?

[MoonYLiang]

Thanks for the question!

The exported file currently contains the parameterized structure of the model, including the weights associated with the U, C, and T components. From these parameters, the corresponding formulas can be reconstructed.

More specifically, for an input vector z, each BL unit defines three terms:

• U(z): utility term
• C(z): inequality-constraint
• T(z): equality-constraint

A simplified form is

$$U(z) = \sum_i a_i z_i \quad C(z) = \sum_i b_i z_i \quad T(z) = \sum_i c_i z_i$$

and the unit output is combined as

$$score(z) = \lambda_U * U(z) - \lambda_C * C(z) - \lambda_T * T(z)$$

where lambda_U, lambda_C, and lambda_T control the relative contribution of the three parts. In the optimization view, this corresponds to a utility-maximization problem of the form

$$\max_z U(z) \quad s.t. C(z) \le 0, T(z) = 0$$

and in the implemented scoring form, the constraint terms are relaxed into penalties weighted by lambda_C and lambda_T.

For the Boston Housing example with depth [2, 1]:

First layer

The first layer takes the original input variables as input. For example, for a first-layer block, the exported U component

lambda 1.35
CRIM -0.1
ZN -0.09
...

corresponds to $U_{B11}(z) = -0.10·CRIM - 0.09·ZN + ...$

and similarly $C_{B11}(z)$ and $T_{B11}(z)$ are constructed from their own learned coefficients.

The final score of that unit is then

$$B_{11}(z) = \lambda_U·U_{B11}(z) - \lambda_C·C_{B11}(z) - \lambda_T·T_{B11}(z)$$

Second layer and optimization interpretation

The second layer takes the outputs of lower-level blocks as inputs. For example, the exported parameters of block B21 are

lambda 1.83
B11 -0.78
B12 0.5
C -0.08

Using these parameters, the three components of the top-level block can be written as

$$U(z) = -0.78·B_{11} + 0.5·B_{12} - 0.08$$

$$C(z) = 0.23·B_{11} + 0.09·B_{12} + 0.21$$

$$T(z) = 0.07·B_{11} - 0.76·B_{12} + 0.65$$

where $B_{11}$ and $B_{12}$ are the outputs of the first-layer BL blocks.

The BL structure therefore corresponds to the following optimization problem:

$$\max_z (-0.78·B_{11} + 0.5·B_{12} - 0.08)$$

subject to

$$0.23·B_{11} + 0.09·B_{12} + 0.21 \le 0$$

$$0.07·B_{11} - 0.76·B_{12} + 0.65 = 0$$

In the implemented model, this constrained problem is relaxed into the penalized score

$$score(z)=\lambda_U U(z)-\lambda_C C(z)-\lambda_T T(z)$$

with the learned penalty weights $\lambda_U=1.83$, $\lambda_C=1.10$, and $\lambda_T=1.25$.

Output

Finally, the model output is obtained by applying a linear output layer to the top-level block output:

$$y = w·B + b$$

So the exported file shows the learned parameters that define this hierarchical optimization-inspired structure.