I added return_as="generator" for memory performance which will break the tests on the CI machines that use joblib version 1.2, until the minimum version is bumped to 1.3

I added return_as="generator" for memory performance which will break the tests on the CI machines that use joblib version 1.2, until the minimum version is bumped to 1.3

You could maybe condition on the joblib version ?

You could maybe condition on the joblib version ?

We opened a PR to bump to the next version anyway as it will be old enough by next release 🤷‍♂️

Can someone give me a short summary and in particular the formulae of the implemented features importances?

Can someone give me a short summary and in particular the formulae of the implemented features importances?

The classical MDI uses the impurity of the nodes computed during training, which in binary classification is :
$H(t) = 1 - p_{t,0}^2 - (1 - p_{t,0})^2$. The authors of UFI suggest using a modified impurity measure: $H'(t) = 1 - p_{t,0} p^\prime_{t,0} - (1 - p_{t,0})(1 - p^\prime_{t,0})$ where $p_{t,0}$ and $p^\prime_{t,0}$ are the proportion of training and testing samples respectively of class 0 that go through the evaluated node $t$. Since we are doing bagging in Random Forests, these test samples are readily available in the out-of-bag (oob) samples.

Once these modified impurity measures are computed with the oob samples, we compute the same decrease in impurity as in traditional MDI : $\Delta^\prime (t) = \omega_t H^\prime (t) - \omega_l H^\prime (l) - \omega_r H^\prime (r)$ where $l$ and $r$ designate the left and right children of $t$ and $\omega_t$ is the proportion of training samples going through $t$.

This can be extended to multi-class classification and in regression we use a similar idea to compute node impurity that uses in and out of bag information : $H^\prime(t) = \frac{1}{n^\prime_t} \sum_{x^\prime_i \in R_t}(y^\prime_i - \bar{y}_t)^2$. $\bar{y}_t$ is the node value, $(x^\prime_i, y^\prime_i)_i$ are oob samples.

I went into more details in this document, where I also compare this method to the alternative proposed in this paper.

@lorentzenchr

@GaetandeCast Thanks a lot for your summary.

TLDR (summary)

What is the difference to the out-of-bag (oob) score of a random forest?

Details

The Brier score in binary classification ($y \in {0, 1}$) is a different name for the squared error: $BS(y_{obs}, p_{pred}) = (y_{obs} - p_{pred})^2$.
Each tree node minimizes this score $\sum_{i \in node} BS(y_{obs, i}, p_{pred, i})$, meaning it predicts $p_{pred} = \bar{y} = average(y \in node) = \frac{1}{n_{node}}\sum_{i \in node} y_i$ (so we predict probabilities like good statisticians).
Plugging it back into the Brier score gives the Brier score entropy, aka Gini score, $Gini = \sum_{i \in node} \bar{y}(1 - \bar{y})$.

If one wants to use the oob sample instead of the training sample then one simply can use the Brier score. The same logic applies to all (strictly consistent) loss functions (and to regression).

From the formula of the UFI suggestion, I am unable to recover the Brier score on the oob sample. Neither can I find any derivation in their paper. I also think it is plain wrong because they mix the empirical means of 2 independent data samples, i.e., $p$ and $p^\prime$. Unless I have not understood something fundamental, that, honestly, seems like nonsense to me.

Edit: If one restricts to the oob sample that flows through that fixed node, meaning p_pred is constant, than one has $\bar{BS}=\bar{y}(1-2p_{pred})+p_{pred}^2$.