Bagging draws B bootstrap samples from the training set, fits an unconstrained tree to each, and averages the predictions. Because the trees are grown on resampled but identically-distributed data and averaged, bias is close to that of a single tree while variance falls roughly like Var(single tree) times the average pairwise correlation between trees — adding trees drives that average toward the correlation floor and then the curve plateaus.
A random forest adds one twist: at each split, only a random subset of mtry features is considered. This decorrelates the trees further (lower average correlation), which lowers variance more than bagging alone at the same tree count and depth, at little to no bias cost.
Boosting starts from a constant f₀ and repeatedly fits a shallow tree hₘ to the current residuals, updating f ← f + ν·hₘ. Each new tree explicitly targets what the ensemble still gets wrong, so bias falls fast even with very shallow trees (stumps). But because later trees fit residuals that are increasingly just noise, a large learning rate ν combined with enough rounds lets the ensemble memorize the training noise: variance rises and test error can turn back up, unlike bagging's plateau.
Out-of-bag (OOB) error exploits the ~37% of points each bootstrap sample excludes: averaging each training point's predictions only over the trees that did not see it gives a validation-quality error estimate with no separate test split, and it has no boosting analogue because boosting uses the full training set at every round.
Bagging: f̂(x) = (1/B) Σ_b Tb(x), Var(f̂) ≈ ρσ² + (1-ρ)σ²/B
Random forest: as above, with smaller ρ from feature subsampling at each split
Boosting: f₀ = ȳ, fm(x) = f_{m-1}(x) + ν·hm(x), hm fit to {yi − f_{m-1}(xi)}OOB error: ê_i = (1/|{b: i ∉ boot_b}|) Σ_{b: i∉boot_b} Tb(xi)