Decision Trees & Ensembles by Hand: Entropy, Information Gain, Random Forests & Boosting

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Count the defaults in Kestrel Lending's 20 past loans and measure how mixed the group is with entropy (in bits) and Gini impurity, the two scores every decision tree uses.
• Score three candidate first questions for the tree (employment, income, debt-to-income) by their information gain and let a formula pick the winner.
• Let the data choose the DTI threshold: for six candidate cut-points you count both sides, compute the weighted Gini, and pick the cut-point with the purest split.
• Grow Kestrel's first real tree: DTI at the root, income on the left, employment on the right. You fill the four leaves, turn the tree into one nested IF and measure its training accuracy.
• Score trees of depth 1 to 5 on held-out test loans, chart training against test accuracy, and choose the depth with cost-complexity pruning.
• Combine five trees into a random forest: count the votes, turn them into a probability and a prediction, show that the forest beats every single tree, and count the out-of-bag rows of a bootstrap sample.
• Run three rounds of gradient boosting for a regression target: start from the mean, fit a one-split stump to the residuals, add it with a learning rate, and watch the error fall.
• Compute how much each split reduced the Gini impurity, add the decreases up per feature with SUMIF, normalise them to importances and chart the ranking.