Boosting methods are supervised learning methods that combine weak learning models into more accurate and predictive models. We connect some well-known boosting methods --- AdaBoost and Incremental Forward Stagewise Regression (FS-epsilon) and its variants with adaptive shrinkage parameters --- to first-order methods for convex optimization, and we present new computational guarantees for these boosting methods. In particular, we show that AdaBoost is an instance of Mirror Descent which, through duality, solves the maximum margin problem. Furthermore, we show that Forward Stagewise Regression is an instance of the classical subgradient descent method to minimize the correlation between residuals and predictors. For both AdaBoost and Forward Stagewise Regression, we show that a minor variant of the algorithm, which requires no new assumptions and falls into the same convex optimization family as the original algorithm, corresponds to the Frank-Wolfe method on a constraint-regularized problem, whereby the modified boosting procedures converge to optimal solutions of the appropriate regularized problems at an O(1/k) rate. The flexible variant of FS-epsilon yields an O(1/k) convergent algorithm for the least squares LASSO fit for *any* regularization parameter and *any* data-set --- thereby quantitatively characterizing the nature of regularization implicitly induced by Forward Stagewise Regression and its flexible variants.