The sparsity constrained rank-one matrix approximation problem, also known as sparse PCA, is a difficult mathematical optimization problem which arises in a wide array of useful applications in engineering, machine learning and statistics, and the design of algorithms for this problem has attracted intensive research activities. We survey a variety of approaches for solving this problem including convex relaxations and direct approaches to the original nonconvex formulation. Convex relaxations are solved by applying fast first-order methods, while the direct approach builds on the conditional gradient method. Its simplicity allows for solving large scale problems where our usual convex relaxation techniques are limited. We show that a variety of recent and novel sparse PCA methods which have been derived from disparate approaches can all be viewed as special instances of our approach. Numerical experiments and applications with text data will be given.