The conditional gradient method is a long-studied first-order optimization method for smooth convex optimization. Its main appeal is the low computational complexity: the conditional gradient method requires only a single linear optimization operation per iteration. In contrast, other first-order methods require a projection computation every iteration, which is the computational bottleneck for many large scale optimization tasks. The drawback of the conditional gradient method is its relatively slow convergence rate. In this work we present a new conditional gradient algorithm for smooth and strongly convex optimization over polyhedral sets with a linear convergence rate - an exponential improvement over previous results. We extend the algorithm to the online and stochastic settings, and give the first conditional gradient algorithm that attains optimal regret bounds for both arbitrary convex losses and strongly convex losses. Our online and stochastic algorithms require a single linear optimization step over the domain per iteration.