At BuzzFeed we generate hundreds of articles a day, so choosing better headlines can save us from substantial losses in our audience engagement. Our solution is a tool that takes in multiple headline and thumbnail options for an article and decides which combination is most effective. In this talk, I discuss the models that perform best for this tool under different product scenarios. I also discuss causal analysis of the effectiveness of this tool when A/B testing is infeasible.