When to refactor code?

Only refactor when the problem and the solution are both staring at you in the face.