Hard to Track Inflation