Why The President’s Party Often Determines Midterm Outcomes

Historically, the first-term president’s party loses seats in the first midterm election following his election. Why is that the case?