You keep talking about his minutes as if they exist in a vacuum. They don't.
Against better teams you typically play your best players more.
What happens when you give someone more minutes? Someone else gets less.
His minutes dropping against better teams has much more to do with the fact...