Sorry, don't mean this to be a flippant answer, but it really depends on the rifle, the barrel, and how they were made/assembled.
Some rifles, not much affect at all. Others, quite a bit.
I like to let mine cool a lot when at the range, for a variety of reasons. Notably, hunting shots tend to be cold barrel shots, and also it's easier on the throat of the barrel (especially for overbore/hot loads).
Every gun is different. You just got to shoot and see if it starts to change point of impact, as you shoot. I always have hunting guns, shot in for a cold first shot. Some times guns that are not free floated are worse. Some times free floating barrels change too. Cryogentics may help.