Some touch hole liners are made to be removed, with a screw driver, allen wrench etc. Some are installed semi permanently and are only changed when firing erodes the hole open too far.
The general sense of most target shooters, is that we want to liner coned on the inside, so there are a few grains of powder visible inside the touch hole. In addition, the closer those powder grains are to the pan, the faster the gun goes off. Now we aren't talking a few minutes or seconds faster, we are talking a few thousandths of a second faster mpossibly even a couple of hundredths of a second faster. For all practical purposes, imperceptable to the average human ear. However, given the movement of the barrel, the game, the natural delay from trigger to ball exit, that very short interval of time makes a difference down range, where it counts. Perhaps not a huge difference, If two shooters of equal ability were shooting two like guns, with the only difference being in the touch hole liners, it might make a couple points different on a 100 yd target in a ten shot string. For the average hunter, not so big a difference. To a target shooter, for whom every point matters, it is a very big deal. Matches, trophies and prizes are won or lost over a point or two.
Now, when it comes down to the consistency from shot to shot, we measure, load, prime, squeeze the triger and follow through as consistently as we can. The touch hole liner is just one more slight thing that plays into that consistency. If you want to keep on pulling that liner to clean your hunting gun, have at it. It probably won't matter for your shooting anyway. Once you get away from production guns and worry about flint, powder toward or away from the liner, breathing, how the light hits the sights, cross breezes, etc, then the touch hole liner will matter more than ever. You will want the powder as close to the pan as possible and those allen screw liners keep the powder too far away.
