Two ways to alleviate the problem.
First, and best, swab the bore between shots, or at a minimum every other shot. A wet patch followed by two dry patches. Muzzle loader accuracy is based on consistency. Keeping the fouling consistent from shot to shot is as important as measuring the powder and the patch ball combination. Personally, at a match, I usually have a can or bottle of diet soda with me. I place some patches on the bench and soak them by pouring the soda on them. o the point they could be squeezed and have drops of maisture come out. Use the jag to pump one up and down two or three times and then the dry patches.
The second method, is to use a very wet patch that softens and wipes the fouling with each shot. Success with this depends a bit on the rifling in the bore. Some folks claim bore butter will do this. Others claim odd mixtures of pine oil and other stuff works for them.
I have done a lot of target competition shooting over the years and an overwhelming majority of match competitors swab the bore between shots.
There have been a lot of methods tried over the years to reduce fouling from shot to shot. Everything from wads under the ball, various concoctions of patch lube and changing powder granulation.
Lastly, if your patch and ball combination is a little loose, or, you are shooting more powder than can be consumed within the bore, fouling increases in multiples. As long as black powder is burning under pressure, the fouling stays minimal. When there is no pressure or not much pressure, the fouling left is much higher. So if you shoot a 110 grains in a short barrel traditions deer hunter, the powder will still be burning after the ball exits the muzzle, it leaves much more fouling than backing down to 80 grains, which is about what can be consumed within the bore before the ball exits. (that extra powder does impart some extra velocity, but at a much lower proportion per grain of powder.
For minute of deer accuracy, it comes down to what works for you.