The fuse is in the fan for those of us who put multiple items on one circuit. In this case, the 4A fuse in the fan will blow if the fan motor stalls (say a stick fell down into the fan blades) while the 20A fuse in the distribution box will not blow, leaving your all-important TV and 2000W stereo operating. So, if there is more than the fan on the circuit, I suggest you need the fuse. If not, maybe you don't need the fuse. The general rule is the fuse in the distro box should be sized appropriately for the wiring, while the fuse at the appliance is sized for the appliance.
Depending on the design of your speed controller, there may be periods during each on/off cycle that look like a short circuit to the fuse. This is particularly true if the controller incorporates an H-bridge and the designer wasn't careful to allow some dead time between cycles. That can be hard on fuses, not to mention battery life.
The mystery, as I see it, is that the fuse is blowing while the fan is not in use or immediately when you turn it on. This suggests a wiring error somewhere else in the former case and a poor PWM design in the latter. Perhaps the fuse is blowing while you are "sorting out wiring" and the problem will cease to exist as soon as that is all taken care of.
I could go on, but it will get way too techy. If you have a part number and source for your PWM, please PM me and I can look into it further. I got bored with this thread some time ago and nobody else seemed that interested, but I do have a couple solutions to the whole PWM issue, neither of which is totally satisfactory to me but you may revive my interest

and maybe version 3, on the back burner long enough, may yet see the light of day.
In anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away.
-- Antoine de Saint-Exupery