Brakes keep moving position

Apologies for the lack of correct terms but my bike has cantilever brakes and the rear brake pad holding arm keeps shifting position, causing most often the right rear brake to push up and scrape against the rear wheel. I have tried to tighten all the bolts with an allen key but this shifting still occurs, any tips on how to rectify this?

Are the brake arms not retracting properly or is the pad mount itself that moves?

Yeah the pad mounts aren't stable and keep shifting

You can take the arms off the pegs and grease them without changing any settings - just note which hole the spring slots in to before sliding off the peg.. Each arm has a grub screw which can be used to balance the two blocks on the wheel; screwing tighter pulls that side block away from the wheel.

Also ensure the main cable is hooked on the centre of the straddle wire. If it's sitting to one side, it can cause this problem.

But if the pads themselves are moving, it would suggest they are not the correct type for your brakes and the shaft is too small in diameter for the clamp to hold it.

I've realised the cause is that when I brake the right arm doesn't release back afterwards, it just stays close to the rim, whereas the left arm releases back fine, how can I fix this?

Probably needs a clean and a smear of grease on the pivot. See what Brommyboy said above. Could also be the cable.