I have been that guy.... it's a nightmare. There is nothing worse then being on fish then all chance ruined (for you AND him) by some bloke smashing pegs into the bank. A few years ago I packed up and went home at ~ 9 pm after some geezer showed up with one of these super powered head torches and set his bivvy up with a mallet on the other side of this tiny bay. Game over.
I've never used a mallet, don't carry one, and I've fished on some terrible ground. Where there's a will there's a way I've found.
Honestly, rain permitting, I'd choose to sleep under the stars over using a mallet.