Well, well, well... This mod fixed my problem with this schizo swinging at invisible flying gnomes with his fishing pole instead of fishing.
https://www.nexusmods.com/skyrimspecialedition/mods/134408