First time trying to do a low-poly head.

IDK if I messed up the UVs, the mesh, or what but I'm getting this weird "hairiness" on my model, especially around the eyelids and nose.
Right is the texture I painted in PS. It's a bit rough but it again looks way wobblier on the model for some reason. Trying to fix it in Texture Paint hasn't been successful

Am I gonna have to remesh the head/face?