What should I upgrade to get better fps?

I am running everything on ultra and getting 40-50fps on higher difficulties. It gets really bad on city center missions.

Would upgrading my GPU be enough? Or should I also upgrade my CPU?