Update:
I don't think I can edit the post twice. Can't seem to find the button again.
Just jumped the bike and went for a ride (around 20-30 minutes long) and that didn't seem to help it. I did notice the check engine light came on at high RPM as well which I have never seen before. Might just give up and take it to the shop, it's in need of a service anyway.