You'll have a much easier time with an oscilloscope, but it sounds like your encoder is something like 180 degrees out of alignment.
Read this: https://www.e-kart.fr/images/stories/technique/SEVCON/sevcon-app-note-pmac.pdf
If the link doesn't work, google for "sevcon app note". It explains what you're experiencing.
You probably already know, but keeping phase 1 the same, and flipping leads 2/3 will reverse the motor's direction from forward to reverse.
Next, your cos signal should lead the sin signal by 90 degrees in forward direction, so that's your order of whether or not you need to flip the sin/cos leads to the encoder. If you don't have an oscilloscope you can be very slow with a voltmeter and figure it out.
HOWEVER, if your encoder is fully 180 degrees out of alignment it can turn in the wrong direction, and sometimes it's actually rideable. BE CAREFUL though, because if you're at that point, it can be pretty unstable and race like a banshee with regenerative braking enabled and no load on the motor. This is why the scope is important to use, so you can visually see the encoder's alignment with your phases.
Finally, if everything works, and you've used a scope to guarantee it's well aligned but very rough and sounds bad, it could also be noise. (I'm still measuring PWM noise from the gen4 on my encoder as soon as the contactor closes, that throws off the position whenever I'm pulling more than 300A AC current. That's even with everything heavily shielded and shields grounded.) But it sounds like you're not at that point yet, that it's more likely just encoder alignment.
