They're not compatible; different technologies.
Youll need to get either a brushless controller for your brushless motor, or a brushed motor for your brushed controller.
If your brushless motor has no other wires than three thick ones (no bundle of 5 or more thin ones), you will also need to make sure it is a sensorless controller, as one that requires hall sensors in the motor won't work. Some of them are called dual-mode, but if it doesn't specifically say it works without halls or sensorless, it might not be that kind and wouldn't work with your motor.
Iff your brushless motor does have the additional 5 or more thin wires, and one pair is red and black, and the other three match the thick wire colors, then it is probably a sensored motor and those are the hall power/ground and sensors. There can be other wires for speedometer or temperature sensor, often white or purple; not required for operation but handy in some cases.
