the main thing with the arms isn't even how hard the metal is. it is how precisely the axle flats fit the arm faces. if there is any slop, the axle will either rock or cut into the face of the arms, whcih will eventually let the axle rotate inside the arm.
regarding different metals and their hardnesses...aluminum *can* be soft, but there are quite a few different alloys and treatments of them, resulting in many different hardness levels (and other properties). some of them are harder than some steels.
also...if you have a soft steel, you can harden it to some degree with heating and cooling at the right rates and amounts (same with aluminum). (even harder surfaces can be made with carbon applications, "case hardening", etc, but those may not be within a particular diyer's capabilities, where the simple heating and cooling often is).
for those that havent' seen it, there is a torque arm picture thread with a lot of examples.
https://endless-sphere.com/forums/viewtopic.php?f=2&t=26444
to the op, you may wish to post the details of exactly how you use these, and their origin, in that thread for others to find.