This application sounds fairly similar to nylon couplings, but those nylon sleeves are typically paired with shafts that are teethed rather than smooth. Is there any option to make some mating indentations or scoring on the outside of your aluminum shaft? Or to roughen up the outside of the shaft in any way?

Assuming not, I think that you do need to have multiple broad fingers inside your sleeve to apply the pressure. They should be equally spaced.

Nylon couplings use strong nylon grades, like nylon 66 with glass fibers. I think without mating gears, nylon is a better choice than POM as it tends to be slightly more flexible and so would be more likely to handle a press fit. POM actually has lower friction than nylon, which is probably not a good property for this application, where you will need it to grip once fitted.