I see your problem. Aluminum has a poor fatigue limit (I'm not sure of the proper terms), every time you bend it it gets weaker. If you try to stretch the mirror over your bar end it will snap. It's not that flexible.
If you have access to a lathe you can turn down the bar end on the grip side so that you can slip the mirror over it. A machine shop could do both of them in five minutes. Or if you have a drill press you could mount the bar end using a screw and turn it down with a file.
It's not the worst thing. Or you could purchase the proper type bar end.