If it started spinning before heat, then most likely you loosened the bolt slightly. The OEM bad end has a metal center part, then rubber...then the outer heavy metal part you see.
An alternative option - If you weren't happy with the stock location/angle of OEM bars, now would be a perfect time to try different handle bars. Many of us changed to the "EMGO Daytona Touring" bars. At this point, its easier to swap bars and you may like them much better. Search the forums, we've discussed this in great depths.