Early spring is a good time to remove heavy branches. I'll often remove the topmost branches and reduce some of the other branches to encourage growth in the branches I want to keep. How much you can remove depends on the health of the tree. Removing any branches will reduce the vigor to some extent, but that's necessary to get the lower branches stronger.
If the soil is poor, getting the roots into good soil at repotting time will help too.