Restrictions on liberties of speech, association, worship, and choice of profession interfere more with deliberations about what sort of person to be than restrictions on traffic rules, safety regulations, or income disposal.
Though these liberties evidently include quite a bit, there is no suggestion here that any and all liberty deserves protection. Why not? Insofar as Mill defends individual liberties by appeal to deliberative values, he can distinguish the importance of different liberties in terms of their role in practical deliberation. A central part of practical deliberation is forming ideals and regulating one’s actions and plans in accordance with these ideals. But some liberties seem more central than othe