The social contract doesn't legitimize anything on its own; it depends on consent.

Indeed, if there were no prior agreement, what would give
the minority any obligation to submit to the choice of the
majority (unless the election was unanimous)? A hundred
men want to have a master; what gives them the right to vote
on behalf of ten who don’t? The law of majority voting is itself
something established by agreement, and it presupposes
that on at least one occasion there was a unanimous vote.^[1]

1. The Social Contract. 5-We must always go back to a first agreement ↩

Every man who seeks to hold coercive force over you will seek to legitimize that use of force.