There’s always going to be some people in your church who will never really see you as their leader/elder/pastor. They’ll come along, they’ll attend, they’ll agree with this and disagree with that. They might find you arrogant (ironic!), they might think you’re too young (again, read 1Tim 4:12), they might have some theological idea about how church leaders shouldn’t have authority (???). But in the end, in their eyes, your not their pastor.
The only thing you can do, as Scripture says, is set “an example in speech, in conduct, in love, in faith, in purity”. Only by God’s grace can you become someone’s pastor, as you continue to wash them with the word of the gospel.
Don’t bother trying anything else. And… don’t bother caring much about it either. There’s nothing you can do.