So you’ve already answered your own question then. “You are taking away ‘ecclesiastical authority’ by walking with your feet.” is a false, illogical, or poorly worded statement. A person who leaves a congregation does not take away ecclesiastical authority. That authority is given by consent of the members who are governed within that local church assembly (that’s what ecclesiology is, at least in independent churches). When you leave, you deny them authority to govern your life, but you do not take their authority to willingly govern within a local church framework.