Rape and Sexual Abuse Service Highland and Rape Crisis Scotland said the committee’s decision had had sent a “harmful message” on women’s safety.
Police Scotland had objected to Brown’s operator’s licence continuing.
It is understood that the council licencing committee, which discussed the matter in private, had options to vote to take no action, suspend or revoke the licence.
Grafton and Highland Alliance councillor Macpherson told Scotland News that councillors voted based on information presented to them at the meeting.
Macpherson said he was assured by officials that Brown would never be permitted to drive a taxi again.
He said in a statement: “Speaking as a father of a daughter and a brother to three sisters, and as a grandfather, it troubles me that the decision I took in the committee has created a feeling of concern or insecurity about women’s safety in the Highlands.
“I would never knowingly play any part in compromising the safety of women and girls and I sincerely do not believe that I have done so in this case.
“My concern that the safety of women and girls even appears to have been compromised is too much for my conscience to bear, and this is why I am resigning.”
Macpherson said the case had been discussed in private “to protect the identities of victims”.