Same-sex marriage advocates on Thursday launched legal action in Australia's highest court against a controversial government plan for a postal vote on the issue, as debate raged over whether the ballot should be boycotted.

Although marriage equality is supported by a majority of Australians, the country has not joined other nations in allowing homosexual couples the right to wed and the issue has dragged on after more than a decade of bitter political wrangling.

The opposition Labor Party, along with many gay rights campaigners, want a free parliamentary vote, with MPs not restricted by party policy.

But Prime Minister Malcolm Turnbull, who supports marriage equality, is battling with right-wing members of his own Liberal Party, who are against legalising such unions.

His conservative government last year made an election pledge to hold a national referendum. On Wednesday the upper house Senate, where the government lacks a majority, rebuffed the poll plan for a second time, triggering a voluntary postal vote instead.

Both options are strongly opposed by gay marriage advocates, who argue that a national vote is expensive and will prove divisive, subjecting gay people and their families to hate speech.

Labor senator Penny Wong, a leading pro-gay marriage figure who has two children with her female partner, has accused the government of "handball(ing) a hard decision to the community", and also of failing to defend the LGBTI community against hate speech.

Marriage equality advocates said Canberra may also be exceeding its executive authority in holding the paper ballot, with the Public Interest Advocacy Centre Thursday commencing proceedings in the High Court to stop it.

"We will be arguing that by going ahead without the authorisation of parliament, the government is acting beyond its power," said the advocacy group's chief Jonathon Hunyor.

"These are important issues about the way that power is exercised by governments and the role of parliament in our democracy."

Under the government's plans, costing Aus$122 million (US$96 million), ballots will be sent out by the Australian Bureau of Statistics from September 12, with the outcome of the voluntary vote announced in mid-November.

If a majority of Australians vote "yes", the government would hold a free vote in parliament on the issue, with MPs not bound by party policy or the postal ballot's result.

If there is a "no" outcome, there would be no parliamentary vote.

Some supporters have argued for the public ballot to be boycotted as it would open the door to hate speech against the LGBTI community.

Former High Court judge and prominent gay marriage supporter Michael Kirby said Thursday he would shun the postal vote as it was "irregular and unscientific".

"It's just a complete political improvisation and it's completely unacceptable and it should stop," Kirby, who is gay, told the Australian Broadcasting Corporation.

"And I feel as a citizen I'm being treated in a second-class way."

Prime Minister Turnbull on Thursday spoke out against the calls for a boycott.

"I encourage every Australian to exercise their right to vote on this matter. It's an important question," he told reporters in Canberra.

Source: AFP