A Kevin Rudd challenge for Julia Gillard’s job wouldn’t necessarily make him the prime minister.

First, there would be no vacancy in the prime ministership to fill unless Gillard resigned or was dismissed by Governor-General Quentin Bryce. The G-G cannot be advised by the Labor caucus and would have no cause to dismiss Gillard unless she was the subject of a successful vote of no-confidence in the lower house.

Assuming, however, that Gillard resigned and recommended to the Governor-General that she appoint Rudd PM — or more interestingly, recommended the G-G appoint a non-blue-tie-wearing woman, such as Tanya Plibersek — is the Governor-General obliged to act on that advice? No. The vice-regal role to appoint a prime minister is a “reserve power”. We usually think of the reserve powers as powers that are extremely rarely exercised, such as the dismissal of Gough Whitlam in 1975; it’s exercised without the advice of a responsible minister.

A “responsible minister” is not someone who is particularly responsible in the general sense, but a minister who is responsible to the Parliament as a member of a government that has the support of a majority of the lower house. The Governor-General is not advised by a responsible minister as to the appointment of the PM because an outgoing leader would no longer be responsible to the House for his or her advice.

So the Governor-General has a discretion in the appointment. But this discretion is governed by a very strong constitutional convention — she must appoint the person who is most likely to command the confidence of the lower house. Normally, this is clear — the leader of the party with a majority in the lower house. But in the case of a minority government, things get murky.

If Rudd became Labor leader and the independents made it clear they wouldn’t support him, then a question would arise as to what the Governor-General should do. Are the independents prepared to support Tony Abbott as PM? If so, the Governor-General might contemplate appointing him. More likely, the independents would simply say they don’t support Rudd, but would not express confidence in anyone else.

What should the Governor-General do then? Her best option would be to let the House of Representatives decide by a vote of confidence. If Parliament had been adjourned, she could seek an assurance from Rudd, before appointing him, that he would either recall Parliament and face a vote of confidence or that he would advise the immediate dissolution of the Parliament and an election. While this could not be a formal condition of his appointment, she could legitimately seek such an assurance, just as John Kerr sought and received an assurance from Malcolm Fraser in 1975 that he would ensure the passage of supply and advise a double dissolution.

If it appeared no one held the confidence of the House, she could legitimately appoint Rudd to continue the existing government — when all things are otherwise equal, a principle in favour of incumbency applies. Given an election will be held soon in any case, and Rudd would therefore spend most of his time as prime minister in caretaker governance prior to an election resolving the issue of responsibility, this would be a justifiable approach.