UPDATED 6.34PM Labour leader Andrew Little said Prime Minister John Key's first ejection from Parliament as Prime Minister was "shameful" and showed he had "lost control."

Mr Key was thrown out of Parliament today after he kept talking in a robust exchange about the Panama papers despite the Speaker turning off his microphone and telling him to sit down. 

It was the first time Mr Key has been ordered to leave by the Speaker since he became Prime Minister, although he had been kicked out three times as an Opposition MP.

Green co-leader James Shaw had demanded Mr Key apologise for his claim that Greenpeace and Amnesty were implicated in the Panama Papers, when the charities had been used without their knowledge in sham trusts. Mr Little said Mr Key had lost control.

"It's pretty shameful when the Prime Minister gets kicked out. The one person who we do need to know keeps his cool in Parliament is the Prime Minister. He lost it. He's on the back foot on this issue, so his point of attack now is to distract, throw various dead cats on the table so you start talking about that and when that doesn't work, start demeaning people."

Mr Shaw said it was important the public saw Mr Key was held to account for his actions. "He got himself chucked out because he could not handle it. He got angry and he got himself thrown out of the Chamber."

Deputy Prime Minister Bill English, who was left to field a question to the PM from NZ First's Ron Mark, defended Mr Key, saying his lapse was "accidental." "He was simply addressing the Greens vigorously and he just didn't see the Speaker stand up. If he had seen it, he would have sat down because he was warned about it yesterday." He did not believe it was a bad look for the public. "I think they know Parliament is a pretty robust place for debate. I don't think they'll be overly concerned about it." He said he had seen Question Times which were far more disorderly than today's.

Mr Shaw and Mr Little fronted to media together after Question Time, saying it was important for the Opposition to take a joint stand holding Mr Key to account for his behaviour.

The last time a Prime Minister was told to leave was Helen Clark in June 2005. Then National Party leader Don Brash was ejected at the same time. Neither Mr Little or Mr Shaw have ever been ejected. 

Minister Nick Smith said he had been ejected several times himself "so I would be the last to judge." 

@metiria I recall Speaker Wall throwing PM Lange & Oppn Leader Bolger out together within in first five minutes

— Peter Dunne (@PeterDunneMP) May 11, 2016

After bizarre rant where PM claims he's defending everyday, ordinary NZs who use foreign trusts, he rants over speaker and gets kicked out

— Jacinda Ardern (@jacindaardern) May 11, 2016

Key demeaned the Office of PM today in QT. Even the Speaker couldn't bear the arrogance. #booted #nzqt

— David Cunliffe (@DavidCunliffeMP) May 11, 2016