Speaker Trevor Mallard has been challenged again over his use of the term "rape" to describe someone who was subsequently stood down from working at Parliament.

And he is still refusing to comment, telling the governance and administration select committee this morning that he had been asked not to by a man who "feels identified".

Mallard did not say whether the man was the parliamentary worker who had been stood down, and refused to answer questions after the hearing.

"I am not saying anything," Mallard said repeatedly.

Last month Mallard said that he believed a rapist was working in the parliamentary precinct a day after the Debbie Francis review of the parliamentary workplace was released.

The report found systemic issues of bullying and harassment, poor conduct too often tolerated and normalised, and a perception of low accountability.

Mallard had said that he believed a man was responsible for three serious sexual assaults mentioned in the review, and that he believed the man was still working at Parliament.

His comments - called shocking by some parliamentary workers - sparked a series of turbulent events that led to a historical assault complaint and a parliamentary staffer being stood down.

Mallard then said that a threat to safety had been removed from the premises.

The staffer, who the Herald has chosen not to identify, has since lashed out at Mallard, saying he felt bullied out of the workplace and was the victim of Mallard's "slanderous" comments.

National deputy leader Paula Bennett called Mallard's comments a "serious lapse in judgment", but Prime Minister Jacinda Ardern has said she continues to have confidence in him.

During today's select committee, National MP Melissa Lee told Mallard that his "rape" comments had made a lot of people, including herself, feel uncomfortable and asked him whether he stood by them.

Mallard said he had been contacted by a man asking him not to comment further.

"One of the people who feels that my comments reflected on him ... has asked me not to make any further comments.

"Now I haven't agreed to that but it's something that I'm contemplating and until I've contemplated it, I'm not going to make any further comments."

Mallard told the committee that the culture change recommended by the Frances review would happen, and a special committee was being set up to help make it happen.