Bill English says an inquiry into the abuse of more than a thousand children in state care would be a "significant distraction".
The Human Rights Commission on Monday became the latest group to call for an investigation to determine the extent of the abuse dating back to the 1950s, but the prime minister has questioned whether there would be anything to gain from it.
The commission has written an open letter to Mr English, signed by a number of prominent New Zealanders, urging an inquiry and calling for an apology for all those affected.
"Until we know the full story and until we have the answers to those questions, we are not in a position to learn from what happened and to prevent it from happening again," the letter says.
The letter has been signed by 29 people, including Race Relations Commissioner Susan Devoy, Chief Human Rights Commissioner David Rutherford, UNICEF executive director Vivien Maidaborn and Otago University Dean of Law Professor Mark Henaghan.
The inquiry was first recommended by Judge Carolyn Hendon who chaired the Confidential Listening and Assistance Service panel that heard stories from more than 1100 abuse victims.
As at September last year, the Ministry for Social Development had received 1370 direct claims of abuse prior to 1993.
Mr English said lessons had already been learned from CALS which influenced the overhaul of the state care system to protect children in the future.
"A question is really whether an inquiry that goes back over the same history is going to make it mean that we do a better job of dealing with the people who are in care right now, and in my view it would probably be a significant distraction," he said.
He said inquiries overseas had become "wide ranging, pretty sprawling, expensive and long-running" and he already knew what it would reveal.
"It would tell us there was some pretty significant abuse of various sorts and some people have really suffered from it, and that's why the Crown's got a compensation process in place," he said.
Last year the government revealed it had settled more than 900 abuse claims, including a personal apology and compensation totalling more than $17 million.
Social Development Minister Anne Tolley has previously rejected calls for a blanket apology, preferring to give a personal apology to victims who come forward.
Labour, the Green Party and Maori Party have all backed inquiry calls.