A social worker took an eight-week-old baby from its parents - then gave it back - because she thought she smelled cannabis.

Freyja and Laurence Maisey say the Ministry for Vulnerable Children (Oranga Tamariki) only gave them the cannabis explanation four days after their baby Charlie was taken.

They said police and a social worker turned up at their Whanganui house on October 12 without warning. Within minutes, Charlie was taken.

Laurence Maisey said he did not even get a chance to kiss his baby boy goodbye.

"We asked for an explanation and were told nothing. They just took him and we didn't know if we were getting him back."

Ministry acting Manawatu regional manager Monica Miranda said a temporary placement order was granted so social workers could investigate safety concerns for the baby.

"Sometimes, to do the right thing for children, it is necessary to remove them and place them in care arrangements outside the home, often temporarily, but sometimes long term," Miranda said.

But Freyja Maisey's midwife, Annette Turner-Steele, a former social worker, said there was no reason for Charlie to be taken.

She said it was unfair that the parents were accused of taking drugs without any evidence.

The Maiseys say they were told there were concerns of drugs on the Monday after Charlie was taken away. They immediately went to get drug-tested at Whanganui's Alcohol and Other Drug Service (AODS). The tests came back clean.

AODS sent the results directly to the ministry's Whanganui office and a meeting was set up that same day.

Turner-Steele stepped in as the parents' advocate and attended the meeting with the parents and social workers from the ministry.

"I knew they were drug-free and there were no drugs in their house. But after the drug test came back clear, we were told there were a lot more concerns.

"I don't know what their concerns were, none of their concerns were justified at all and they were just covering their own backs basically because they took a child wrongly."

About half an hour after the meeting, the parents received a phone call and the baby was returned.

Miranda said the baby was returned once social workers were satisfied there were no immediate concerns for his welfare.

"The ministry will continue to work with the baby's family around support that can be offered - the child's well-being is the absolute priority."

Freyja Maisey suffers cerebral palsy and epileptic seizures but an arrangement was made so that either the baby's father or grandfather would be with her and Charlie at all times.

Turner-Steele said she believed the ministry wanted to take Charlie the moment he was born.

"They told me they thought the parents were smelly and a bit slow. I said I don't believe that's grounds to take a child off its parents and we can provide wraparound services for that.

"They agreed, with a plan in place, they would give them a chance," Turner-Steele said.

Turner-Steele said she was visiting every day, sometimes twice a day, and the social worker from the ministry had been doing regular check-ins as well.

"There had been no problem. They seem to be able to do whatever they want and get away with it and they don't have to justify it."