Three people have also been reported missing in provinces south of the main island of Luzon, the disaster agency added in a statement, with communications to some regions still severed.
Typhoon Goni, which battered provinces south of the capital Manila on Sunday, is the 18th to hit the Philippines this year and one of the strongest typhoons since Haiyan killed more than 6,300 people in 2013.
"The goal should be zero casualties, but since people were forcibly evacuated our casualties were reduced," Harry Roque, a spokesman for Duterte, told a news conference.
Duterte is to fly to Manila from his southern hometown of Davao for an aerial inspection of some of the worst-hit areas, Roque added. He was in Davao when the typhoon struck, provoking some public criticism.
The Philippine police chief was also travelling to Guinobatan in the province of Albay after a lawmaker reported about 300 houses were buried under volcanic rock and mud flows from the Mayon volcano.
Goni, packing gusts of up to 310 kph (190 mph), destroyed up to 80% of homes in several towns in Catanduanes, Senator Richard Gordon, the chief of the Philippine Red Cross, told media.
With its communication and power lines down, the province of 275,000 people was cut off, said Energy Secretary Alfonso Cusi.
Goni hit 2.1 million residents of Luzon, which accounts for more than two-thirds of the economy, leaving more than 50,000 homes without power by Monday, while authorities said more than 300,000 people were still in evacuation centres.
Strong winds and torrential rain have damaged crops worth 1.1 billion pesos ($23 million), mainly of rice and corn, Agriculture Secretary William Dar told a news conference.
Before Goni hit land, the Philippines had been grappling with the impact of typhoon Molave, which killed 22 people, most of whom drowned, in provinces south of Manila.
Another storm, Atsani, is gaining strength in the Pacific Ocean as it approaches the Philippines, which is usually lashed by about 20 tropical storms every year.
Vietnam said Goni is forecast to hit its central coast on Wednesday night, dumping more heavy rain in an area where floods and landslides in the past month have already killed about 160 people, with dozens missing.