The fire erupted about 03:30 (22:15 GMT Saturday) when an explosion was set off in a store of fireworks ready for celebrations of a local Hindu new year festival, local media reports say.
Sparks from the display ignited a magazine of fireworks, causing a bigger blaze, police said.
Some reports suggested the explosions caused the temple’s roof to collapse, crushing worshippers inside.
The tragic accident happened a few days before the Vishu festival slated for April 14 in Kerala.
According to local media, the fire has been brought under control and the victims have been rushed to a hospital in Kollam, while the critically injured ones were transferred to a hospital in the state capital Thiruvananthapuram.
It has been reported that one of the firecrackers sparked a cache of pyrotechnics, which caused a massive explosion, the Times of India says.
The high number of casualties was due to the collapse of the temple building, the Indian Express reports.
Fireworks and firecrackers are commonly used at temple festivals and other public events and accidents are not uncommon.