The rebels had hidden inside the mosque in Kishtwar district, about 210 kilometres north-east of state's winter capital Jammu after they were chased by an army patrol, a senior official said.
'They all belong to the Lashkar-e-Taiba (LET, a militant group based in Pakistan),' Brigadier General Gurdip Singh was quoted as saying by IANS news agency.
Singh said one of the militants killed had been identified as an LET area commander Yusuf Gujjar.
Lakshar rebels have been active in India-administered Kashmir for nearly two decades. The organization, ostensibly banned in Pakistan, was formed in the late 1980s with the aim of driving Indian security forces out of Kashmir.
The LET was recently accused by India of having planned and executed the terrorist attack on financial hub Mumbai in November 2008 in which over 170 people were killed.
The disputed Kashmir region is divided into two parts, one administered by India the other by Pakistan. The neighbours have fought two wars over the region.
More than 45,000 people have died in India-administered Kashmir since violent secessionist militant movement emerged in the 1980s.
India has repeatedly accused Pakistan of aiding Kashmiri militants. Islamabad has denied the charge, calling the insurgents freedom fighters.