"The first matter that the slave will be brought to account for on the Day of Judgment is the prayer.
The Prophet (pubh) said, " The first thing for which a person will account during resurrection day is his prayer. If it was good, he will be successful; and if it was bad, he will be a loser." ( Tirmidhee, Nasaa'ee and others)
Prayer is one of the pillars of Islam, so when the person who neglects it is described as a kaafir, this implies the kind of kufr that puts a person beyond the pale of Islam, because he has destroyed one of the pillars of Islam.