When one usually states "I could care less", they usually mean "I could not care less". [e.g. "I could care less about this conversation."]
In order for one to "care less" about a subject, they must first care about it somewhat. Saying "I could care less about ... " does indeed imply, nay dictate, that there is some degree of care.
I've put together a handy chart to help visualize...
At zero interest you care so little that you couldn't possibly care any less. It can easily be seen that, in absolute terms, there is no caring at the zero marker. It is impossible to care less than that amount.
So for those of you unsure you may now be able to annoyingly correct your friends. How rewarding. :P