Kerala Devaswom Minister K Radhakrishnan on Tuesday said caste system was a "stain" in the minds of certain people and it was not easy to remove it instantly, remarks that came in the wake of his disclosure of facing discrimination at a temple in the state.