Senior Congress leader Kamal Nath, who recently dismissed speculation of him switching to the Bharatiya Janata Party (BJP), set rumour mills abuzz again after telling party workers that he would not "impose" himself on them and would "leave" if they want him to.