What should i do? please help?

well there is this retarded guy that sis infatuated with me... he texts me and calls me. while we were at school he would hit on me. he asks me to go to his house and to call him and he asks how me and my boyfriend are doing. i told him that my boyfriend and i couldnt see eachother much and we cant talk much either and he got worse. what should i do? i dont want to hurt his feelings.