I am using LG E410 running on android 4.1 and the phone has an headphone icon at the notification bar even if the headphone is not connected. This means for me to answer a call, i have to put loud speaker on.
Though this is not at all a question for this platform but try cleaning your headphone port on your phone. I've been through similar situation. It can be dust otherwise water. Try drying the phone with blow dryer :)