Eye gaze input system has been used for severe physical disabilities such as ALS patients who are difficult to communicate with others. In conventional eye gaze input systems under natural light, the condition that human wears glasses has not been into account. In this study, we propose a method to estimate eye gaze direction of human who wears glasses. In the method, image processing is conducted as follows; the parts of glasses in eye image after thresholding are identified by conducting labeling processing, and removed them to detect inner corner of the eye, outer corner of the eye, and part of black eye. A three layer neural network is composed to estimate eye gaze direction. In the neural network, the input signals are position coordinates of inner corner of the eye, outer corner of the eye, and center of black eye, and the output signals are the eye gaze directions. To examine the effectiveness of the proposed method, experiments in which human subjects follow a moving target on the screen by eye movement are conducted for both conditions of wearing glasses and without glasses. As the results, it is confirmed that the accuracy of the proposed method for the condition wearing glasses is as good as that without glasses, which shows the effectiveness of the proposed method.