This work presents a study about a special class of infeasible solutions called here as pseudo-feasible solutions in bilevel optimization. This work is focused on determining how such solutions can affect the performance of an evolutionary algorithm. After its formal definition, and based on theoretical results, two conditions to detect and deal with them are proposed. Moreover, a novel and scalable set of test problems with characterized pseudo-feasible solutions is introduced. Furthermore, an algorithm designed to solve bilevel optimization problems (BOP) is adapted with the above mentioned conditions and tested in already known test problems and also in the new testbed so as to analyze its performance when compared with state-of-the-art evolutionary approaches for BOPs. The obtained results suggest that the presence of pseudo-feasible solutions can be considered as a source of difficulty in this type of optimization problems, since their presence may lead to incorrect comparisons among algorithms.