It is difficult to remove a site from a directory once a submission has been made. However, there should be no reason why one would want to have a site removed from a directory. It is completely harmless to have a listing in a directory even if you have doubts about the quality of the directory, or have made mistakes by submitting to wrong categories, with wrong titles and descriptions. At the most, search engines will ignore these listings.
You will not be penalized on the basis of your incoming links, since you have no control over them. If this were the case, i.e., if search engines did penalize you based on your incoming link profile, it would be easy to abuse and manipulate competitor rankings by submitting rival websites to poor quality directories, using irrelevant titles and descriptions. Hence, getting your site removed from a directory should not be a cause for concern.