Submitting to directories or receiving links from directories, can in no way negatively impact your site's rankings. If that were the case, it would be easy to negatively impact competitor rankings, by submitting their sites to poor quality directories. Search engines do not penalize a site based on its incoming links since one does not have any control over who links to their site and how they choose to do so. A drop in website rankings, therefore, could be due to various other factors, a few being:

• Giving outgoing links to spammy or irrelevant websites
• Engaging in excessive reciprocal linking with irrelevant websites
• Competitors improving the quality and content of their sites
• Proactive link building by other sites