Nice post. It's a duh and a jeh all in once. We see a lot of things go wrong with robots.txt and sitemaps because somehow all website developers, hobbyist and pro alike, need to do some seo it seems. They then turn to the usual suspects that are top of mind resulting in people putting in regular expressions in the robots as well. So the want to disallow a url and end up take down a whole tag or category with 40 valuable pages on it.

Truth of the matter is that the real masters of the crawl and indexing process do not use robots.txt nor a xml sitemap. A well designed and structured site should provide ample breadcrumbs for the spider to follow starting with an effective menu and navigation structure on the front page for every website as not doing this causes a chain reaction like effect on the indexability of a site as a whole. When we do not cut corners we manage to get millions of pages indexed in the first year for new websites. How? With with followed links on to targets we choose and for the rest it’s never cutting corners.