# robots.txt for http://www.neprimer.com/ # NO AI bots 2023.12.05: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: FacebookBot Disallow: / User-agent: SBIder Disallow: / User-agent: * Disallow: /ads.txt Disallow: /_private/ Disallow: /_vti_cnf/ Disallow: /_vti_log/ Disallow: /_vti_pvt/ Disallow: /ap/ Disallow: /bus-min/ Disallow: /cc-dwld/ Disallow: /emof/ Disallow: /scripts/ Disallow: /stats/ Disallow: /whis/ Disallow: /ws_ftp.log Disallow: /xnep-bkm.htm Disallow: /mail-ord.txt Disallow: /mSendmail-rr.com.txt User-agent: www.deadlinkchecker.com Disallow: /shoppingbasket/ Crawl-delay: 1 # 8/29/11 changed: User-agent: * 174.121.174.74 # 2/4/11 `SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)` spider agent was purposly causing 404 errors so I would look them up and read their self-advert. # Disallow: /images/ removed 8/20/05 b/c of https://validator.w3.org/checklink?uri=neprimer.com%2Fimages%2F&summary=on&hide_type=all&depth=&check=Check # - Disallow: # not required for = /images/cc/ # - Disallow: # = /images/catapics/ # is not required # Must have 2 carrage returns to properly terminate file otherwise robot can # ignore robots.txt # 12/6/00 = moved Disallow: /cc/ # # EVERYTHING YOU WANTED TO KNOW ABOUT robots.txt at: https://www.robotstxt.org/orig.html # "Any number of agent id(s) can be placed on the User-Agent line so long as # they are separated by white space (WS), but the User-Agent line must have at # least one agent id." or see: http://www.seoconsultants.com/robots-text-file/ # # 1/31/01 changed Disallow: /whis/ to protect emof for his-wayh.org # 3/4/02 added 'Disallow: /scripts/' # 7/31/02 was User-agent: * ip3000.com cyveillance.com mts.net omniseek.com # 4/7/05 added Disallow: /cc-dwld/ and The real answer is that /robots.txt is not # intended for access control, so don't try to use it as such. Think of it as a # "No Entry" sign, not a locked door. If you have files on your web site that # you don't want unauthorized people to access, then configure your server to do # authentication, and configure appropriate athorization. Basic Authentication has # been around since the early days of the web (and in e.g. Apache on UNIX is trivial # to configure), and if you're really serious, SSL is commonplace in web servers. # http://www.robotstxt.org/wc/faq.html#nosecurity