Ребят, кто может подсказать? Как правильно настроить robots.txt Вот что сейчас у меня User-agent: * Disallow: /engine/go.php Disallow: /engine/download.php Disallow: /user/ Disallow: /newposts/ Disallow: /statistics.html Disallow: /*subaction=userinfo Disallow: /*subaction=newposts Disallow: /*do=lastcomments Disallow: /*do=feedback Disallow: /*do=register Disallow: /*do=lostpassword Disallow: /*do=addnews Disallow: /*do=stats Disallow: /*do=pm Disallow: /*do=search Host: www.url.com(тут ссылка на мой сайт)
@Webster, поиск сломался? http://zerocoolpro.biz/forum/threads/robots-txt-kak-pravilno.4755/ Только нет в названии темы " настроить", но суть - таже!
тогда с яндекса бери правильный роботс http://ya.ru/robots.txt Code: User-agent: * Disallow: /? Disallow: /404.html Disallow: /about.html Disallow: /adddata Disallow: /advanced_engl.html Disallow: /advertising Disallow: /articles Disallow: /chisla.html Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /cy Disallow: /discounts/ Disallow: /dzen.html Disallow: /i/ Disallow: /ie3/yandsearch Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /norobot Disallow: /polling Disallow: /redir Disallow: /regions.html? Disallow: /s/ Disallow: /setup Disallow: /skazki Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /yaca Disallow: /ya Disallow: /yandsearch Disallow: /catalog/?text= Disallow: /msearch Disallow: /themes Disallow: /showcaptcha Disallow: /sitesearch Disallow: /sl/*.html Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /adresa-segmentator Disallow: /all-supported-params Disallow: /cgi-bin/hidereferer Disallow: /cgi-bin/set-intl Disallow: /cgi-bin/xmlsearch.pl Disallow: /cgi-bin/yandpage Disallow: /cgi-bin/yandsearch Disallow: /click|/cy Disallow: /clck Disallow: /cycounter Disallow: /dzen Allow: /design-school$ Allow: /edu$ Allow: /edu/$ Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /edu/ping Disallow: /experiments.xml Disallow: /family Disallow: /familysearch Disallow: /formfeedback Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/yandex-big.gaf Disallow: /ie3/yandsearch Allow: /jobs$ Allow: /jobs/$ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/smart/$ Allow: /images/touch/$ Disallow: /index_m Disallow: /infected Disallow: /largesearch Disallow: /map/.+/news.html Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people Disallow: /person Disallow: /podpiska/login.pl Disallow: /quotes Disallow: /redir Disallow: /redir_warning Disallow: /region_map Disallow: /regions_list.xml Disallow: /rubric2sport Disallow: /save* Disallow: /promo/skype* Disallow: /schoolsearch Disallow: /search/advanced Disallow: /search/customize Disallow: /search/extra-snippet Disallow: /search/inforequest Disallow: /search Disallow: /sitesearch Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /telsearch Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /v Disallow: /versions Disallow: /video/* Allow: /video/$ Allow: /video/touch/$ Disallow: /wpage Disallow: /xmlsearch Disallow: /yandpage Disallow: /yandsearch Allow: /yac2014 Disallow: /yca/cy
С гугла тоже работает Code: User-agent: * Disallow: /search Allow: /search/about Disallow: /sdch Disallow: /groups Disallow: /index.html? Disallow: /? Allow: /?hl= Disallow: /?hl=*& Allow: /?hl=*&gws_rd=ssl$ Disallow: /?hl=*&*&gws_rd=ssl Allow: /?gws_rd=ssl$ Allow: /?pt1=true$ Disallow: /imgres Disallow: /u/ Disallow: /preferences Disallow: /setprefs Disallow: /default Disallow: /m? Disallow: /m/ Allow: /m/finance Disallow: /wml? Disallow: /wml/? Disallow: /wml/search? Disallow: /xhtml? Disallow: /xhtml/? Disallow: /xhtml/search? Disallow: /xml? Disallow: /imode? Disallow: /imode/? Disallow: /imode/search? Disallow: /jsky? Disallow: /jsky/? Disallow: /jsky/search? Disallow: /pda? Disallow: /pda/? Disallow: /pda/search? Disallow: /sprint_xhtml Disallow: /sprint_wml Disallow: /pqa Disallow: /palm Disallow: /gwt/ Disallow: /purchases Disallow: /local? Disallow: /local_url Disallow: /shihui? Disallow: /shihui/ Disallow: /products? Disallow: /product_ Disallow: /products_ Disallow: /products; Disallow: /print Disallow: /books/ Disallow: /bkshp?*q=* Disallow: /books?*q=* Disallow: /books?*output=* Disallow: /books?*pg=* Disallow: /books?*jtp=* Disallow: /books?*jscmd=* Disallow: /books?*buy=* Disallow: /books?*zoom=* Allow: /books?*q=related:* Allow: /books?*q=editions:* Allow: /books?*q=subject:* Allow: /books/about Allow: /booksrightsholders Allow: /books?*zoom=1* Allow: /books?*zoom=5* Disallow: /ebooks/ Disallow: /ebooks?*q=* Disallow: /ebooks?*output=* Disallow: /ebooks?*pg=* Disallow: /ebooks?*jscmd=* Disallow: /ebooks?*buy=* Disallow: /ebooks?*zoom=* Allow: /ebooks?*q=related:* Allow: /ebooks?*q=editions:* Allow: /ebooks?*q=subject:* Allow: /ebooks?*zoom=1* Allow: /ebooks?*zoom=5* Disallow: /patents? Disallow: /patents/download/ Disallow: /patents/pdf/ Disallow: /patents/related/ Disallow: /scholar Disallow: /citations? Allow: /citations?user= Disallow: /citations?*cstart= Allow: /citations?view_op=new_profile Allow: /citations?view_op=top_venues Disallow: /s? Allow: /maps?*output=classic* Allow: /maps?*file= Allow: /maps/api/js? Allow: /maps/d/ Disallow: /maps? Disallow: /mapstt? Disallow: /mapslt? Disallow: /maps/stk/ Disallow: /maps/br? Disallow: /mapabcpoi? Disallow: /maphp? Disallow: /mapprint? Disallow: /maps/api/js/ Disallow: /maps/api/staticmap? Disallow: /maps/api/streetview Disallow: /mld? Disallow: /staticmap? Disallow: /maps/preview Disallow: /maps/place Disallow: /help/maps/streetview/partners/welcome/ Disallow: /help/maps/indoormaps/partners/ Disallow: /lochp? Disallow: /center Disallow: /ie? Disallow: /blogsearch/ Disallow: /blogsearch_feeds Disallow: /advanced_blog_search Disallow: /uds/ Disallow: /chart? Disallow: /transit? Disallow: /extern_js/ Disallow: /xjs/ Disallow: /calendar/feeds/ Disallow: /calendar/ical/ Disallow: /cl2/feeds/ Disallow: /cl2/ical/ Disallow: /coop/directory Disallow: /coop/manage Disallow: /trends? Disallow: /trends/music? Disallow: /trends/hottrends? Disallow: /trends/viz? Disallow: /trends/embed.js? Disallow: /trends/fetchComponent? Disallow: /trends/beta Disallow: /musica Disallow: /musicad Disallow: /musicas Disallow: /musicl Disallow: /musics Disallow: /musicsearch Disallow: /musicsp Disallow: /musiclp Disallow: /urchin_test/ Disallow: /movies? Disallow: /wapsearch? Allow: /safebrowsing/diagnostic Allow: /safebrowsing/report_badware/ Allow: /safebrowsing/report_error/ Allow: /safebrowsing/report_phish/ Disallow: /reviews/search? Disallow: /orkut/albums Disallow: /cbk Allow: /cbk?output=tile&cb_client=maps_sv Disallow: /kh Disallow: /vt Disallow: /maps/vt Disallow: /maps/api/js/AuthenticationService.Authenticate Disallow: /maps/api/js/QuotaService.RecordEvent Disallow: /recharge/dashboard/car Disallow: /recharge/dashboard/static/ Disallow: /profiles/me Allow: /profiles Disallow: /s2/profiles/me Allow: /s2/profiles Allow: /s2/oz Allow: /s2/photos Allow: /s2/search/social Allow: /s2/static Disallow: /s2 Disallow: /transconsole/portal/ Disallow: /gcc/ Disallow: /aclk Disallow: /cse? Disallow: /cse/home Disallow: /cse/panel Disallow: /cse/manage Disallow: /tbproxy/ Disallow: /imesync/ Disallow: /shenghuo/search? Disallow: /support/forum/search? Disallow: /reviews/polls/ Disallow: /hosted/images/ Disallow: /ppob/? Disallow: /ppob? Disallow: /accounts/ClientLogin Disallow: /accounts/ClientAuth Disallow: /accounts/o8 Allow: /accounts/o8/id Disallow: /topicsearch?q= Disallow: /xfx7/ Disallow: /squared/api Disallow: /squared/search Disallow: /squared/table Disallow: /qnasearch? Disallow: /app/updates Disallow: /sidewiki/entry/ Disallow: /quality_form? Disallow: /labs/popgadget/search Disallow: /buzz/post Disallow: /compressiontest/ Disallow: /analytics/reporting/ Disallow: /analytics/admin/ Disallow: /analytics/web/ Disallow: /analytics/feeds/ Disallow: /analytics/settings/ Disallow: /analytics/portal/ Disallow: /analytics/uploads/ Allow: /alerts/manage Allow: /alerts/remove Disallow: /alerts/ Allow: /alerts/$ Disallow: /ads/search? Disallow: /ads/plan/action_plan? Disallow: /ads/plan/api/ Disallow: /ads/hotels/partners Disallow: /phone/compare/? Disallow: /travel/clk Disallow: /hotelfinder/rpc Disallow: /hotels/rpc Disallow: /flights/rpc Disallow: /commercesearch/services/ Disallow: /evaluation/ Disallow: /chrome/browser/mobile/tour Disallow: /compare/*/apply* Disallow: /forms/perks/ Disallow: /shopping/suppliers/search Disallow: /ct/ Disallow: /edu/cs4hs/ Disallow: /trustedstores/s/ Disallow: /trustedstores/tm2 Disallow: /trustedstores/verify Disallow: /adwords/proposal Disallow: /shopping/product/ Disallow: /shopping/seller Disallow: /shopping/reviewer Disallow: /about/careers/apply/ Disallow: /about/careers/applications/ Disallow: /landing/signout.html Disallow: /webmasters/sitemaps/ping? Disallow: /ping? Disallow: /gallery/ Disallow: /landing/now/ontap/ # Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact [email protected]. User-agent: Twitterbot Allow: /imgres User-agent: facebookexternalhit Allow: /imgres Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: https://www.google.com/sitemap.xml
За то майл всех троллит Code: User-Agent: Yandex Allow: /$ Allow: /all$ Disallow: / Host: https://mail.ru User-agent: Twitterbot Disallow: / Allow: /?logo= User-Agent: * Allow: /$ Allow: /all$ Disallow: /
@Karabas Barabas, ну вот с них и брать... к чему темы плодить какой правильный роботс делать... и все думают что это относится именно только к их сайту
Code: User-agent: * Allow: /engine/classes/min/index.php? Allow: /engine/data/emoticons/ Disallow: /engine/ Host: www.,,,.com Sitemap: http://,,,/sitemap.xml В общем так оставил
зато 4 карты сайта есть и только одна с того домена с которого роботс Code: Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: https://www.google.com/sitemap.xml
Привет всем!У кого как настроен robots.txt? У меня стоит так,брал с форума не помню с какой темы Code: User-agent: * Disallow: /engine/* Disallow: */page/*/ Disallow: */page/ Disallow: /index.php?do=* Allow: /engine/classes/js/ Allow: /engine/classes/min/index.php Allow: /engine/data/emoticons/ Host: site.ru Sitemap: http://site.ru/sitemap.xml
