Система Светлый Тёмный

Редактор темы

Настройки страницы Внешний вид

  • Ширина страницы

    Переключить ширину страницы

    Позволяет изменить ширину страницы

    Выбор цвета

    Палитра цветов

    Позволяет изменить цвета некоторых элементов

    Типографика

    Выберите шрифт

    Раскладка разделов

    Сеточная раскладка

    Выстраивает разделы форума аккуратной сеткой.

  • Языки

    Выберите язык

    Вариант темы

    Светлая или тёмная тема
    Система Светлый Тёмный
Реклама
Реклама

Robots.txt, как правильно настроить?

  • Автор темы Автор темы Webster
  • Дата начала Дата начала
Webster

Webster

PRO Users
Регистрация
25 Окт 2015
Сообщения
730
Рейтинг реакций
138
Баллы
43
Возраст
35
1/3
Автор темы
Ребят, кто может подсказать? Как правильно настроить robots.txt
Вот что сейчас у меня
User-agent: *
Disallow: /engine/go.php
Disallow: /engine/download.php
Disallow: /user/
Disallow: /newposts/
Disallow: /statistics.html
Disallow: /*subaction=userinfo
Disallow: /*subaction=newposts
Disallow: /*do=lastcomments
Disallow: /*do=feedback
Disallow: /*do=register
Disallow: /*do=lostpassword
Disallow: /*do=addnews
Disallow: /*do=stats
Disallow: /*do=pm
Disallow: /*do=search
Host: www.url.com(тут ссылка на мой сайт)
 
Автор темы
Webster написал(а):
Я смотрел, думал для его сайта отдельно
Нажмите для раскрытия...
тогда с яндекса бери правильный роботс http://ya.ru/robots.txt
Код: 
User-agent: *
Disallow: /?
Disallow: /404.html
Disallow: /about.html
Disallow: /adddata
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /articles
Disallow: /chisla.html
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /cy
Disallow: /discounts/
Disallow: /dzen.html
Disallow: /i/
Disallow: /ie3/yandsearch
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /norobot
Disallow: /polling
Disallow: /redir
Disallow: /regions.html?
Disallow: /s/
Disallow: /setup
Disallow: /skazki
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl
Disallow: /yaca
Disallow: /ya
Disallow: /yandsearch
Disallow: /catalog/?text=
Disallow: /msearch
Disallow: /themes
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /sl/*.html
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /adresa-segmentator
Disallow: /all-supported-params
Disallow: /cgi-bin/hidereferer
Disallow: /cgi-bin/set-intl
Disallow: /cgi-bin/xmlsearch.pl
Disallow: /cgi-bin/yandpage
Disallow: /cgi-bin/yandsearch
Disallow: /click|/cy
Disallow: /clck
Disallow: /cycounter
Disallow: /dzen
Allow: /design-school$
Allow: /edu$
Allow: /edu/$
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /edu/ping
Disallow: /experiments.xml
Disallow: /family
Disallow: /familysearch
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/yandex-big.gaf
Disallow: /ie3/yandsearch
Allow: /jobs$
Allow: /jobs/$
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Disallow: /index_m
Disallow: /infected
Disallow: /largesearch
Disallow: /map/.+/news.html
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /quotes
Disallow: /redir
Disallow: /redir_warning
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /rubric2sport
Disallow: /save*
Disallow: /promo/skype*
Disallow: /schoolsearch
Disallow: /search/advanced
Disallow: /search/customize
Disallow: /search/extra-snippet
Disallow: /search/inforequest
Disallow: /search
Disallow: /sitesearch
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /telsearch
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /v
Disallow: /versions
Disallow: /video/*
Allow: /video/$
Allow: /video/touch/$
Disallow: /wpage
Disallow: /xmlsearch
Disallow: /yandpage
Disallow: /yandsearch
Allow: /yac2014
Disallow: /yca/cy
нихуяси
 
byroot написал(а):
тогда с яндекса бери правильный роботс http://ya.ru/robots.txt
Код: 
User-agent: *
Disallow: /?
Disallow: /404.html
Disallow: /about.html
Disallow: /adddata
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /articles
Disallow: /chisla.html
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /cy
Disallow: /discounts/
Disallow: /dzen.html
Disallow: /i/
Disallow: /ie3/yandsearch
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /norobot
Disallow: /polling
Disallow: /redir
Disallow: /regions.html?
Disallow: /s/
Disallow: /setup
Disallow: /skazki
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl
Disallow: /yaca
Disallow: /ya
Disallow: /yandsearch
Disallow: /catalog/?text=
Disallow: /msearch
Disallow: /themes
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /sl/*.html
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /adresa-segmentator
Disallow: /all-supported-params
Disallow: /cgi-bin/hidereferer
Disallow: /cgi-bin/set-intl
Disallow: /cgi-bin/xmlsearch.pl
Disallow: /cgi-bin/yandpage
Disallow: /cgi-bin/yandsearch
Disallow: /click|/cy
Disallow: /clck
Disallow: /cycounter
Disallow: /dzen
Allow: /design-school$
Allow: /edu$
Allow: /edu/$
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /edu/ping
Disallow: /experiments.xml
Disallow: /family
Disallow: /familysearch
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/yandex-big.gaf
Disallow: /ie3/yandsearch
Allow: /jobs$
Allow: /jobs/$
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Disallow: /index_m
Disallow: /infected
Disallow: /largesearch
Disallow: /map/.+/news.html
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /quotes
Disallow: /redir
Disallow: /redir_warning
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /rubric2sport
Disallow: /save*
Disallow: /promo/skype*
Disallow: /schoolsearch
Disallow: /search/advanced
Disallow: /search/customize
Disallow: /search/extra-snippet
Disallow: /search/inforequest
Disallow: /search
Disallow: /sitesearch
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /telsearch
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /v
Disallow: /versions
Disallow: /video/*
Allow: /video/$
Allow: /video/touch/$
Disallow: /wpage
Disallow: /xmlsearch
Disallow: /yandpage
Disallow: /yandsearch
Allow: /yac2014
Disallow: /yca/cy
нихуяси
Нажмите для раскрытия...
😱😱😱
 
С гугла тоже работает
Код: 
User-agent: *
Disallow: /search
Allow: /search/about
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /preferences
Disallow: /setprefs
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Disallow: /s?
Allow: /maps?*output=classic*
Allow: /maps?*file=
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview
Disallow: /mld?
Disallow: /staticmap?
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /extern_js/
Disallow: /xjs/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /kh
Disallow: /vt
Disallow: /maps/vt
Disallow: /maps/api/js/AuthenticationService.Authenticate
Disallow: /maps/api/js/QuotaService.RecordEvent
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /ads/hotels/partners
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /flights/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/reviewer
Disallow: /about/careers/apply/
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Disallow: /webmasters/sitemaps/ping?
Disallow: /ping?
Disallow: /gallery/
Disallow: /landing/now/ontap/

# Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact [email protected].
User-agent: Twitterbot
Allow: /imgres

User-agent: facebookexternalhit
Allow: /imgres

Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: https://www.google.com/sitemap.xml
 
За то майл всех троллит 😀
Код: 
User-Agent: Yandex
Allow: /$
Allow: /all$
Disallow: /
Host: https://mail.ru

User-agent: Twitterbot
Disallow: /
Allow: /?logo=

User-Agent: *
Allow: /$
Allow: /all$
Disallow: /
 
@Karabas Barabas, ну вот с них и брать... к чему темы плодить какой правильный роботс делать... нихуяси и все думают что это относится именно только к их сайту нихуясинихуясинихуяси
 
Автор темы
Код: 
User-agent: *
Allow: /engine/classes/min/index.php?
Allow: /engine/data/emoticons/
Disallow: /engine/
Host: www.,,,.com
Sitemap: http://,,,/sitemap.xml
В общем так оставил
 
А где хост, где карта сайта? Понизить их в выдаче! 😀
 
Olegjan написал(а):
А где хост, где карта сайта? Понизить их в выдаче! 😀
Нажмите для раскрытия...
зато 4 карты сайта есть 😱 и только одна с того домена с которого роботс
Код: 
Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: https://www.google.com/sitemap.xml
 
Автор темы
Ребят, офф флуд!!
 
Привет всем!У кого как настроен robots.txt?

У меня стоит так,брал с форума не помню с какой темы

Код: 
User-agent: *
Disallow: /engine/*
Disallow: */page/*/
Disallow: */page/
Disallow: /index.php?do=*
Allow: /engine/classes/js/
Allow: /engine/classes/min/index.php
Allow: /engine/data/emoticons/

Host: site.ru
Sitemap: http://site.ru/sitemap.xml
 
Чтобы ответить, войдите или зарегистрируйтесь.

Похожие темы Последние темы Популярные темы

Сейчас на сайте
667 онлайн  (1 участн., 666 гост.)
Кто онлайн
Темы 20.865
Сообщения 407.512
Участники 17.868
Новый пользователь hupomu
Назад
Сверху