Dismiss Notice
ВАШИ ПРАВА ОГРАНИЧЕНЫ!

Зарегистрируйтесь на форуме, чтобы стать полноценным участником сообщества!

Robots.txt, как правильно настроить?

Discussion in 'Вопросы- ответы' started by Webster, Jul 17, 2016.

17.07.16 в 22:49
Webster
05.09.17 в 20:59
texnar349
25
5,814
0
Reply to Thread
Page 1 of 2
  1. TopicStarter Overlay
    Webster

    Webster
    PRO Users

    Joined:
    Oct 25, 2015
    Messages:
    730
    Best Answers:
    3
    Ratings:
    +163 / 3 / -0
    Ребят, кто может подсказать? Как правильно настроить robots.txt
    Вот что сейчас у меня
    User-agent: *
    Disallow: /engine/go.php
    Disallow: /engine/download.php
    Disallow: /user/
    Disallow: /newposts/
    Disallow: /statistics.html
    Disallow: /*subaction=userinfo
    Disallow: /*subaction=newposts
    Disallow: /*do=lastcomments
    Disallow: /*do=feedback
    Disallow: /*do=register
    Disallow: /*do=lostpassword
    Disallow: /*do=addnews
    Disallow: /*do=stats
    Disallow: /*do=pm
    Disallow: /*do=search
    Host: www.url.com(тут ссылка на мой сайт)
     
    #1 Webster, Jul 17, 2016
  2. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    #2 Ботяра, Jul 17, 2016
  3. TopicStarter Overlay
    Webster

    Webster
    PRO Users

    Joined:
    Oct 25, 2015
    Messages:
    730
    Best Answers:
    3
    Ratings:
    +163 / 3 / -0
    #3 Webster, Jul 17, 2016
  4. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    тогда с яндекса бери правильный роботс http://ya.ru/robots.txt
    Code:
    User-agent: *
Disallow: /?
Disallow: /404.html
Disallow: /about.html
Disallow: /adddata
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /articles
Disallow: /chisla.html
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /cy
Disallow: /discounts/
Disallow: /dzen.html
Disallow: /i/
Disallow: /ie3/yandsearch
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /norobot
Disallow: /polling
Disallow: /redir
Disallow: /regions.html?
Disallow: /s/
Disallow: /setup
Disallow: /skazki
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl
Disallow: /yaca
Disallow: /ya
Disallow: /yandsearch
Disallow: /catalog/?text=
Disallow: /msearch
Disallow: /themes
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /sl/*.html
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /adresa-segmentator
Disallow: /all-supported-params
Disallow: /cgi-bin/hidereferer
Disallow: /cgi-bin/set-intl
Disallow: /cgi-bin/xmlsearch.pl
Disallow: /cgi-bin/yandpage
Disallow: /cgi-bin/yandsearch
Disallow: /click|/cy
Disallow: /clck
Disallow: /cycounter
Disallow: /dzen
Allow: /design-school$
Allow: /edu$
Allow: /edu/$
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /edu/ping
Disallow: /experiments.xml
Disallow: /family
Disallow: /familysearch
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/yandex-big.gaf
Disallow: /ie3/yandsearch
Allow: /jobs$
Allow: /jobs/$
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Disallow: /index_m
Disallow: /infected
Disallow: /largesearch
Disallow: /map/.+/news.html
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /quotes
Disallow: /redir
Disallow: /redir_warning
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /rubric2sport
Disallow: /save*
Disallow: /promo/skype*
Disallow: /schoolsearch
Disallow: /search/advanced
Disallow: /search/customize
Disallow: /search/extra-snippet
Disallow: /search/inforequest
Disallow: /search
Disallow: /sitesearch
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /telsearch
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /v
Disallow: /versions
Disallow: /video/*
Allow: /video/$
Allow: /video/touch/$
Disallow: /wpage
Disallow: /xmlsearch
Disallow: /yandpage
Disallow: /yandsearch
Allow: /yac2014
Disallow: /yca/cy
    нихуяси
     
    #4 Ботяра, Jul 17, 2016
  5. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    :eek::eek::eek:
     
    #5 Karabas Barabas, Jul 17, 2016
    • Нравится Нравится x 1
  6. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    :D:D:D
     
    #6 Ботяра, Jul 17, 2016
  7. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    С гугла тоже работает
    Code:
    User-agent: *
Disallow: /search
Allow: /search/about
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /preferences
Disallow: /setprefs
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Disallow: /s?
Allow: /maps?*output=classic*
Allow: /maps?*file=
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview
Disallow: /mld?
Disallow: /staticmap?
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /extern_js/
Disallow: /xjs/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /kh
Disallow: /vt
Disallow: /maps/vt
Disallow: /maps/api/js/AuthenticationService.Authenticate
Disallow: /maps/api/js/QuotaService.RecordEvent
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /ads/hotels/partners
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /flights/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/reviewer
Disallow: /about/careers/apply/
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Disallow: /webmasters/sitemaps/ping?
Disallow: /ping?
Disallow: /gallery/
Disallow: /landing/now/ontap/

# Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact [email protected].
User-agent: Twitterbot
Allow: /imgres

User-agent: facebookexternalhit
Allow: /imgres

Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: https://www.google.com/sitemap.xml
     
    #7 Karabas Barabas, Jul 17, 2016
    • Нравится Нравится x 1
  8. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    За то майл всех троллит :D
    Code:
    User-Agent: Yandex
Allow: /$
Allow: /all$
Disallow: /
Host: https://mail.ru

User-agent: Twitterbot
Disallow: /
Allow: /?logo=

User-Agent: *
Allow: /$
Allow: /all$
Disallow: /
     
    #8 Karabas Barabas, Jul 17, 2016
    • Нравится Нравится x 1
  9. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    @Karabas Barabas, ну вот с них и брать... к чему темы плодить какой правильный роботс делать... нихуяси и все думают что это относится именно только к их сайту нихуясинихуясинихуяси
     
    #9 Ботяра, Jul 17, 2016
  10. TopicStarter Overlay
    Webster

    Webster
    PRO Users

    Joined:
    Oct 25, 2015
    Messages:
    730
    Best Answers:
    3
    Ratings:
    +163 / 3 / -0
    Code:
    User-agent: *
Allow: /engine/classes/min/index.php?
Allow: /engine/data/emoticons/
Disallow: /engine/
Host: www.,,,.com
Sitemap: http://,,,/sitemap.xml
    В общем так оставил
     
    #10 Webster, Jul 17, 2016
  11. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    @Karabas Barabas, пошёл все поисковики ковырять на ротобсы? :eek:
     
    #11 Ботяра, Jul 17, 2016
  12. Olegjan

    Olegjan
    Staff Member VIP Кинотрафик v2

    Joined:
    Jan 31, 2015
    Messages:
    2,266
    Best Answers:
    0
    Ratings:
    +826 / 47 / -0
    А где хост, где карта сайта? Понизить их в выдаче! :D
     
    #12 Olegjan, Jul 17, 2016
    • Нравится Нравится x 1
  13. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    зато 4 карты сайта есть :eek: и только одна с того домена с которого роботс
    Code:
    Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: https://www.google.com/sitemap.xml
     
    #13 Ботяра, Jul 17, 2016
  14. TopicStarter Overlay
    Webster

    Webster
    PRO Users

    Joined:
    Oct 25, 2015
    Messages:
    730
    Best Answers:
    3
    Ratings:
    +163 / 3 / -0
    Ребят, офф флуд!!
     
    #14 Webster, Jul 17, 2016
  15. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    Да не, решил проверить, кто из них троллит лучше своим роботсом :D
     
    #15 Karabas Barabas, Jul 17, 2016
  16. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    Во всяком флуде, рождается истина :p
     
    #16 Karabas Barabas, Jul 17, 2016
    • Нравится Нравится x 1
  17. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    http://www.bind.com/robots.txt
    Code:
    User-agent: *
Disallow:

User-agent: Googlebot
Disallow:
     
    #17 Ботяра, Jul 17, 2016
  18. Karabas Barabas

    Karabas Barabas
    Staff Member VIP Кинотрафик v2

    Joined:
    Nov 20, 2015
    Messages:
    8,150
    Best Answers:
    1
    Ratings:
    +3,748 / 12 / -0
    #18 Karabas Barabas, Jul 17, 2016
  19. Ботяра

    Ботяра
    Забанен

    Joined:
    Oct 10, 2015
    Messages:
    0
    Best Answers:
    54
    Ratings:
    +711 / 74 / -0
    :rolleyes: но все же ))
     
    #19 Ботяра, Jul 17, 2016
  20. Demx

    Demx Бывалый

    Joined:
    May 13, 2016
    Messages:
    117
    Best Answers:
    0
    Ratings:
    +15 / 2 / -0
    Привет всем!У кого как настроен robots.txt?

    У меня стоит так,брал с форума не помню с какой темы

    Code:
    User-agent: *
Disallow: /engine/*
Disallow: */page/*/
Disallow: */page/
Disallow: /index.php?do=*
Allow: /engine/classes/js/
Allow: /engine/classes/min/index.php
Allow: /engine/data/emoticons/

Host: site.ru
Sitemap: http://site.ru/sitemap.xml
     
    #20 Demx, Sep 1, 2017
(You must log in or sign up to post here.)
Page 1 of 2
Similar Threads
  1. Расим

    Robots.txt или как правильно?

    Расим, Oct 21, 2015, in forum: Свободное общение
    Replies:
    28
    Views:
    5,459
    Doktormgdn
    Dec 12, 2015
  2. Wizyn4ik

    Кто может настроить robots.txt и htaccess на dle?

    Wizyn4ik, Dec 14, 2015, in forum: Заказчики - продвижение сайтов
    Replies:
    10
    Views:
    1,720
    MIKE_FM
    Feb 28, 2016
  3. Superwanget

    Помогите настроить файл robots.txt

    Superwanget, Feb 21, 2017, in forum: Аудиты сайтов - оценка функционала, юзабилити
    Replies:
    2
    Views:
    917
    Karabas Barabas
    Feb 21, 2017
  4. Lazyexpert225

    Как правильно настроить модуль алохи

    Lazyexpert225, Jan 30, 2026, in forum: Модули DLE
    Replies:
    3
    Views:
    626
    daimon4
    Feb 5, 2026
Loading...
Toggle Width
Style
Zerocool-New
Language
English (US)
Яндекс.Метрика