답변 1개
채택된 답변
+20 포인트
6개월 전
차단하면 나중에 구글에 검색이 잘 안 될 수도.....
robots.txt 이용해 크롤링 주기를 조정해 보세요.
https://www.google.com/search?q=robots.txt+%EC%84%A4%EC%A0%95%EB%B2%95
로그인 후 평가할 수 있습니다
답변에 대한 댓글 5개
�
맛동산파티
6개월 전
�
맛동산파티
6개월 전
신비님 혹시, 아래처럼 설정하면 될까요?
User-agent: googlebot
Disallow:
Crawl-delay: 20
User-agent: googlebot-image
Disallow:
Crawl-delay: 20
User-agent: googlebot-mobile
Disallow:
Crawl-delay: 20
User-agent: googlebot
Disallow:
Crawl-delay: 20
User-agent: googlebot-image
Disallow:
Crawl-delay: 20
User-agent: googlebot-mobile
Disallow:
Crawl-delay: 20
s
5개월 전
Disallow: 부분은 빼셔야 할 것 같아요.
옛날 구글링해서 저는 아래처럼 설정해 사용 중입니다.
# 허용 X
User-agent: AI-Bot-Name
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: archive.org_bot
User-agent: Bagoodex
User-agent: Bing AI
User-agent: BDCbot
User-agent: Brave
User-agent: BusinessBot
User-agent: CCBot
User-agent: Character.AI
User-agent: ChatGPT-User
User-agent: Claude
User-agent: CLOVA
User-agent: Codium
User-agent: Copilot
User-agent: CopyAI
User-agent: Dataprovider.com
User-agent: DeepL
User-agent: DialoGPT
User-agent: DialogSearch.com Bot
User-agent: Digincore crawler bot
User-agent: DomainSigmaCrawler
User-agent: ELSA
User-agent: Gemini
User-agent: Google-Extended
User-agent: Googlebot-Image
User-agent: GPTBot
User-agent: Hix
User-agent: iAsk
User-agent: Jasper
User-agent: KakaoBot
User-agent: kisaBot
User-agent: Komo
User-agent: Lilys
User-agent: Mail.RU_Bot
User-agent: MediavBot
User-agent: memoryBot
User-agent: mfibot
User-agent: msnbot-media
User-agent: NeumobBot
User-agent: Otter
User-agent: PageAnalyzer
User-agent: PagesInventory
User-agent: Perplexity
User-agent: Plukkie
User-agent: Quora
User-agent: roboto
User-agent: rogerbot
User-agent: RSNBOT
User-agent: SafeDNSBot
User-agent: semanticbot
User-agent: Sidetrade Indexer Bot
User-agent: SMTBot
User-agent: Socratic
User-agent: SurveyBot
User-agent: TabNine
User-agent: ThinkAny
User-agent: TurnitinBot
User-agent: UXCrawlerBot
User-agent: Vaiv
User-agent: Wotbox
User-agent: Writesonic
User-agent: YouChat
Disallow: /
# 허용 △
User-agent: AdsBot-Naver
User-agent: msnbot
User-agent: bingbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: DomainCrawler
User-agent: MegaIndex.ru
User-agent: AlphaBot
User-agent: ltx71
User-agent: CCBot
User-agent: Sogou
User-agent: DotBot
User-agent: PiplBot
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: MauiBot
User-agent: AhrefsBot
User-agent: ezooms
User-agent: sistrix
User-agent: Yandex
User-agent: ZumBot
Crawl-delay: 30
# 허용 O
User-agent: Daum
User-agent: Daumoa
User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Yeti
Allow: /
옛날 구글링해서 저는 아래처럼 설정해 사용 중입니다.
# 허용 X
User-agent: AI-Bot-Name
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: archive.org_bot
User-agent: Bagoodex
User-agent: Bing AI
User-agent: BDCbot
User-agent: Brave
User-agent: BusinessBot
User-agent: CCBot
User-agent: Character.AI
User-agent: ChatGPT-User
User-agent: Claude
User-agent: CLOVA
User-agent: Codium
User-agent: Copilot
User-agent: CopyAI
User-agent: Dataprovider.com
User-agent: DeepL
User-agent: DialoGPT
User-agent: DialogSearch.com Bot
User-agent: Digincore crawler bot
User-agent: DomainSigmaCrawler
User-agent: ELSA
User-agent: Gemini
User-agent: Google-Extended
User-agent: Googlebot-Image
User-agent: GPTBot
User-agent: Hix
User-agent: iAsk
User-agent: Jasper
User-agent: KakaoBot
User-agent: kisaBot
User-agent: Komo
User-agent: Lilys
User-agent: Mail.RU_Bot
User-agent: MediavBot
User-agent: memoryBot
User-agent: mfibot
User-agent: msnbot-media
User-agent: NeumobBot
User-agent: Otter
User-agent: PageAnalyzer
User-agent: PagesInventory
User-agent: Perplexity
User-agent: Plukkie
User-agent: Quora
User-agent: roboto
User-agent: rogerbot
User-agent: RSNBOT
User-agent: SafeDNSBot
User-agent: semanticbot
User-agent: Sidetrade Indexer Bot
User-agent: SMTBot
User-agent: Socratic
User-agent: SurveyBot
User-agent: TabNine
User-agent: ThinkAny
User-agent: TurnitinBot
User-agent: UXCrawlerBot
User-agent: Vaiv
User-agent: Wotbox
User-agent: Writesonic
User-agent: YouChat
Disallow: /
# 허용 △
User-agent: AdsBot-Naver
User-agent: msnbot
User-agent: bingbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: DomainCrawler
User-agent: MegaIndex.ru
User-agent: AlphaBot
User-agent: ltx71
User-agent: CCBot
User-agent: Sogou
User-agent: DotBot
User-agent: PiplBot
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: MauiBot
User-agent: AhrefsBot
User-agent: ezooms
User-agent: sistrix
User-agent: Yandex
User-agent: ZumBot
Crawl-delay: 30
# 허용 O
User-agent: Daum
User-agent: Daumoa
User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Yeti
Allow: /
댓글을 작성하려면 로그인이 필요합니다.
답변을 작성하려면 로그인이 필요합니다.
로그인전체 질문 목록
답변대기
답변대기
답변대기
답변대기
답변대기
답변대기
답변대기
답변대기
채택
답변대기
답변대기
답변대기
답변대기
채택
채택
답변대기
답변대기
답변대기
채택
바로 풀어야겠네요..ㄷㄷ
감사합니다!