{"id":875,"date":"2023-03-20T09:38:24","date_gmt":"2023-03-20T09:38:24","guid":{"rendered":"https:\/\/www.devopstrainer.in\/blog\/?p=875"},"modified":"2023-03-20T09:38:24","modified_gmt":"2023-03-20T09:38:24","slug":"what-is-robots-txt","status":"publish","type":"post","link":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/","title":{"rendered":"What is robots.txt?"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png\" alt=\"\" class=\"wp-image-877\" width=\"766\" height=\"383\" srcset=\"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png 1024w, https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-300x150.png 300w, https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-768x384.png 768w, https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1536x768.png 1536w, https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8.png 2048w\" sizes=\"auto, (max-width: 766px) 100vw, 766px\" \/><figcaption class=\"wp-element-caption\"><strong><em>Robot.txt File<\/em><\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n<p>Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a simple statement, I want to describe that which page of your website content you want to allow or disallow, it means which page you want to show or hide but according to search engine crawlers, they shall crawl the<br>every part of the website or domain.<\/p>\n\n\n\n<p>You know how many credentials you have scattered over the different websites related to bank details, and secret credentials of bills &amp; files but according to robots exclusion protocol(REP), you may allow which section, page, or content of the website you want to allow or disallow it.<\/p>\n\n\n\n<p><strong>Robot.txt Format:<\/strong><br>User-agent: *<br>Disallow: \/our-placed-students<br>Disallow: \/cgi-bin\/<br>Disallow: \/VIET-webworkshop\/<br>Disallow: \/MenClothing<br>Allow: \/MenClothing\/tshirts<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a&#8230; <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-875","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>What is robots.txt? - DevOps | SRE | DevSecOps<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is robots.txt? - DevOps | SRE | DevSecOps\" \/>\n<meta property=\"og:description\" content=\"Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"DevOps | SRE | DevSecOps\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-20T09:38:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png\" \/>\n<meta name=\"author\" content=\"rahulkr kr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"rahulkr kr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/\"},\"author\":{\"name\":\"rahulkr kr\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\"},\"headline\":\"What is robots.txt?\",\"datePublished\":\"2023-03-20T09:38:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/\"},\"wordCount\":141,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-8-1024x512.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/\",\"name\":\"What is robots.txt? - DevOps | SRE | DevSecOps\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-8-1024x512.png\",\"datePublished\":\"2023-03-20T09:38:24+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-8.png\",\"contentUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-8.png\",\"width\":2048,\"height\":1024},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/what-is-robots-txt\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is robots.txt?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/\",\"name\":\"DevOps | SRE | DevSecOps\",\"description\":\"Automation means Cost, Quality, Time\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\",\"name\":\"rahulkr kr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"caption\":\"rahulkr kr\"},\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/author\\\/rahulkr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is robots.txt? - DevOps | SRE | DevSecOps","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"What is robots.txt? - DevOps | SRE | DevSecOps","og_description":"Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a...","og_url":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/","og_site_name":"DevOps | SRE | DevSecOps","article_published_time":"2023-03-20T09:38:24+00:00","og_image":[{"url":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png","type":"","width":"","height":""}],"author":"rahulkr kr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"rahulkr kr","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#article","isPartOf":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/"},"author":{"name":"rahulkr kr","@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d"},"headline":"What is robots.txt?","datePublished":"2023-03-20T09:38:24+00:00","mainEntityOfPage":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/"},"wordCount":141,"commentCount":0,"image":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/","url":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/","name":"What is robots.txt? - DevOps | SRE | DevSecOps","isPartOf":{"@id":"https:\/\/www.devopstrainer.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#primaryimage"},"image":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8-1024x512.png","datePublished":"2023-03-20T09:38:24+00:00","author":{"@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d"},"breadcrumb":{"@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#primaryimage","url":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8.png","contentUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-8.png","width":2048,"height":1024},{"@type":"BreadcrumbList","@id":"https:\/\/www.devopstrainer.in\/blog\/what-is-robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.devopstrainer.in\/blog\/"},{"@type":"ListItem","position":2,"name":"What is robots.txt?"}]},{"@type":"WebSite","@id":"https:\/\/www.devopstrainer.in\/blog\/#website","url":"https:\/\/www.devopstrainer.in\/blog\/","name":"DevOps | SRE | DevSecOps","description":"Automation means Cost, Quality, Time","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.devopstrainer.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d","name":"rahulkr kr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","caption":"rahulkr kr"},"url":"https:\/\/www.devopstrainer.in\/blog\/author\/rahulkr\/"}]}},"_links":{"self":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/875","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/comments?post=875"}],"version-history":[{"count":2,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/875\/revisions"}],"predecessor-version":[{"id":878,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/875\/revisions\/878"}],"wp:attachment":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/media?parent=875"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/categories?post=875"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/tags?post=875"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}