{"id":888,"date":"2023-03-21T08:58:22","date_gmt":"2023-03-21T08:58:22","guid":{"rendered":"https:\/\/www.devopstrainer.in\/blog\/?p=888"},"modified":"2023-03-21T09:03:26","modified_gmt":"2023-03-21T09:03:26","slug":"why-do-we-need-robots-txt","status":"publish","type":"post","link":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/","title":{"rendered":"Why do we need robots.txt?"},"content":{"rendered":"
\n
\"\"
Robots.txt File<\/em><\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

Before starting, I will give you a brief idea about robots.txt. You know it is a search engine crawler that merely prevents the web page or content of the website according to your allowances. The website page which you will allow or disallow then the search engine bots will show or hide according to the conditions you have set in your robots.txt file. so, the prominent question is why we need it.<\/p>\n\n\n\n

Suppose, you have a lot of secret information, credentials, site passwords, or account details that you want to hide from all the bot crawlers. I know it’s not possible but at certain limits, you can allow or disallow your website pages or content of the site. you will set the robots.txt file according to your own format and get the command over your own website pages which you want to hide or show limited sources of content to the visitors or users.<\/p>\n\n\n

\n
\"\"
Importance of Robots.txt<\/em><\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

The other need is to if you want to protect your site from overloading issues with your requests and there is no different mechanism that your site will be out from the google search engine but what content, column, paragraph, and credentials of your website do you want to be out from the Google so, you may easily possible with the robots.txt file.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Part-2 Referral link: How-robots-txt-works <\/a><\/strong><\/h2>\n","protected":false},"excerpt":{"rendered":"

Before starting, I will give you a brief idea about robots.txt. You know it is a search engine crawler that merely prevents the web page or content… <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-888","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nWhy do we need robots.txt? - DevOps | SRE | DevSecOps<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why do we need robots.txt? - DevOps | SRE | DevSecOps\" \/>\n<meta property=\"og:description\" content=\"Before starting, I will give you a brief idea about robots.txt. You know it is a search engine crawler that merely prevents the web page or content...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"DevOps | SRE | DevSecOps\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-21T08:58:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-21T09:03:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10-1024x478.png\" \/>\n<meta name=\"author\" content=\"rahulkr kr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"rahulkr kr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/\"},\"author\":{\"name\":\"rahulkr kr\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\"},\"headline\":\"Why do we need robots.txt?\",\"datePublished\":\"2023-03-21T08:58:22+00:00\",\"dateModified\":\"2023-03-21T09:03:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/\"},\"wordCount\":239,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-10-1024x478.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/\",\"name\":\"Why do we need robots.txt? - DevOps | SRE | DevSecOps\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-10-1024x478.png\",\"datePublished\":\"2023-03-21T08:58:22+00:00\",\"dateModified\":\"2023-03-21T09:03:26+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-10.png\",\"contentUrl\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/03\\\/image-10.png\",\"width\":1200,\"height\":560},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/why-do-we-need-robots-txt\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Why do we need robots.txt?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/\",\"name\":\"DevOps | SRE | DevSecOps\",\"description\":\"Automation means Cost, Quality, Time\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/#\\\/schema\\\/person\\\/e9fd1d88de76754aa189257cd197394d\",\"name\":\"rahulkr kr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g\",\"caption\":\"rahulkr kr\"},\"url\":\"https:\\\/\\\/www.devopstrainer.in\\\/blog\\\/author\\\/rahulkr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Why do we need robots.txt? - DevOps | SRE | DevSecOps","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Why do we need robots.txt? - DevOps | SRE | DevSecOps","og_description":"Before starting, I will give you a brief idea about robots.txt. You know it is a search engine crawler that merely prevents the web page or content...","og_url":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/","og_site_name":"DevOps | SRE | DevSecOps","article_published_time":"2023-03-21T08:58:22+00:00","article_modified_time":"2023-03-21T09:03:26+00:00","og_image":[{"url":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10-1024x478.png","type":"","width":"","height":""}],"author":"rahulkr kr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"rahulkr kr","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#article","isPartOf":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/"},"author":{"name":"rahulkr kr","@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d"},"headline":"Why do we need robots.txt?","datePublished":"2023-03-21T08:58:22+00:00","dateModified":"2023-03-21T09:03:26+00:00","mainEntityOfPage":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/"},"wordCount":239,"commentCount":0,"image":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10-1024x478.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/","url":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/","name":"Why do we need robots.txt? - DevOps | SRE | DevSecOps","isPartOf":{"@id":"https:\/\/www.devopstrainer.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#primaryimage"},"image":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10-1024x478.png","datePublished":"2023-03-21T08:58:22+00:00","dateModified":"2023-03-21T09:03:26+00:00","author":{"@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d"},"breadcrumb":{"@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#primaryimage","url":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10.png","contentUrl":"https:\/\/www.devopstrainer.in\/blog\/wp-content\/uploads\/2023\/03\/image-10.png","width":1200,"height":560},{"@type":"BreadcrumbList","@id":"https:\/\/www.devopstrainer.in\/blog\/why-do-we-need-robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.devopstrainer.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Why do we need robots.txt?"}]},{"@type":"WebSite","@id":"https:\/\/www.devopstrainer.in\/blog\/#website","url":"https:\/\/www.devopstrainer.in\/blog\/","name":"DevOps | SRE | DevSecOps","description":"Automation means Cost, Quality, Time","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.devopstrainer.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.devopstrainer.in\/blog\/#\/schema\/person\/e9fd1d88de76754aa189257cd197394d","name":"rahulkr kr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/79531cbaf0b831cebc9631f6ac28f21fb3fb0dc2615eeecdeeec43038b513473?s=96&d=mm&r=g","caption":"rahulkr kr"},"url":"https:\/\/www.devopstrainer.in\/blog\/author\/rahulkr\/"}]}},"_links":{"self":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/888","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/comments?post=888"}],"version-history":[{"count":3,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/888\/revisions"}],"predecessor-version":[{"id":893,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/posts\/888\/revisions\/893"}],"wp:attachment":[{"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/media?parent=888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/categories?post=888"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopstrainer.in\/blog\/wp-json\/wp\/v2\/tags?post=888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}