Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a simple statement, I want to describe that which page of your website content you want to allow or disallow, it means which page you want to show or hide but according to search engine crawlers, they shall crawl the
every part of the website or domain.
You know how many credentials you have scattered over the different websites related to bank details, and secret credentials of bills & files but according to robots exclusion protocol(REP), you may allow which section, page, or content of the website you want to allow or disallow it.
Robot.txt Format:
User-agent: *
Disallow: /our-placed-students
Disallow: /cgi-bin/
Disallow: /VIET-webworkshop/
Disallow: /MenClothing
Allow: /MenClothing/tshirts
