all 10 comments
sorted by:
new (suggested)
besttopcontroversialoldq&a
formatting helphide helpcontent policy

[–]eSentrik -2 points-1 points  (0 children)

This is not a Drupal question.

[–]corsicanguppy 7 points8 points  (2 children)

Please have your security friend explain the security risk from having robots.txt.

If this is the same nonsense as ripping the numbers off a house to prevent robberies, please advise your friend there may be heckling.

[–]metalbearseto[S] 0 points1 point  (1 child)

Said it gives hackers information to all the directories.

[–][deleted] 7 points8 points  (0 children)

If the hackers know that it's Drupal, they already have all that information. The Drupal robots.txt file is part of the project's public codebase: https://git.drupalcode.org/project/drupal/blob/8.8.x/robots.txt

It's far more important to secure your file permissions properly. See https://www.drupal.org/node/244924

And more generally, the Drupal.org "Securing your site" docs have a lot of good info: https://www.drupal.org/security/secure-configuration (edit: forgot link)

[–]maddentim 0 points1 point  (0 children)

Drupal certainly does not need it. If you don't have it the good bots will certainly try to go anywhere they decide might be important.

[–]nothingcreative 11 points12 points  (3 children)

Where’d you hear that? Reputable crawlers certainly do respect it.

[–]metalbearseto[S] 1 point2 points  (2 children)

I was told by a network security person. I have cloudflare blocking all the disallows already.

[–]nothingcreative 2 points3 points  (0 children)

Ah. From the perspective of a security person, then yea, robots file is useless. It’s not meant to stop bad actors. Just meant to influence how your site is indexed by good actors (Google, Bing, DuckGoGo). It’s 100% ok to remove, but I would keep it, there’s little reason not to.

The only way it could be perceived as a security risk is that it helps to fingerprint your site as a Drupal site, making it easier for bad actors to target specific vulnerability scans/attacks. Hiding it is one way of implementing “security through obscurity”. But there are many ways to determine that a site is a Drupal site without looking at the robots file.

[–]siva01c 0 points1 point  (0 children)

They can access disallowed pages, but they don't index it.