You can check if your server is exposing directory listings by navigating to a directory that does not have an index file (e.g., ://yourwebsite.com ).
Launching aggressive scraping tools like wget against a small, privately owned server can exhaust its bandwidth or crash the server. Responsible researchers always throttle their download speeds. index of parent directory exclusive
Keep Options -Indexes (Apache) or autoindex off; (Nginx) as your global default. Only enable indexing explicitly in folders meant for public downloads. You can check if your server is exposing
To turn indexing off globally or for a specific parent location, set autoindex to off : location /secret-parent/ autoindex off; Use code with caution. Restricting Upward Access index of parent directory exclusive