URL inspector
Break a web URL into its protocol, host, path, query and fragment.
Open toolBack
Web & SEO
Robots.txt generator
Create a robots.txt file from allowed and disallowed path lists.
Free to useProcessed on your device
Open tool…
About this tool
Create a robots.txt file from allowed and disallowed path lists.
How to use
- Enter disallowed paths and allowed paths in the form.
- Choose the available options, then run robots.txt generator.
- Review the result and copy or download it for your work.
Robots.txt is a crawler preference, not an access-control mechanism.
Comment
Related tools
Relative URL resolver
Resolve relative paths against a base URL.
Open toolCampaign URL builder
Add UTM campaign parameters without losing existing query parameters.
Open toolInternational domain encoder
Convert a Unicode domain to its ASCII IDN form using the URL standard.
Open tool