URL Filtering Concepts
The following topics describe the URL filtering components and how they are used on a Palo Alto Networks firewall:
URL Categories
Each website defined in the URL filtering database is assigned one of approximately 60 different categories. These categories can then be used in a URL filtering profile to block or allow access based on category, or you can configure the firewall to use a category as a match criteria in policy. For example, to block all gaming websites, in the URL filtering profile you would set the block action for the URL category games. As an example of using a URL category as a match criteria in a policy, you could use the URL category streaming-media in a QoS policy to apply bandwidth controls to all websites that are categorized as streaming media.
By grouping websites into categories, it makes it easy to define actions based on certain types of websites. In addition to the standard URL categories, there are three additional categories:
Not-resolvedIndicates that the website was not found in the local URL filtering database and the firewall was unable to connect to the cloud database to check the category. When a URL category lookup is performed, the firewall first checks the dataplane cache for the URL, if no match is found, it will then check the management plane cache, and if no match is found there, it queries the URL database in the cloud.
When deciding on what action to take for traffic that is categorized as not-resolved, be aware that setting the action to block may be very disruptive to users.
For more information on troubleshooting lookup issues, see Troubleshoot URL Filtering.
Private-ip-addressesIndicates that the website is a single domain (no sub-domains), the IP address is in the private IP range, or the URL root domain is unknown to the cloud. UnknownThe website has not yet been categorized, so it does not exist in the URL filtering database on the firewall or in the URL cloud database.
When deciding on what action to take for traffic categorized as unknown, be aware that setting the action to block may be very disruptive to users because there could be a lot of valid sites that are not in the URL database yet. If you do want a very strict policy, you could block this category, so websites that do not exist in the URL database cannot be accessed.
URL Filtering Profiles
A URL filtering profile is a collection of URL filtering controls that are applied to individual security policies to enforce your web access policy. The firewall comes with a default profile that is configured to block websites such as known malware sites, phishing sites, and adult content sites. You can use the default profile in a security policy, clone it to be used as a starting point for new URL filtering profiles, or add a new URL profile that will have all categories set to allow for visibility into the traffic on your network. You can then customize the newly added URL profiles and add lists of specific websites that should always be blocked or allowed, which provides more granular control over URL categories. For example, you may want to block social-networking sites, but allow some websites that are part of the social-networking category.
The section describes how URL filtering profiles are applied and the various options that can be defined:
URL Filtering Actions
Each URL filtering category can be set to perform the following actions:
Action Description
Alert The website is allowed and a log entry is generated in the URL filtering log.
Allow The website is allowed and no log entry is generated.
Block The website is blocked and the user will see a response page and will not be able to continue to the website. A log entry is generated in the URL filtering log.
Continue The user will be prompted with a response page indicating that the site has been blocked due to company policy, but the user is prompted with the option to continue to the website. The continue action is typically used for categories that are considered benign and is used to improve the user experience by giving them the option to continue if they feel the site is incorrectly categorized. The response page message can be customized to contain details specific to your company. A log entry is generated in the URL filtering log. The Continue page will not be displayed properly on client machines that are configured to use a proxy server.
Override The user will see a response page indicating that a password is required to allow access to websites in the given category. With this option, the security admin or helpdesk person would provide a password that will grant temporary access to all websites in the given category. A log entry is generated in the URL filtering log. The Override page will not be displayed properly on client machines that are configured to use a proxy server.
None The None action only applies to custom URL categories. The purpose of selecting None is to ensure that if multiple URL profiles exist, the custom category will not have any impact on other profiles. For example, if you have two URL profiles and the custom URL category is set to block in one of the profiles, the other profile should have the action set to None if you do not want it to apply. Also, in order to delete a custom URL category, it must be set to none in any profile where it is used.
Block and Allow Lists
Block and allow lists allow you to define specific URLs or IP addresses in the URL filtering profile that are always allowed or always blocked, regardless of the action defined for the URL category. When entering URLs in the Block List or Allow List , enter each URL or IP address in a new row separated by a new line. When using wildcards in the URLs, follow these rules:
Do not include HTTP and HTTPS when defining URLs. For example, enter www.paloaltonetworks.com or paloaltonetworks.com instead of https://www.paloaltonetworks.com. Entries in the block list must be an exact match and are case-insensitive.
For example: If you want to prevent a user from accessing any website within the domain paloaltonetworks.com, you would also add *.paloaltonetworks.com, so whatever domain prefix (http://, www, or a sub-domain prefix such as mail.paloaltonetworks.com) is added to the address, the specified action will be taken. The same applies to the sub-domain suffix; if you want to block paloaltonetworks.com/en/US, you would need to add paloaltonetworks.com/* as well. Block and allow lists support wildcard patterns. The following characters are considered separators:
.
/
?
&
=
;
+
Every substring that is separated by the characters listed above is considered a token. A token can be any number of ASCII characters that does not contain any separator character or *. For example, the following patterns are valid:
*.yahoo.com (tokens are: "*", "yahoo" and "com")
www.*.com (tokens are: "www", "*" and "com")
www.yahoo.com/search=* (tokens are: "www", "yahoo", "com", "search", "*")
The following patterns are invalid because the character “*” is not the only character in the token.
ww*.yahoo.com and www.y*.com
Safe Search Enforcement
Many search engines have a safe search setting that filters out adult images and videos in search query return traffic. On the firewall, you can Enable Safe Search Enforcement so that the firewall will block search results if the end user is not using the strictest safe search settings in the search query. The firewall can enforce safe search for the following search providers: Google, Yahoo, Bing, Yandex, and YouTube. This is a best-effort setting and is not guaranteed by the search providers to work with every website.
To use this feature you must enable the Safe Search Enforcement option in a URL filtering profile and attach it to a security policy. The firewall will then block any matching search query return traffic that is not using the strictest safe search settings. There are two methods for blocking the search results:
Block Search Results that are not Using Strict Safe Search Settings —When an end user attempts to perform a search without first enabling the strictest safe search settings, the firewall blocks the search query results and displays the URL Filtering Safe Search Block Page. By default, this page will provide a URL to the search provider settings for configuring safe search. Enable Transparent Safe Search Enforcement —When an end user attempts to perform a search without first enabling the strict safe search settings, the firewall blocks the search results with an HTTP 503 status code and redirects the search query to a URL that includes the safe search parameters. You enable this functionality by importing a new URL Filtering Safe Search Block Page containing the Javascript for rewriting the search URL to include the strict safe search parameters. In this configuration, users will not see the block page, but will instead be automatically redirected to a search query that enforces the strictest safe search options. This safe search enforcement method requires Content Release version 475 or later and is only supported for Google, Yahoo, and Bing searches.
Also, because most search providers now use SSL to return search results, you must also configure a Decryption policy for the search traffic to enable the firewall to inspect the search traffic and enforce safe search.
Safe search enforcement enhancements and support for new search providers is periodically added in content releases. This information is detailed in the Application and Threat Content Release Notes. How sites are judged to be safe or unsafe is performed by each search provider, not by Palo Alto Networks.
Safe search settings differ by search provider as detailed in Table: Search Provider Safe Search Settings.
Table: Search Provider Safe Search Settings
Search Provider Safe Search Setting Description
Google/YouTube Offers safe search on individual computers or network-wide through Google’s safe search virtual IP address: Safe Search Enforcement for Google Searches on Individual Computers In the Google Search Settings, the Filter explicit results setting enables safe search functionality. When enabled, the setting is stored in a browser cookie as FF= and passed to the server each time the user performs a Google search. Appending safe=active to a Google search query URL also enables the strictest safe search settings. Safe Search Enforcement for Google and YouTube Searches using a Virtual IP Address Google provides servers that Lock SafeSearch (forcesafesearch.google.com) settings in every Google and YouTube search. By adding a DNS entry for www.google.com and www.youtube.com (and other relevant Google and YouTube country subdomains) that includes a CNAME record pointing to forcesafesearch.google.com to your DNS server configuration, you can ensure that all users on your network are using strict safe search settings every time they perform a Google or YouTube search. Keep in mind, however, that this solution is not compatible with Safe Search Enforcement on the firewall. Therefore, if you are using this option to force safe search on Google, the best practice is to block access to other search engines on the firewall by creating custom URL categories and adding them to the block list in the URL filtering profile. If you plan to use the Google Lock SafeSearch solution, consider configuring DNS Proxy ( Network > DNS Proxy) and setting the inheritance source as the Layer 3 interface on which the firewall receives DNS settings from service provider via DHCP. You would configure the DNS proxy with Static Entries for www.google.com and www.youtube.com, using the local IP address for the forcesafesearch.google.com server.
Yahoo Offers safe search on individual computers only. The Yahoo Search Preferences includes three SafeSearch settings: Strict, Moderate, or Off. When enabled, the setting is stored in a browser cookie as vm= and passed to the server each time the user performs a Yahoo search. Appending vm=r to a Yahoo search query URL also enables the strictest safe search settings. When performing a search on Yahoo Japan (yahoo.co.jp) while logged into a Yahoo account, end users must also enable the SafeSearch Lock option.
Bing Offers safe search on individual computers or through their Bing in the Classroom program. The Bing Settings include three SafeSearch settings: Strict, Moderate, or Off. When enabled, the setting is stored in a browser cookie as adlt= and passed to the server each time the user performs a Bing search. Appending adlt=strict to a Bing search query URL also enables the strictest safe search settings. The Bing SSL search engine does not enforce the safe search URL parameters and you should therefore should consider blocking Bing over SSL for full safe search enforcement.
Container Pages
A container page is the main page that a user accesses when visiting a website, but additional websites may be loaded within the main page. If the Log Container page only option is enabled in the URL filtering profile, only the main container page will be logged, not subsequent pages that may be loaded within the container page. Because URL filtering can potentially generate a lot of log entries, you may want to turn on this option, so log entries will only contain those URIs where the requested page file name matches the specific mime-types. The default set includes the following mime-types:
application/pdf application/soap+xml application/xhtml+xml text/html text/plain text/xml
If you have enabled the Log container page only option, there may not always be a correlated URL log entry for threats detected by Antivirus or Vulnerability Protection.
URL Category as Policy Match Criteria
URL categories can be used as a match criteria in a policy to provide more granularity in the policy. For example, you may have a decryption policy defined, but you would like specific websites to bypass decryption. To do this, you would configure a decryption policy with the no-decrypt action and a URL category would be defined as match criteria for the policy rule, so the policy would only match traffic flows to websites that are part of the specified category.
The following table describes the policy types that can utilize URL categories:
Policy Type Description
Captive Portal To ensure that users authenticate before being allowed access to a specific category, you can attach a URL category as a match criteria for the captive portal policy.
Decryption Decryption policies can use URL categories as a match criteria to determine if specified websites should be decrypted or not. For example, if you have a decryption policy with the action decrypt for all traffic between two zones, there may be specific website categories, such as financial-services and/or health-and-medicine, that should not be decrypted. In this case, you would create a new decryption policy with the action of no-decrypt that precedes the decrypt policy and then defines a list of URL categories as match criteria for the policy. By doing this, each URL category that is part of the no-decrypt policy will not be decrypted. You could also configure a custom URL category to define your own list of URLs that can then be used in the no decrypt policy.
QoS A QoS policy can use URL categories to determine throughput levels for specific website categories. For example, you may want to allow the streaming-media category, but limit throughput by adding the URL category as match criteria to the QoS policy.
Security In security policies you can use URL categories both as a match criteria in the Service/URL Category tab, and in URL filtering profiles that are attached in the Actions tab. If for example, the IT-security group in your company needs access to the hacking category, while all other users are denied access to the category, you must create the following rules: A security rule that allows the IT-Security group to access content categorized as hacking. The security rule references the hacking category in the S ervices/URL Category tab and IT-Security group in the Users tab. Another security rule that allows general web access for all users. To this rule you attach a URL filtering profile that blocks the hacking category. The policy that allows access to hacking must be listed before the policy that blocks hacking. This is because security policy rules are evaluated top down, so when a user who is part of the security group attempts to access a hacking site, the policy rule that allows access is evaluated first and will allow the user access to the hacking sites. Users from all other groups are evaluated against the general web access rule which blocks access to the hacking sites.

Related Documentation