Article Directory
URL filtering technology
URL filtering principle
URl filtering technology performs access control on users' URLs, allowing or prohibiting users from accessing certain network resources, which can achieve the purpose of regulating Internet access. For HTTP packets with URRl classification, the NGFW can modify the DSCP (Differentiated Services Code Point) in the packets, which is the basis for classifying network equipment traffic.
The URL filtering function only supports filtering URL requests of the HTTP protocol and HTTPS protocol.
When you need to filter the URL requests of the HTTPS protocol, you also need to configure the SSL decryption function. The NGFW performs URL filtering on the decrypted HTTP traffic.
URL address structure
URL (Uniform Resource Locator)
URL一般格式：protocol://hostname[:port]/path[?query]
protocol: the application protocol used, such as http, https
hostname: DNS host name or IP address of the WEB server
: Port: optional, communication port. Various application protocols have default port numbers. For example, the default port number for HTTP protocol is 80, and the default port number for HTTPS protocol is 443. When the web server uses a non-default port, the port number cannot be omitted from the URL.
? query: Optional, used to pass parameters to the dynamic page number.
URL matching method
- Prefix match: match all URLs beginning with the specified string
- Suffix match: match all URLs ending with the specified string
- Keyword matching: match all URLs that contain the specified string
- Exact match: Firstly, judge whether the URL and the specified string match. If it does not match, remove the last directory of the URL and match the specified string again. ...
URL entries are not case sensitive
URL filtering method
Black and white list
The NGFW matches the parsed URL address with the black and white list. If it matches the white list, the URL request is allowed; if it matches the black list, the URL request is blocked, and the Web push page is displayed at the same time. When the URL of the online request matches the whitelist, subsequent matching processing will not be performed on the online request.
Setting a whitelist is helpful to improve matching efficiency.
URL custom classification
After the device extracts the URL information, it will prioritize the query of the custom classification. If it matches a custom category, it will be processed according to the response action configured in the URL filtering configuration file. When the control action is blocking, the NGFW will block the URL request and display the web push page at the same time.
URL predefined category query
- Local cache query: When the device is powered on for the first time, the URL classification preset library has been loaded into the cache. After the device retrieves the URL information, it will first query the category corresponding to the URL in the cache. If the URL category is queried, it will be processed according to the response action configured in the URL filtering configuration file. When the control action is blocking, the NGFW will block the URL request and display the web push page at the same time. If it can't be found, it will continue to search on the remote classification server.
- Remote classification server query: The server is deployed on the WAN to provide a larger URL classification information. When the category corresponding to the URL cannot be queried in the local cache, the device sends the URL to the remote query server to continue the query. If the category corresponding to the URL is queried, it will be processed according to the response action configured in the URL filtering configuration file, and the URL and its corresponding category information will be saved in the local cache for quick query next time. When the control action is blocking, the NGFW will block the URL request and display the web push page at the same time. If the query is not found, it will be processed according to the response action classified as "other".
URL filtering control actions
Allowed: Refers to allowing the user to access the requested URL.
Alarm: Refers to the URL that allows the user to access the request and records the log at the same time.
Blocking: Refers to blocking the user's access to the requested URL and recording the log at the same time.
URL filtering process
1. The user initiates a URL access request. If the data stream matches the security policy, and the action of the security policy is allowed, the URL filtering process is performed.
2. NGFW matches the URL information with the whitelist.
- If it matches the whitelist, the request is allowed to pass.
- If it does not match the whitelist, proceed to the next step of detection.
3. NGFW matches the URl information with the blacklist
- If it matches the black block list, the request is allowed to pass.
- If the blacklist is not matched, proceed to the next step of detection.
4. NGFW matches the URL information with the custom classification.
- If it matches a custom category, the request is processed according to the control action of the custom URL category.
Note: The URL added by the administrator to the predefined category belongs to the URL of the custom category.
- If it does not match the custom classification, proceed to the next step of detection.
5. The NGFW matches the URL information with the predefined categories in the local cache.
- If the corresponding category is queried in the local cache, the request will be processed according to the control action of the category
- If the corresponding category is not found in the local cache, the remote server category query is performed.
- If the remote server is available, proceed with the remote server classification query.
- If the remote server is unavailable, the request is processed according to the default action.
6. Start remote server classification query
- If the remote server category query times out, it will be processed according to the pre-defined category query timeout action configured by the administrator.
- If the URL classification server clearly finds that the URL belongs to a certain category of the predefined category, it will be processed according to the control action of the category.
URL filtering configuration roadmap
- Configure the URL custom category.
- Configure black and white lists
- Configure URL filtering profile
- Configure security policy
- Submit for compilation
URL troubleshooting ideas
1. The rules configured under the policy have a priority relationship
2. The user/group has no security policy applied or applied error
3. The traffic matches the higher priority security policy rules
4. The URL filtering configuration file is not applied in the security policy
5. The modified URL filtering configuration file is not submitted for compilation
When URL filtering is matched, follow: whitelist>blacklist>custom classification>predefined classification