Merge pull request #12 from PerimeterX/dev

Version 1.1.0

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [1.1.0] - 2020-09-02
+### Added
+- Support for `monitored_specific_routes`
+- Support for Regex patters in sensitive/whitelist/monitored/enforced routes
+
 ## [1.0.1] - 2019-11-10
 ### Fixed
 - Using hashlib pbkdf2 implementation.

README.md

@@ -1,11 +1,11 @@
 [![Build Status](https://travis-ci.org/PerimeterX/perimeterx-python-3-wsgi.svg?branch=master)](https://travis-ci.org/PerimeterX/perimeterx-python-3-wsgi)
 [![Known Vulnerabilities](https://snyk.io/test/github/PerimeterX/perimeterx-python-3-wsgi/badge.svg)](https://snyk.io/test/github/PerimeterX/perimeterx-python-3-wsgi)
 
-![image](https://s.perimeterx.net/logo.png)
+![image](https://storage.googleapis.com/perimeterx-logos/primary_logo_red_cropped.png)
 
 [PerimeterX](http://www.perimeterx.com) Python 3 Middleware
 =============================================================
-> Latest stable version: [v1.0.1](https://pypi.org/project/perimeterx-python-3-wsgi/)
+> Latest stable version: [v1.1.0](https://pypi.org/project/perimeterx-python-3-wsgi/)
 
 Table of Contents
 -----------------
@@ -20,15 +20,20 @@ Table of Contents
     * [Send Page Activities](#send_page_activities)
     * [Debug Mode](#debug_mode)
     * [Sensitive Routes](#sensitive_routes)
+    * [Sensitive Routes Regex](#sensitive_routes_regex)
     * [Whitelist Routes](#whitelist_routes)
+    * [Whitelist Routes Regex](#whitelist_routes_regex)
+    * [Enforce Specific Routes](#enforce_specific_routes)
+    * [Enforce Specific Routes Regex](#enforce_specific_routes_regex)
+    * [Monitor Specific Routes](#monitor_specific_routes)
+    * [Monitor Specific Routes Regex](#monitor_specific_routes_regex)
     * [Sensitive Headers](#sensitive_headers)
     * [IP Headers](#ip_headers)
     * [First-Party Enabled](#first_party_enabled)
     * [Custom Request Handler](#custom_request_handler)
     * [Additional Activity Handler](#additional_activity_handler)
     * [Px Disable Request](#px_disable_request)
     * [Test Block Flow on Monitoring Mode](#bypass_monitor_header)
-    * [Enforce Specific Routes](#enforce_specific_routes)
 
 ## <a name="installation"></a> Installation
 
@@ -59,10 +64,10 @@ px_config = {
 application = get_wsgi_application()
 application = PerimeterX(application, px_config)
 ```
-- The PerimeterX **Application ID** / **AppId** and PerimeterX **Token** / **Auth Token** can be found in the Portal, in [Applications](https://console.perimeterx.com/#/app/applicationsmgmt).
-- PerimeterX **Risk Cookie** / **Cookie Key** can be found in the portal, in [Policies](https://console.perimeterx.com/#/app/policiesmgmt).
+- The PerimeterX **Application ID** / **AppId** and PerimeterX **Token** / **Auth Token** can be found in the Portal, in [Applications](https://console.perimeterx.com/botDefender/admin?page=applicationsmgmt).
+- PerimeterX **Risk Cookie** / **Cookie Key** can be found in the portal, in [Policies](https://console.perimeterx.com/botDefender/admin?page=policiesmgmt).
 The Policy from where the **Risk Cookie** / **Cookie Key** is taken must correspond with the Application from where the **Application ID** / **AppId** and PerimeterX **Token** / **Auth Token**.
-For details on how to create a custom Captcha page, refer to the [documentation](https://console.perimeterx.com/docs/server_integration_new.html#custom-captcha-section)
+For details on how to create a custom Captcha page, refer to the [documentation](https://docs.perimeterx.com/pxconsole/docs/customize-challenge-page)
 
 ## <a name="configuration"></a>Optional Configuration
 In addition to the basic installation configuration [above](#required_config), the following configurations options are available:
@@ -152,6 +157,21 @@ config = {
   ...
 }
 ```
+
+#### <a name="sensitive_routes_regex"></a> Sensitive Routes Regex
+
+An array of regex patterns that trigger a server call to PerimeterX servers every time the page is viewed, regardless of viewing history.
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  sensitive_routes_regex: [r'^/login$', r'^/user']
+  ...
+}
+```
+
 #### <a name="whitelist_routes"></a> Whitelist Routes
 
 An array of route prefixes which will bypass enforcement (will never get scored).
@@ -166,6 +186,78 @@ config = {
 }
 ```
 
+#### <a name="whitelist_routes_regex"></a> Whitelist Routes Regex
+
+An array of regex patterns which will bypass enforcement (will never get scored).
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  whitelist_routes_regex: [r'^/about']
+  ...
+}
+```
+
+#### <a name="enforce_specific_routes"></a> Enforce Specific Routes
+
+An array of route prefixes that are always validated by the PerimeterX Worker (as opposed to whitelisted routes).
+When this property is set, any route which is not added - will be whitelisted.
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  enforced_specific_routes: ['/profile']
+  ...
+};
+```
+
+#### <a name="enforce_specific_routes_regex"></a> Enforce Specific Routes Regex
+
+An array of regex patterns that are always validated by the PerimeterX Worker (as opposed to whitelisted routes).
+When this property is set, any route which is not added - will be whitelisted.
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  enforced_specific_routes_regex: [r'^/profile$']
+  ...
+};
+```
+
+#### <a name="monitor_specific_routes"></a> Monitor Specific Routes
+
+An array of route prefixes that are always set to be in [monitor mode](#module_mode). This configuration is effective only when the module is enabled and in blocking mode.
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  monitored_specific_routes: ['/profile']
+  ...
+};
+```
+
+#### <a name="monitor_specific_routes_regex"></a> Monitor Specific Routes Regex
+
+An array of regex patterns that are always set to be in [monitor mode](#module_mode). This configuration is effective only when the module is enabled and in blocking mode.
+
+**Default:** Empty
+
+```python
+config = {
+  ...
+  monitored_specific_routes_regex: [r'^/profile/me$']
+  ...
+};
+```
+
 #### <a name="sensitive_headers"></a>Sensitive Headers
 
 An array of headers that are not sent to PerimeterX servers on API calls.
@@ -279,18 +371,4 @@ config = {
   bypass_monitor_header: 'x-px-block',
   ...
 }
-```
-
-#### <a name="enforce_specific_routes"></a> Enforced Specific Routes
-
-An array of route prefixes that are always validated by the PerimeterX Worker (as opposed to whitelisted routes).
-When this property is set, any route which is not added - will be whitelisted.
-
-**Default:** Empty
- ```python
-config = {
-  ...
-  enforced_specific_routes: ['/profile']
-  ...
-};
-```
+```

dev-requirements.txt

@@ -1,3 +1,3 @@
 mock==3.0.5
 requests_mock==1.7.0
-pylint==2.4.2
+pylint==2.5.0

perimeterx/middleware.py

@@ -32,7 +32,6 @@ def __init__(self, app, config=None):
         self._config = px_config
         self._request_verifier = PxRequestVerifier(px_config)
         px_activities_client.init_activities_configuration(px_config)
-        px_activities_client.send_enforcer_telemetry_activity(config=px_config, update_reason='initial_config')
 
     def __call__(self, environ, start_response):
         px_activities_client.send_activities_in_thread()
@@ -53,7 +52,7 @@ def __call__(self, environ, start_response):
             return verified_response(environ, pxhd_callback)
 
         except Exception as err:
-            self._config.logger.error("Caught exception, passing request1111. Exception: {}".format(err))
+            self._config.logger.error("Caught exception, passing request. Exception: {}".format(err))
             if context:
                 self.report_pass_traffic(context)
             else:
@@ -63,7 +62,7 @@ def __call__(self, environ, start_response):
     def verify(self, request):
         config = self.config
         logger = config.logger
-        logger.debug('Starting request verification')
+        logger.debug("Starting request verification {}".format(request.path))
         ctx = None
         try:
             if not config._module_enabled:
@@ -72,7 +71,7 @@ def verify(self, request):
             ctx = PxContext(request, config)
             return ctx, self._request_verifier.verify_request(ctx, request)
         except Exception as err:
-            logger.error("Caught exception222, passing request. Exception: {}".format(err))
+            logger.error("Caught exception in verify, passing request. Exception: {}".format(err))
             if ctx:
                 self.report_pass_traffic(ctx)
             else:

perimeterx/px_activities_client.py

@@ -47,7 +47,6 @@ def send_to_perimeterx(activity_type, ctx, config, detail):
             'http_method': ctx.http_method,
             'http_version': ctx.http_version,
             'module_version': config.module_version,
-            'risk_mode': config.module_mode,
         }
 
         if len(detail.keys()) > 0:
@@ -84,7 +83,7 @@ def send_block_activity(ctx, config):
         'risk_rtt': ctx.risk_rtt,
         'cookie_origin': ctx.cookie_origin,
         'block_action': ctx.block_action,
-        'simulated_block': config.module_mode is px_constants.MODULE_MODE_MONITORING,
+        'simulated_block': config.module_mode is px_constants.MODULE_MODE_MONITORING or ctx.monitored_route,
     })
 
 

perimeterx/px_api.py

@@ -112,6 +112,7 @@ def verify(ctx, config):
 
 def prepare_risk_body(ctx, config):
     logger = config.logger
+    risk_mode  = 'monitor' if config.module_mode == px_constants.MODULE_MODE_MONITORING or ctx.monitored_route else 'active_blocking'
     body = {
         'request': {
             'ip': ctx.ip,
@@ -125,7 +126,7 @@ def prepare_risk_body(ctx, config):
             'http_method': ctx.http_method,
             'http_version': ctx.http_version,
             'module_version': config.module_version,
-            'risk_mode': config.module_mode,
+            'risk_mode': risk_mode,
             'cookie_origin': ctx.cookie_origin
         }
     }

perimeterx/px_config.py

@@ -54,6 +54,31 @@ def __init__(self, config_dict):
             raise TypeError('enforced_specific_routes must be a list')
         self._enforced_specific_routes = enforced_routes
 
+        monitored_routes = config_dict.get('monitored_specific_routes', [])
+        if not isinstance(monitored_routes, list):
+            raise TypeError('monitored_specific_routes must be a list')
+        self._monitored_specific_routes = monitored_routes
+
+        sensitive_routes_regex = config_dict.get('sensitive_routes_regex', [])
+        if not isinstance(sensitive_routes_regex, list):
+            raise TypeError('sensitive_routes_regex must be a list')
+        self._sensitive_routes_regex = sensitive_routes_regex
+
+        whitelist_routes_regex = config_dict.get('whitelist_routes_regex', [])
+        if not isinstance(whitelist_routes_regex, list):
+            raise TypeError('whitelist_routes_regex must be a list')
+        self._whitelist_routes_regex = whitelist_routes_regex
+
+        enforced_routes_regex = config_dict.get('enforced_specific_routes_regex', [])
+        if not isinstance(enforced_routes_regex, list):
+            raise TypeError('enforced_specific_routes must be a list')
+        self._enforced_specific_routes_regex = enforced_routes_regex
+
+        monitored_routes_regex = config_dict.get('monitored_specific_routes_regex', [])
+        if not isinstance(monitored_routes_regex, list):
+            raise TypeError('monitored_specific_routes_regex must be a list')
+        self._monitored_specific_routes_regex = monitored_routes_regex
+
         self._block_html = 'BLOCK'
         self._logo_visibility = 'visible' if custom_logo is not None else 'hidden'
         self._telemetry_config = self.__create_telemetry_config()
@@ -165,6 +190,14 @@ def sensitive_routes(self):
     def whitelist_routes(self):
         return self._whitelist_routes
 
+    @property
+    def sensitive_routes_regex(self):
+        return self._sensitive_routes_regex
+
+    @property
+    def whitelist_routes_regex(self):
+        return self._whitelist_routes_regex
+
     @property
     def block_html(self):
         return self._block_html
@@ -205,6 +238,18 @@ def bypass_monitor_header(self):
     def enforced_specific_routes(self):
         return self._enforced_specific_routes
 
+    @property
+    def monitored_specific_routes(self):
+        return self._monitored_specific_routes
+
+    @property
+    def enforced_specific_routes_regex(self):
+        return self._enforced_specific_routes_regex
+
+    @property
+    def monitored_specific_routes_regex(self):
+        return self._monitored_specific_routes_regex
+
     def __instantiate_user_defined_handlers(self, config_dict):
         self._custom_request_handler = self.__set_handler('custom_request_handler', config_dict)
         self._get_user_ip = self.__set_handler('get_user_ip', config_dict)

perimeterx/px_constants.py

@@ -27,7 +27,7 @@
 EMPTY_GIF_B64 = 'R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw=='
 COLLECTOR_HOST = 'collector.perimeterx.net'
 FIRST_PARTY_FORWARDED_FOR = 'X-FORWARDED-FOR'
-MODULE_VERSION = 'Python 3 WSGI Module v1.0.1'
+MODULE_VERSION = 'Python 3 WSGI Module v1.1.0'
 API_RISK = '/api/v3/risk'
 PAGE_REQUESTED_ACTIVITY = 'page_requested'
 BLOCK_ACTIVITY = 'block'

perimeterx/px_context.py

@@ -1,9 +1,10 @@
+import re
+
 from requests.structures import CaseInsensitiveDict
 
 from perimeterx.px_constants import *
 from perimeterx.px_data_enrichment_cookie import PxDataEnrichmentCookie
 
-
 class PxContext(object):
 
     def __init__(self, request, config):
@@ -49,9 +50,10 @@ def __init__(self, request, config):
         uri = request.path
         full_url = request.url
         hostname = request.host
-        sensitive_route = sum(1 for _ in filter(lambda sensitive_route_item: uri.startswith(sensitive_route_item), config.sensitive_routes)) > 0
-        whitelist_route = sum(1 for _ in filter(lambda whitelist_route_item: uri.startswith(whitelist_route_item), config.whitelist_routes)) > 0
-        enforced_route = sum(1 for _ in filter(lambda enforced_route_item: uri.startswith(enforced_route_item), config.enforced_specific_routes)) > 0
+        sensitive_route = sum(1 for _ in filter(lambda sensitive_route_item:  re.search(sensitive_route_item, uri), config.sensitive_routes_regex)) > 0 or sum(1 for _ in filter(lambda sensitive_route_item: uri.startswith(sensitive_route_item), config.sensitive_routes)) > 0
+        whitelist_route = sum(1 for _ in filter(lambda whitelist_route_item: re.search(whitelist_route_item, uri), config.whitelist_routes_regex)) > 0 or sum(1 for _ in filter(lambda whitelist_route_item: uri.startswith(whitelist_route_item), config.whitelist_routes)) > 0
+        enforced_route = sum(1 for _ in filter(lambda enforced_route_item: re.search(enforced_route_item, uri), config.enforced_specific_routes_regex)) > 0 or sum(1 for _ in filter(lambda enforced_route_item: uri.startswith(enforced_route_item), config.enforced_specific_routes)) > 0
+        monitored_route = not enforced_route and (sum(1 for _ in filter(lambda monitored_route_item: re.search(monitored_route_item, uri), config.monitored_specific_routes_regex)) > 0 or sum(1 for _ in filter(lambda monitored_route_item: uri.startswith(monitored_route_item), config.monitored_specific_routes)) > 0)
 
         protocol_split = request.environ.get('SERVER_PROTOCOL', '').split('/')
         if protocol_split[0].startswith('HTTP'):
@@ -78,6 +80,7 @@ def __init__(self, request, config):
         self._sensitive_route = sensitive_route
         self._whitelist_route = whitelist_route
         self._enforced_route = enforced_route
+        self._monitored_route = monitored_route
         self._s2s_call_reason = 'none'
         self._cookie_origin = cookie_origin
         self._is_mobile = cookie_origin == "header"
@@ -259,6 +262,14 @@ def whitelist_route(self):
     def whitelist_route(self, whitelist_route):
         self._whitelist_route = whitelist_route
 
+    @property
+    def monitored_route(self):
+        return self._monitored_route
+
+    @monitored_route.setter
+    def monitored_route(self, monitored_route):
+        self._monitored_route = monitored_route
+
     @property
     def s2s_call_reason(self):
         return self._s2s_call_reason

perimeterx/px_cookie.py

@@ -64,7 +64,7 @@ def decrypt_cookie(self):
             if iterations < 1 or iterations > 10000:
                 return False
             data = base64.b64decode(parts[2])
-            dk = hashlib.pbkdf2_hmac(hash_name='sha256', password=config.cookie_key.encode(), salt=salt, iterations=iterations, dklen=48)
+            dk = hashlib.pbkdf2_hmac(hash_name='sha256', password=self._config.cookie_key.encode(), salt=salt, iterations=iterations, dklen=48)
             key = dk[:32]
             iv = dk[32:]
             cipher = AES.new(key, AES.MODE_CBC, iv)

perimeterx/px_httpc.py

@@ -28,7 +28,7 @@ def send(full_url, body, headers, config, method, raise_error = False):
             response = requests.post(url='https://' + full_url, headers=headers, data=body, timeout=config.api_timeout)
 
         if response.status_code >= 400:
-            logger.debug('PerimeterX server call failed')
+            logger.debug('PerimeterX server call failed: ' + str(response.status_code))
             return False
         finish = time.time()
         request_time = finish - start