Skip to content

Version 4.12.0
Compare
Choose a tag to compare
@zuo zuo released this 23 Dec 16:52
· 2 commits to master since this release
v4.12.0
bdeb557

General Audience Stuff

  • [setup, lib, etc/docker] Debian 12 (bookworm) and CPython 3.11 are now the officially recommended operating system and Python implementation. (CPython 3.9 is still supported.)

  • [data sources] New data sources: turris-cz.greylist-csv (collector and parser), withaname.ddosia (collector and parser) and shadowserver.bgp (just another shadowerver parser).

  • [data sources] Changed the shadowserver.ftp parser's constant value of the name event attribute to "ftp allow password wo ssl" (previously it was "ftp, clear text pass").

  • [data sources] Fixed a bug in the abuse-ch.urlhaus-urls collector by removing the (mistakenly kept) rigid limit on numbers of events being sent.

  • [data sources] Removed the malwarepatrol.malurl collector.

  • [portal, rest api, stream api, admin panel, data pipeline] Added a new feature: Ignore Lists. From now on, n6 administrators/operators can use Admin Panel to create and manage Ignore Lists, each identified by a unique label, with optional comment, flagged as active or not, and -- what is most interesting -- containing any number of Ignored IP Networks (note: bare IP addresses are also accepted; they are automatically converted to .../32 networks). The n6filter component will mark as ignored (by setting the ignored event field to True) each event that contains the address field whose value is a non-empty list including only dicts with ignored IP addresses (by an ignored IP address we mean an ip item which matches at least one Ignored IP Network belonging to any active Ignore List); any other events are marked as not ignored (by setting the ignored field to False). For non-privileged users (i.e., those whose organizations have full_access=False in the Auth DB) results generated by Portal, REST API (+ Test REST API) and Stream API/n6anonymizer do not include events marked as ignored. On the other hand, for privileged users (those whose organizations have full_access=True in the Auth DB) results generated by those n6 components include both not ignored and ignored events, and then each event contains the ignored field (set either to True or False) -- except that in the case of Stream API/n6anonymizer all users are treated as if they were non-privileged. Additionally, privileged users can filter results from REST API (and Portal API) by using a new query parameter: ignored (Boolean).

  • [portal, admin panel, docs] Added a new feature: Organization Agreements. It allows the administrators/operators of an n6 instance to use Admin Panel to define optional terms (agreements) which then can be accepted/rejected, via Portal, by any existing and new (future) users of n6 -- on behalf of their organizations. The new feature is comprehensively documented.

  • [portal, admin panel] Enhanced the Edit organization settings form in the Portal frontend and the corresponding backend stuff as well as the related Admin Panel stuff -- to allow adding and/or removing users within the logged user's organization (actually: requesting n6 administrators/operators to, respectively, add/re-activate and/or deactivate users...).

  • [portal, rest api, stream api, data sources, data pipeline, event db, lib] The name event field (event attribute) is now coerced by the n6 data pipeline's machinery (namely, by n6lib.record_dict.RecordDict...) to pure ASCII (by replacing each non-ASCII character with ?), and is, generally, required by all other parts of n6 to be pure ASCII... (However, when it comes to how events' id values are computed by parsers, efforts have been made to keep that unaffected by the coercion -- so that resultant id values remain the same as previously for the same input values of name.) Events stored in the Event DB are now also expected to have name (if present) already coerced that way. (See also the descriptions of the Event-DB-related changes below...)

  • [portal, rest api, data pipeline, event db, lib] The count event field (event attribute) is no longer constrained to be less than or equal to 32767 (now its maximum value is 4294967295 which seems big enough for any practical purposes...). Therefore, n6aggregator does not set the count_actual field anymore. (See also the descriptions of the Event-DB-related changes below...)

  • [portal, rest api, data pipeline, event db, lib] Non-BMP Unicode characters (i.e., Unicode codepoints greater than 0xFFFF) are now properly supported (if present) in values of the url and target event fields (attributes), i.e., now they can be reliably stored, looked up and retrieved in/from the Event DB, thanks to using the utf8mb4 charset at the database level. (Previously, that was broken because of using the legacy max-3-bytes charset utf8. See also the descriptions of the Event-DB-related changes below...)

  • [portal, rest api, event db, lib] Filtering the results by the url event field (attribute) -- by using the url or url.sub query parameter -- is now stricter in some ways, because the underlying MariaDB collation (for the Event DB's column url in the event table) changed from utf8_unicode_ci to utf8mb4_bin (in particular, now url values are compared in a case-sensitive manner).

  • [portal, rest api, event db, lib] Filtering the results by the target event field (attribute) may behave in a slightly different way, because the underlying MariaDB collation (for the Event DB's column target in the event table) changed from utf8_unicode_ci to utf8mb4_unicode_520_ci.

  • [portal, rest api, event db, lib] The modified event field (attribute) is now mandatory (i.e., guaranteed to be present in every event). See also the descriptions of the Event-DB-related changes below...

  • [portal, rest api, data pipeline, auth db, lib] Implemented several performance enhancements/fixes and optimizations regarding retrieving and caching authorization data from the Auth DB (that is, concerning the stuff implemented in the n6lib.auth_api module and related modules; the addition of the recent_write_op_commit Auth DB table, mentioned later, is also related to that...). One of those enhancements is a new optional mechanism called pickle cache (see the related configuration options mentioned later...).

  • [portal, rest api, lib] n6lib.db_events.n6NormalizedData.like_query(): fixed a bug causing injecting LIKE's wildcards when querying REST API or Portal API using query parameters url.sub/fqdn.sub (SQL pattern injection). It was not a security problem, but it caused that for some queries involving the affected parameters too large results (supersets of correct results) were obtained.

  • [portal, lib] n6lib.pyramid_commons.mfa_helpers: fixed the value and the use of MFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS (previously named MFA_CODE_MAX_ACCEPTABLE_AGE_IN_SECONDS). Before the fix, if a Portal user successfully used an MFA code to log in, doing that "too early" but still within that MFA code's validity period (making use of the clock drift tolerance feature), it was then possible, for the same user, to successfully use the same MFA code once again, by doing that sufficiently late yet still within the same validity period. The crux of the bug was that the period of treating MFA codes as "already spent" was too short. (Note that the fixed bug does not look like a serious security flaw.)

  • [portal] Applied many GUI/UX-related Portal fixes and enhancements... Among others, from now on, dates/times on the Incidents page are consistently processed/presented using UTC times; also, support for some additional search parameters have been added.

  • [stream api, auth db, lib] Since now, all new organizations have Stream API enabled by default (the default value of the stream_api_enabled field of the n6lib.auth_db.models.Org model is now True).

  • [admin panel, lib] All editable fields in the Admin Panel accepting an IP network (in the CIDR notation) now also accept a bare IP address (which is automatically converted to a .../32 network). What has actually been changed is the validation procedure for all ip_network fields defined in n6lib.auth_db.models.... (To make that possible, n6sdk.data_spec.fields.IPv4NetField, and all its subclasses, gained a new option: accept_bare_ip -- of type bool, specifiable as a subclass attribute or a keyword argument to the constructor, with False as the default value).

  • [admin panel, lib] Added a new column, Is Active, to the Admin Panel's User list view; the new column represents a newly added property of n6lib.auth_db.models.User: is_active -- whose value is always a logical negation of the (already existing) User model's field is_blocked (representing the user Auth DB table's column is_blocked).

  • [docs] The n6's documentation: added a new article: n6 REST API; significantly improved/updated two existing articles: n6 Stream API and Docker-Based Installation; applied a bunch of fixes, improvements and updates to other parts of the documentation.

System/Configuration/Programming-Only

  • [event db, lib] Made numerous changes to the schema and basic setup of the Event DB (see, in particular, the etc/mysql/initdb/*.sql files...). Namely: the MariaDB engine used for the Event DB is now RocksDB (rather than TokuDB); the general Event DB's character set and collation (that apply, among others, to the name column in the event table...) are now ascii and ascii_general_ci (rather than the legacy max-3-bytes charset utf8 with the collation utf8_unicode_ci), except that, in the event table, the character set and collation for the url column are now utf8mb4 and utf8mb4_bin, and the character set and collation for the target column are now utf8mb4 and utf8mb4_unicode_520_ci; the order of the components of the event table's primary key is now: time, ip, id (previously: id, time, ip); the event table's columns modified and dip are now NOT NULL (in the case of dip, the value 0 means that there is no actual value; note that, for that column, this convention has been used for a long time); the event table's columns dport and sport are now of type SMALLINT UNSIGNED (previously: INTEGER, which was an unnecessary waste of space); the event table's column cc is now of type CHAR(2) (previously: VARCHAR(2)); the event table's column count is now of type INTEGER UNSIGNED whose max. value is 4294967295 (previously: SMALLINT with max. value 32767, which was far too small); several database indexes have been added/adjusted/removed; also, as a part of implementation of the aforementioned Ignore Lists feature, a new column has been added to the event table: ignored, of type BOOL; apart from all that, several SQL variables are now consistently set to sensible values (max_allowed_packet, sql_mode, time_zone)... What is most important from the point of view of the administrators of an n6 instance is that a suitable migration of the whole production Event DB content needs to be performed (manually).

  • [auth db, lib] As a part of implementation of the aforementioned Ignore Lists feature, added two new Auth DB tables: ignore_list and ignored_ip_network. Apart from them, added new Auth DB tables related to other features/mechanisms (also mentioned above...): agreement, org_agreement_link, org_config_update_request_user_addition_or_activation_request, org_config_update_request_user_deactivation_request, recent_write_op_commit, registration_request_agreement_link. Obviously, related model classes have been added as needed (see n6lib.auth_db.models) and any necessary field validators have been implemented (see n6lib.auth_db.validators). What is important from the point of view of the administrators of an n6 instance is that the Alembic migrations machinery needs to be used to update the schema of the production Auth DB (for the instructions how to do it, see N6Lib/n6lib/auth_db/alembic/README.md).

  • [config, data pipeline] From now on, the n6recorder's configuration option connect_charset (in the configuration section recorder) is expected to be set to the value utf8mb4 (not to the value utf8 anymore!) -- unless there are some special circumstances and you really now what you are doing, and why!

  • [config, portal, rest api] From now on, the configuration option sqlalchemy_event_db_connect_charset (in REST API's and Portal API's *.ini files) is expected to be set to the value utf8mb4 (not to the value utf8 anymore!) -- unless there are some special circumstances and you really now what you are doing, and why!

  • [config, data sources] The collectors whose classes inherit (directly or indirectly) from n6datasources.collectors.base.BaseDownloadingCollector now support a new configuration option, download_timeout, which can be set to customize HTTP(s) request timeouts.

  • [data pipeline] Added a new auxiliary executable: n6exchange_updater -- to update Stream-API-related AMQP exchange declarations and bindings (adding and deleting them as appropriate), according to the relevant Stream API settings in Auth DB. (The implementation of the component resides in the n6datapipeline.aux.exchange_updater module.)

  • [config, portal, rest api] New configuration options regarding certain performance improvement mechanisms can now be specified in the REST API's and Portal API's *.ini files (see the auth api prefetching configuration part of the relevant configuration prototype files). In particular, the aforementioned optional mechanism called pickle cache can be activated (see the comments in the related configuration prototype files regarding the options auth_api_prefetching.pickle_cache_dir and auth_api_prefetching.pickle_cache_signature_secret; please, take seriously the security considerations those comments include...).

  • [config, portal] A new configuration option, session_cookie_sign_secret, can now be specified in the Portal API's *.ini file to explicitly set the secret key for signing user session cookies (please, see the comments regarding that option in the related configuration prototype files...). By default, the option's value is empty, causing the legacy behavior (a new secret for signing session cookies being automatically generated on each start of the Portal API server application). Note: setting the option to a non-empty value is necessary if the Portal API server application is run using multiple OS processes (not just threads), otherwise user sessions cannot be handled properly.

  • [config, portal, rest api, broker auth api, admin panel, data sources, data pipeline] From now on, wherever in n6 an AMQP connection is established, authentication to RabbitMQ (the AMQP server) can be configured to be made using the PLAIN mechanism, i.e., with username and password (note: SSL-based EXTERNAL authentication, with an X.509 client certificate, is still possible -- just no longer as the only option). To learn how to configure your n6 components to use the PLAIN (username-and-password-based) mechanism, see respective comments in the relevant config prototype files: ad source/pipeline components -- 00_global.conf; ad input for collectors based on n6datasources.collectors.AMQPCollector -- 60_amqp.conf; ad logging using n6lib.log_helpers.AMQPHandler -- either logging.conf or production.ini (the latter -- only regarding Portal API and REST API). Note that on production systems, no matter which authentication mechanism is in use (client-certificate-based or username-and-password-based), connections should always be secured with SSL (TLS).

  • [config, portal, rest api, broker auth api, admin panel, data sources, data pipeline] From now on, in configuration files for any n6 components, all configuration options that concern filesystem paths (or lists of filesystem paths) are expected to be specified using only absolute paths, i.e., relative paths might no longer be accepted. Note that paths like ~/something and ~user/something (intended to be expanded by replacing a ~/~user marker with the user's home directory path) are still OK.

  • [setup, lib, admin panel] Updated versions of some external dependencies (including some security-related cases...); also, added a few new dependencies.

  • [lib, portal, rest api, broker auth api, admin panel, data sources, data pipeline, config, cli, docs, etc/docker, tests] Made a bunch of various changes/enhancements (including backward incompatible ones) and additions to the code (many related to the features and changes mentioned above...), plus various fixes/cleanups, some refactoring, modernization and adjustments/updates (among other things, many changes to accommodate some of the major Event-DB-related changes described above; as well as certain temporary hacks to ease the transition process)... Also, many tests (plus related data/fixtures/helpers) have been added, enhanced, fixed, refactored, adjusted/updated... More or less the same can be said about many n6 components' configuration prototype files, and about some other configuration-or-Docker-related stuff...

  • [portal, setup, tests] Regarding the implementation of the n6 Portal's frontend (React-based TS/JS code and related resources, together with development tooling...): made a bunch of additions, changes/enhancements, fixes/cleanups as well as some refactoring, plus external package updates and additions... Among others, upgraded Node, React and TypeScript, and implemented a comprehensive Jest-based test suite...

  • [etc/docker, docs] Added Mailhog to the Docker-related stuff.

Programming-Only

  • [lib] Removed some constants/classes/methods/attributes/functions, in particular: n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_LEGACY_CHARSET, n6lib.data_selection_tools.CondPredicateMaker.visit_RecItemParamCond() (replaced with visit_RecItemCond() mentioned below), n6lib.db_events.CustomInteger, n6lib.db_events.JSONText, (replaced with JSONMediumText mentioned below), n6lib.db_events.n6ClientToEvent.__json__(), n6lib.db_events.n6NormalizedData.to_raw_result_dict(), n6lib.ldap_api_replacement.LdapAPIConnectionError, n6lib.pyramid_commons.mfa_helpers.MFA_CODE_MAX_ACCEPTABLE_AGE_IN_SECONDS (replaced with MFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS mentioned below).

  • [lib] Added a new module: n6lib.file_helpers (providing three utility classes: FileAccessor, StampedFileAccessor, SignedStampedFileAccessor; and one utility function: as_path(); see their docstrings for more information...).

  • [lib, data sources] Added numerous constants/classes/methods/attributes/functions, in particular: n6datasources.base.parsers.BaseParser.ignored_csv_raw_row_prefixes (and overridden, as appropriate, in some subclasses of BaseParser...), n6lib.amqp_helpers.AMQPConnectionParamsError, n6lib.amqp_helpers.GUEST_PASSWORD, n6lib.amqp_helpers.GUEST_USERNAME, n6lib.amqp_helpers.MIN_REQUIRED_PASSWORD_LENGTH, n6lib.amqp_helpers.get_amqp_connection_params_dict_from_args.set_log_warning_func(), n6lib.amqp_helpers.SimpleAMQPExchangeTool, n6lib.auth_api.AuthAPI.get_ignore_lists_criteria_resolver(), n6lib.auth_db.fields.HTTPAbsoluteURLField, n6lib.auth_db.models.Agreement, n6lib.auth_db.models.IgnoredIPNetwork, n6lib.auth_db.models.IgnoreList, n6lib.auth_db.models.Org.agreements, n6lib.auth_db.models.org_agreement_link, n6lib.auth_db.models.OrgConfigUpdateRequest.user_addition_or_activation_requests, n6lib.auth_db.models.OrgConfigUpdateRequest.user_deactivation_requests, n6lib.auth_db.models.OrgConfigUpdateRequestUserAdditionOrActivationRequest, n6lib.auth_db.models.OrgConfigUpdateRequestUserDeactivationRequest, n6lib.auth_db.models.RecentWriteOpCommit, n6lib.auth_db.models.RegistrationRequest.agreements, n6lib.auth_db.models.registration_request_agreement_link, n6lib.auth_db.models.User.is_active, n6lib.class_helpers.LackOf, n6lib.common_helpers.ip_int_to_str() (+ n6sdk.addr_helpers.ip_int_to_str()), n6lib.common_helpers.PY_NON_ASCII_ESCAPED_WITH_BACKSLASHREPLACE_HANDLER_REGEX (+ n6sdk.regexes.PY_NON_ASCII_ESCAPED_WITH_BACKSLASHREPLACE_HANDLER_REGEX), n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_CONNECT_CHARSET_DEFAULT, n6lib.data_backend_api.N6DataBackendAPI.EVENT_DB_SQL_MODE, n6lib.data_selection_tools.IsTrueCond, n6lib.data_selection_tools.CondBuilder.RecItemCondBuilder.is_true(), n6lib.data_selection_tools.CondPredicateMaker.visit_RecItemCond(), n6lib.data_spec.N6DataSpec.ignored (a new event field specification), n6lib.db_events.JSONMediumText, n6lib.db_events.n6NormalizedData.single_flag_query(), n6lib.ldap_api_replacement.LdapAPI.peek_database_ver_and_timestamp(), n6lib.pyramid_commons.mfa_helpers.DELAY_TO_BE_SURE_THAT_MFA_CODE_EXPIRES, n6lib.pyramid_commons.mfa_helpers.MFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS, n6lib.record_dict.N6DataSpecWithOptionalModified, n6lib.record_dict.RecordDict.adjust_ignore(), n6lib.sqlalchemy_related_test_helpers.sqlalchemy_type_to_str(), n6lib.threaded_async.Future.peek_result(), n6lib.typing_helpers.HashObj.

Assets 2
Loading