Version 4.12.1

General Audience Stuff

• [etc/docker, docs] Fixed/updated certain technical details in the base image's Dockerfile. Applied minor updates, fixes and improvements to various parts of the documentation (including this changelog).

System/Configuration/Programming-Only

• [setup, lib] Added the redis==2.10.6 pinned requirement to N6Lib/requirements.

Version 4.12.0

General Audience Stuff

• [setup, lib, etc/docker] Debian 12 (bookworm) and CPython 3.11 are now the officially recommended operating system and Python implementation. (CPython 3.9 is still supported.)

• [data sources] New data sources: turris-cz.greylist-csv (collector and parser), withaname.ddosia (collector and parser) and shadowserver.bgp (just another shadowerver parser).

• [data sources] Changed the shadowserver.ftp parser's constant value of the name event attribute to "ftp allow password wo ssl" (previously it was "ftp, clear text pass").

• [data sources] Fixed a bug in the abuse-ch.urlhaus-urls collector by removing the (mistakenly kept) rigid limit on numbers of events being sent.

• [data sources] Removed the malwarepatrol.malurl collector.

• [portal, rest api, stream api, admin panel, data pipeline] Added a new feature: Ignore Lists. From now on, n6 administrators/operators can use Admin Panel to create and manage Ignore Lists, each identified by a unique label, with optional comment, flagged as active or not, and -- what is most interesting -- containing any number of Ignored IP Networks (note: bare IP addresses are also accepted; they are automatically converted to .../32 networks). The n6filter component will mark as ignored (by setting the ignored event field to True) each event that contains the address field whose value is a non-empty list including only dicts with ignored IP addresses (by an ignored IP address we mean an ip item which matches at least one Ignored IP Network belonging to any active Ignore List); any other events are marked as not ignored (by setting the ignored field to False). For non-privileged users (i.e., those whose organizations have full_access=False in the Auth DB) results generated by Portal, REST API (+ Test REST API) and Stream API/n6anonymizer do not include events marked as ignored. On the other hand, for privileged users (those whose organizations have full_access=True in the Auth DB) results generated by those n6 components include both not ignored and ignored events, and then each event contains the ignored field (set either to True or False) -- except that in the case of Stream API/n6anonymizer all users are treated as if they were non-privileged. Additionally, privileged users can filter results from REST API (and Portal API) by using a new query parameter: ignored (Boolean).

• [portal, admin panel, docs] Added a new feature: Organization Agreements. It allows the administrators/operators of an n6 instance to use Admin Panel to define optional terms (agreements) which then can be accepted/rejected, via Portal, by any existing and new (future) users of n6 -- on behalf of their organizations. The new feature is comprehensively documented.

• [portal, admin panel] Enhanced the Edit organization settings form in the Portal frontend and the corresponding backend stuff as well as the related Admin Panel stuff -- to allow adding and/or removing users within the logged user's organization (actually: requesting n6 administrators/operators to, respectively, add/re-activate and/or deactivate users...).

• [portal, rest api, stream api, data sources, data pipeline, event db, lib] The name event field (event attribute) is now coerced by the n6 data pipeline's machinery (namely, by n6lib.record_dict.RecordDict...) to pure ASCII (by replacing each non-ASCII character with ?), and is, generally, required by all other parts of n6 to be pure ASCII... (However, when it comes to how events' id values are computed by parsers, efforts have been made to keep that unaffected by the coercion -- so that resultant id values remain the same as previously for the same input values of name.) Events stored in the Event DB are now also expected to have name (if present) already coerced that way. (See also the descriptions of the Event-DB-related changes below...)

• [portal, rest api, data pipeline, event db, lib] The count event field (event attribute) is no longer constrained to be less than or equal to 32767 (now its maximum value is 4294967295 which seems big enough for any practical purposes...). Therefore, n6aggregator does not set the count_actual field anymore. (See also the descriptions of the Event-DB-related changes below...)

• [portal, rest api, data pipeline, event db, lib] Non-BMP Unicode characters (i.e., Unicode codepoints greater than 0xFFFF) are now properly supported (if present) in values of the url and target event fields (attributes), i.e., now they can be reliably stored, looked up and retrieved in/from the Event DB, thanks to using the utf8mb4 charset at the database level. (Previously, that was broken because of using the legacy max-3-bytes charset utf8. See also the descriptions of the Event-DB-related changes below...)

• [portal, rest api, event db, lib] Filtering the results by the url event field (attribute) -- by using the url or url.sub query parameter -- is now stricter in some ways, because the underlying MariaDB collation (for the Event DB's column url in the event table) changed from utf8_unicode_ci to utf8mb4_bin (in particular, now url values are compared in a case-sensitive manner).

• [portal, rest api, event db, lib] Filtering the results by the target event field (attribute) may behave in a slightly different way, because the underlying MariaDB collation (for the Event DB's column target in the event table) changed from utf8_unicode_ci to utf8mb4_unicode_520_ci.

• [portal, rest api, event db, lib] The modified event field (attribute) is now mandatory (i.e., guaranteed to be present in every event). See also the descriptions of the Event-DB-related changes below...

• [portal, rest api, data pipeline, auth db, lib] Implemented several performance enhancements/fixes and optimizations regarding retrieving and caching authorization data from the Auth DB (that is, concerning the stuff implemented in the n6lib.auth_api module and related modules; the addition of the recent_write_op_commit Auth DB table, mentioned later, is also related to that...). One of those enhancements is a new optional mechanism called pickle cache (see the related configuration options mentioned later...).

• [portal, rest api, lib] n6lib.db_events.n6NormalizedData.like_query(): fixed a bug causing injecting LIKE's wildcards when querying REST API or Portal API using query parameters url.sub/fqdn.sub (SQL pattern injection). It was not a security problem, but it caused that for some queries involving the affected parameters too large results (supersets of correct results) were obtained.

• [portal, lib] n6lib.pyramid_commons.mfa_helpers: fixed the value and the use of MFA_CODE_MAX_VALIDITY_DURATION_IN_SECONDS (previously named MFA_CODE_MAX_ACCEPTABLE_AGE_IN_SECONDS). Before the fix, if a Portal user successfully used an MFA code to log in, doing that "too early" but still within that MFA code's validity period (making use of the clock drift tolerance feature), it was then possible, for the same user, to successfully use the same MFA code once again, by doing that sufficiently late yet still within the same validity period. The crux of the bug was that the period of treating MFA codes as "already spent" was too short. (Note that the fixed bug does not look like a serious security flaw.)

• [portal] Applied many GUI/UX-related Portal fixes and enhancements... Among others, from now on, dates/times on the Incidents page are consistently processed/presented using UTC times; also, support for some additional search parameters have been added.

• [stream api, auth db, lib] Since now, all new organizations have Stream API enabled by default (the default value of the stream_api_enabled field of the n6lib.auth_db.models.Org model is now True).

• [admin panel, lib] All editable fields in the Admin Panel accepting an IP network (in the CIDR notation) now also accept a bare IP address (which is automatically converted to a .../32 network). What has actually been changed is the validation procedure for all ip_network fields defined in n6lib.auth_db.models.... (To make that possible, n6sdk.data_spec.fields.IPv4NetField, and all its subclasses, gained a new option: accept_bare_ip -- of type bool, specifiable as a subclass attribute or a keyword argument to the constructor, with False as the default value).

• [admin panel, lib] Added a new column, Is Active, to the Admin Panel's User list view; the new column represents a newly added property of n6lib.auth_db.models.User: is_active -- whose value is always a logical negation of the (already existing) User model's field is_blocked (representing the user Auth DB table's column is_blocked).

• [docs] The n6's documentation: added a new article: n6 REST API; significantly improved/updated two existing articles: n6 Stream API and Docker-Based Installation; applied a bunch of fixes, improvements and updates to other parts of the documentation.

System/Configuration/Programming-Only

• [event db, lib] Made numerous changes to the schema and basic setup of the Event DB (see, in particular, the etc/mysql/initdb/*.sql files...). Namely: the MariaDB engine used for the Event DB is now RocksDB (rather than TokuDB); the general Event DB's character set and collation (that apply, among others, to the name column in the event table...) are now ascii and ascii_general_ci (rather than the legacy max-3-bytes charset utf8 with the collation utf8_unicode_ci), except that, in the event table, the character set and collation for the url column are now utf8mb4 and utf8mb4_bin, and the cha...

Version 4.5.0

General Audience Stuff

• [data pipeline, lib] n6filter: fixed a bug (in the machinery of n6lib.auth_api.InsideCriteriaResolver...) related to event ownership criteria (aka "inside" resource events criteria) regarding the very unlikely (yet not impossible) corner case of the 0.0.0.0/32 IP network defined as such a criterion in the Auth DB... The bug might make n6filter reject all incoming data (because of raised exceptions).

• [tests, docs] Non-major enhancements and fixes regarding some unit tests and documentation.

System/Configuration/Programming-Only

• [data sources, setup, config, etc/docker, tests] Globally renamed the spamhaus.edrop parser's class SpamhausEdrop202303Parser (defined in n6datasources.parsers.spamhaus and referred to in a few other places -- in particular, being the name of the-parser-dedicated configuration section!) to SpamhausEdropParser, as well as the executable script n6parser_spamhausedrop202303 to n6parser_spamhausedrop; also, fixed n6datasources.tests.parsers.test_spamhaus.TestSpamhausEdropParser by removing its attribute PARSER_RAW_FORMAT_VERSION_TAG. The rationale for these changes is that no raw format version tag has ever been assigned to the spamhaus.edrop parser.

Programming-Only

• [tests] n6datasources.tests.parsers._parser_test_mixin: enhanced certain ParserTestMixin-provided checks related to raw format version tags.

Version 4.4.0

Features and Notable Changes

General Audience Stuff

• [data sources, config] Added support for the shadowserver.msmq source (by adding the parser for it, as there already exists one common collector for all shadowserver.* sources; obviously, appropriate additions have been made in the collector's and parser's sections in the N6DataSources/n6datasources/data/conf/60_shadowserver.conf config prototype file).

• [data sources, config] Removed support for the following sources: blueliv.map and darklist-de.bl (removed both collectors and parsers!) as well as shadowserver.modbus (removed just this source's parser).

• [data sources] The parsers for the dataplane.* sources have been changed to support the current data format (there was a need to change the delimiter and the row parsing mechanism...).

• [data sources] The collector for the abuse-ch.ssl-blacklist source (implemented in n6datasources.collectors.abuse_ch as the class named AbuseChSslBlacklistCollector) used to be able to load the collector state in a legacy format related to the value of the class attribute row_time_legacy_state_key -- that format is no longer supported, as the base class _BaseAbuseChDownloadingTimeOrderedRowsCollect no longer makes use of that attribute. Note: these changes are relevant and breaking only if you need to load your collector state in that old format -- almost certainly you do not.

• [data sources] A new processing mechanism has been added to numerous existing parsers for shadowserver.* sources (by enhancing the _BaseShadowserverParser class, defined in the n6datasources.parsers.shadowserver module) -- concerning events categorized as "amplifier". The mechanism is activated when a CVE-...-like-regex-based match is found in the tag field of the input data -- then the parser, apart from yielding an event (hereinafter referred to as a basic event) with category set to "amplifier", also yields an extra event -- which is identical to the basic one, except that its category is set to "vulnerable" and its name is set to the regex-matched value (which is, basically, the CVE identifier). Because of that, name and category should no longer be declared as parser's constant_items, so now _BaseShadowserverParser provides support for additional_standard_items (which is a parser class's attribute similar to constant_items). For relevant parser classes, the name and category items have been moved from their constant_items to their additional_standard_items.

• [data sources] Now the generic *.misp collector supports loading state also in its legacy Python-2-specific format.

• [data sources, data pipeline, lib] A new restriction (implemented in n6lib.data_spec.fields, concerning the IPv4FieldForN6 and AddressFieldForN6 classes) is that, from now on, the zero IP address (0.0.0.0) is neither a valid component IP within a record dict's address (i.e., its items' ip) or enriched (i.e., keys in the mapping being its second item), nor a valid value of a record dict's dip. Note that this restriction regards all parsers and most of the other data pipeline components (via the machinery of n6lib.record_dict.RecordDict et consortes...).

• [data pipeline] The name of the AMQP input queue declared by n6enrich has been changed (!) from enrichement to enrichment.

• [data pipeline] The n6enrich pipeline component (implemented in n6datapipeline.enrich): from now on, the zero IP address (0.0.0.0), irrespective of its exact formatting (i.e., regardless whether some octets are formatted with redundant leading zeros), is no longer taken into account when IPs are extracted from urls, and when fqdns are resolved to IPs.

• [data pipeline, event db, config] From now on, when n6recorder, during its activity (i.e., within Recorder.input_callback()...), encounters an exception which represents a database/DB API error (i.e., an instance of a MySQLdb.MySQLError subclass, possibly wrapped in (an) SQLAlchemy-specific exception(s)...) whose error code (i.e., <exception>.args[0] being an int, if any) indicates a fatal condition -- then a SystemExit(<appropriate message>) is raised, so that the AMQP input message is requeued and the n6recorder executable script exits with a non-zero status. The set of error codes which are considered fatal (i.e. which trigger this behavior) is configurable -- by setting the fatal_db_api_error_codes configuration option in the recorder section; by default, that set includes only one value: 1021 (i.e., the ERR_DISK_FULL code -- see the error codes listing on the MariaDB website).

• [portal, rest api, stream api, data pipeline, lib] A security-related behavioral fix has been applied to the event access rights and event ownership machinery (implemented in n6lib.auth_api...): from now on, IP-network-based access or ownership criteria (those stored in the criteria_ip_network and inside_filter_ip_network Auth DB tables) referring to networks that contain the zero IP address (0.0.0.0) are translated to IP address ranges whose lower bound is 0.0.0.1 (in other words, 0.0.0.0 is excluded). Thanks to that, events without ip are no longer erroneously considered as matching such IP-network-based criteria. In practice, from the security point of view, the fix is most important when it comes to Portal and REST API (considering that those components query the Event DB, in records of which the absence of an IP is, for certain technical reasons, represented by the value 0 rather than NULL). For other involved components, i.e., n6filter and n6anonymizer/Stream API, the security risk was rather small or non-existent. Note: as the fix is also related to n6filter, it affects values of min_ip in the inside_criteria part of the JSON returned by the Portal API's endpoint /info/config; they are displayed by the Portal's GUI: in the Account information page, in the "Inside" resource events criteria section, below the IP network filter label -- as IP ranges' lower bounds.

• [portal, rest api, lib] A behavioral fix related to the one described above (yet, this time, not related to security) has been applied to the procedure of translation of the ip.net request parameter to the corresponding fragment of Event DB queries (see: the ip_net_query() method of n6lib.db_events.n6NormalizedData...): from now on, each value that refers to a network which contains the zero IP address (0.0.0.0) is translated to an IP address range whose lower bound is 0.0.0.1 (in other words, 0.0.0.0 is excluded); thanks to that, events with no ip are no longer erroneously included in such cases.

• [portal, rest api, lib] A new restriction (implemented in n6lib.data_spec.fields, concerning the IPv4FieldForN6 and AddressFieldForN6 classes) is that the zero IP address (0.0.0.0) is no longer a valid value of the ip and dip request parameters received by REST API's endpoints and analogous Portal API's endpoints. Also, regarding the Portal's GUI, the front-end validation part related to the IP search parameter has been appropriately adjusted.

• [portal, rest api, lib] The mechanism of result data cleaning (implemented as a part of a certain non-public stuff invoked in n6lib.data_spec.N6DataSpec.clean_result_dict()) has been enhanced in such a way that the address field of cleaned result dicts no longer includes any items with ip equal to the zero IP address (0.0.0.0), i.e., they are filtered out even if they appear in some Event DB records (they could when it comes to legacy data). Note that it is complemented by the already existing mechanism of removing from raw result dicts any ip and dip fields whose values are equal to the zero IP address (see: n6lib.db_events.make_raw_result_dict()...).

• [rest api, config, lib] n6lib.generate_test_events: several changes and enhancements regarding the RandomEvent class have been made, including backward incompatible additions/removals/modifications of options defined by its config spec, affecting the way the optional test REST API application (provided by n6web.main_test_api et consortes...) is configured using generator_rest_api.* options... Also, most of the RandomEvent's configuration-related stuff has been factored out to a new mixin class, RandomEventGeneratorConfigMixin.

System/Configuration/Programming-Only

• [data sources, data pipeline, config, etc/docker] Added, fixed, changed and removed several config prototype (*.conf) files in the directories: N6DataSources/n6datasources/data/conf/, N6DataPipeline/n6datapipeline/data/conf/ and etc/n6/. Note: for some of them, manual adjustments in user's actual configuration files are required (see the relevant comments in those files...).

• [setup, lib] N6Lib's dependencies: changed the version of dnspython from 1.16 to 2.4. Also, added a new dependency, importlib_resources, with version locked as >=5.12, <5.13.

• [setup, data pipeline] N6DataPipeline's dependencies: temporarily locked the version of intelmq as <3.2.

Programming-Only

• [data pipeline] n6datapipeline.enrich.Enricher: renamed the url_to_fqdn_or_ip() method to url_to_hostname(), and changed its interface regarding the return value: now it is always either a non-empty str or None.

• [lib] n6lib.common_helpers and n6sdk.encoding_helpers: renamed the try_to_normalize_surrogate_pairs_to_proper_codepoints() function to replace_surrogate_pairs_with_proper_codepoints().

• [lib] Removed three functions from n6lib.common_helpers: is_ipv4(), is_pure_ascii() and lower_if_pure_ascii().

• [lib] n6lib.db_events: removed IPAddress's constant attributes NONE a...

Version 4.0.1

• [docs, setup] Fixed generation of the docs by upgrading mkdocs to the version 1.2.4.

Version 4.0.0

This release is a big milestone.

Among others:

• the n6 Portal gained support for OpenID Connect-based single sign-on (SSO) authentication;

• the n6 Stream API (STOMP-based) now supports authentication based on API keys (those which have already been accepted by the n6 REST API); the new mechanism, implemented as a part of the N6BrokerAuthApi package, replaces the previously used mechanism (which was based on X.509 client certificates);

• added a significant number of components obtaining and processing security event data from external sources: 26 collectors and 86 parsers; now, in total, we have 35 collectors and 91 parsers (see the N6DataSources package);

• got rid of the Python-2-compatible legacy code (most of which were Python 2 versions of collectors and parsers) that used to reside in N6Core; the accompanying Python 2 packages (N6CoreLib, N6Lib-py2 and N6SDK-py2) have also been removed; note that the components related to active data sources have been migrated to Python 3 (8 collectors and 7 parsers -- now they reside in N6DataSources); therefore, n6 is now Python-3-only (finally!);

• significant performance improvements have been accomplished: certain kinds of data queries (via the n6 REST API or n6 Portal) have become much faster, and n6aggregator's memory consumption has been considerably reduced;

• also, many minor improvements, a bunch of fixes, some refactoring and various cleanups have been made.

Note that some of the changes are not backwards-compatible.

Version 3.0.1

• [docs] A bunch of fixes and improvements regarding the documentation, including major changes to its structure, layout and styling.

• [setup] do_setup.py: regarding the default value of the option --additional-packages under Python 3, the version of the mkdocs package has been pinned (1.2.3), and the mkdocs-material package (providing the material docs theme) has been added (and its version is also pinned: 8.0.3); regarding the same under Python 2, the mkdocs package has been removed.

Version 3.0.0

This release is a big milestone. It includes, among others:

• migration to Python 3

• in the n6 data pipeline infrastructure: optional integration with IntelMQ

• in the n6 Portal: a new frontend (implemented using React), two-factor authentication (based on TOTP), user's/organization's own data management (including config update and password reset forms, with related e-mail notices), and other goodies...

• in the n6 REST API: API-key-based authentication

• and many, many more improvements, a bunch of fixes, as well as some refactoring, removals and cleanups...

Note that many of the changes are not backwards-compatible.

Also, note that most of the main elements of n6 -- namely: N6DataPipeline, N6DataSources, N6Portal, N6RestApi, N6AdminPanel, N6BrokerAuthApi, N6Lib and N6SDK -- are now Python-3-only (more precisely: are compatible with CPython 3.9).

---

The legacy, Python-2-only stuff -- most of which are collectors and parsers (external-data-sources-related components) -- reside in N6Core and N6CoreLib; the collectors and parsers placed in N6Core, if related to non-obsolete external data sources, will be gradually migrated to Python-3-only N6DataSources (so that, finally, we will be able to rid of N6Core and N6CoreLib). There are also Python-2-only variants of N6Lib and N6SDK: N6Lib-py2 and N6SDK-py2 (needed only as dependencies of N6Core/N6CoreLib).

Version 2.0.0

This was the first public release of n6 (on 2018-06-22).