Merge branch 'master' into python3.11

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Current (in progress)
 
+- :warning: **breaking change** `DATASET_MAX_RESOURCES_UNCOLLAPSED` config is removed.
+- Replace schemas API with a simple proxy to the `schemas.json` file [#2989](https://github.com/opendatateam/udata/pull/2989)
 - Topic: add filters in API [#3007](https://github.com/opendatateam/udata/pull/3007)
 - Move constants outside `models.py` files to `constants.py` [#3001](https://github.com/opendatateam/udata/pull/3001)
 - Move `db` and Mongo fields classes outside `udata.models` [#3005](https://github.com/opendatateam/udata/pull/3005)
@@ -863,7 +865,6 @@ Search refactor [#2680](https://github.com/opendatateam/udata/pull/2680)
 
 - Harvest sources are now filterable through the harvest source create/edit admin form [#1812](https://github.com/opendatateam/udata/pull/1812)
 - Harvest sources can now enable or disable some optional backend features [#1875](https://github.com/opendatateam/udata/pull/1875)
-- Static assets are now compatible with long-term caching (ie. their hash is present in the filename) [#1826](https://github.com/opendatateam/udata/pull/1826)
 - Post UIs have been reworked: publication date, publish/unpublish action, save and continue editing, dynamic sidebar, alignments fixes... [#1857](https://github.com/opendatateam/udata/pull/1857)
 
 ### Minor changes
@@ -879,6 +880,7 @@ Search refactor [#2680](https://github.com/opendatateam/udata/pull/2680)
 
 ### Breaking changes
 
+- Static assets are now compatible with long-term caching (ie. their hash is present in the filename). :warning: On your development environment you need to run `inv assets-build` to generate an initial `manifest.json`, both in `udata` and in your theme extension if it uses manifest. See [#1826](https://github.com/opendatateam/udata/pull/1826) for full details.
 - Theme are now responsible for adding their CSS markup on template (no more assumptions on `theme.css` and `admin.css`). Most of the time, overriding `raw.html` and `admin.html` should be sufficient
 - The discussions API `posted_by` attribute is now an embedded user instead of an user ID to avoid extra API calls [#1839](https://github.com/opendatateam/udata/pull/1839)
 

docs/adapting-settings.md

@@ -549,14 +549,6 @@ Whether or not discussions should be enabled on posts
 
 The default page size for post listing
 
-## Datasets configuration
-
-### DATASET_MAX_RESOURCES_UNCOLLAPSED
-
-**default** `6`
-
-Max number of resources to display uncollapsed in dataset view.
-
 ## Sentry configuration
 
 ### SENTRY_DSN

docs/development-environment.md

@@ -121,8 +121,6 @@ or [the documentation](building-documentation.md).
 
 [Python Virtual Environments - a Primer]: https://realpython.com/blog/python/python-virtual-environments-a-primer/
 [dev-server]: http://localhost:7000/
-[docker-compose-install]: https://docs.docker.com/compose/install/
-[docker-compose]: https://docs.docker.com/compose/
 [git]: https://git-scm.com/
 [github]: https://github.com/opendatateam/udata
 [new issue]: https://github.com/opendatateam/udata/issues/new

docs/docker-deployment.md

@@ -11,7 +11,7 @@ Head directly to [the dedicated repository][docker-repository] for an up to date
 [Docker Compose][] makes it easy to spawn [system dependencies](system-dependencies.md) out of the box by running the following command:
 
 ```shell
-docker-compose up
+docker compose up
 ```
 
 [Docker]: https://www.docker.com/

docs/getting-started.md

@@ -14,7 +14,7 @@ We’ll use the following repositories:
 
 udata requires several libraries to be installed to work. You can see them on the udata documentation link below.
 
-We’ll use [docker-compose](https://docs.docker.com/compose/) to manage external services so you don’t have to install native mongodb and redis.
+We’ll use [docker compose](https://docs.docker.com/compose/) to manage external services so you don’t have to install native mongodb and redis.
 
 # Setup udata
 
@@ -50,11 +50,11 @@ In this new directory, clone udata :
 git clone [email protected]:opendatateam/udata.git
 ```
 
-You can start your local development environment with docker-compose.
+You can start your local development environment with docker compose.
 
 ```bash
 cd udata
-docker-compose up
+docker compose up
 ```
 
 !!! warning

docs/quickstart.md

@@ -1,7 +1,7 @@
 The prefered way to test udata is to use the [docker images][]
 to easily have an up and ready udata instance.
 Pre-configured udata with needed middlewares are given as examples.
-You will need to install [docker]() and [docker-compose][] before getting started.
+You will need to install [docker][] and [docker-compose][] before getting started.
 
 ```
 git clone https://github.com/opendatateam/docker-udata

docs/testing-code.md

@@ -8,16 +8,16 @@ unit tests for the frontend and integration tests.
 If you need to use an alternative Mongo instance during tests, you can provide
 the alternate urls in you `udata.cfg` with `MONGODB_HOST_TEST`.
 
-**E.g.**: To make use of the tmpfs based middleware provided by docker-compose, use:
+**E.g.**: To make use of the tmpfs based middleware provided by docker compose, use:
 
 ```python
 MONGODB_HOST_TEST = 'mongodb://localhost:27018/udata'
 ```
 
-And then start docker-compose with the extra file:
+And then start docker compose with the extra file:
 
 ```shell
-$ docker-compose -f docker-compose.yml -f docker-compose.test.yml up
+$ docker compose -f docker-compose.yml -f docker-compose.test.yml up
 ```
 
 This will start a MongoDB extra service, tmpfs based and your tests will

js/components/dataset/resource/form.vue

@@ -249,14 +249,14 @@ export default {
         },
         schema_field() {
             if (this.hasSchemas) {
-                const values = [{id: '', label: ''}].concat(schemas.data);
+                const values = [{name: '', title: ''}].concat(schemas.data);
                 return [{
                     id: 'schema.name',
                     label: (this.resource.schema && this.resource.schema.url) ? this._('Schema (Url already set)') : this._('Schema'),
                     widget: 'select-input',
                     values,
                     map: function(item) {
-                        return {value: item.id, text: item.label};
+                        return {value: item.name, text: item.title};
                     }
                 }];
             }

js/config.js

@@ -149,11 +149,6 @@ export const license_groups = _jsonMeta('license-groups-options') && Object.from
  */
 export const harvest_validation_contact_form = _meta('harvest-validation-contact-form')
 
-/**
- * Max number of resources to display uncollapsed in dataset view
- */
-export const dataset_max_resources_uncollapsed = _jsonMeta('dataset-max-resources-uncollapsed');
-
 /**
  * The expected business identification format
  */
@@ -202,7 +197,6 @@ export default {
     tags,
     license_groups,
     harvest_validation_contact_form,
-    dataset_max_resources_uncollapsed,
     org_bid_format,
     is_search_autocomplete_enabled,
     search_autocomplete_debounce,

udata/core/dataset/api.py

@@ -734,7 +734,7 @@ def get(self):
         '''List all available schemas'''
         try:
             # This method call is cached as it makes HTTP requests
-            return ResourceSchema.objects()
+            return ResourceSchema.assignable_schemas()
         except SchemasCacheUnavailableException:
             abort(503, description='No schemas in cache and endpoint unavailable')
         except SchemasCatalogNotFoundException:

udata/core/dataset/api_fields.py

@@ -279,9 +279,28 @@
     'label': fields.String(description='The resource type display name')
 })
 
-
+# follow the specification of https://schema.data.gouv.fr/schemas/schemas.json
 catalog_schema_fields = api.model('CatalogSchema', {
-    'id': fields.String(description='The schema identifier'),
-    'label': fields.String(description='The schema display name'),
-    'versions': fields.List(fields.String, description='The available versions of the schema'),
+    'name': fields.String(),
+    'title': fields.String(),
+    'description': fields.String(),
+    'schema_url': fields.String(description="Often the link to the latest version"),
+    'schema_type': fields.String(enum=['tableschema', 'datapackage', 'jsonschema', 'other']),
+    'contact': fields.String(),
+    'examples': fields.List(fields.Nested(api.model('CatalogSchemaExample', {
+        'title': fields.String(),
+        'path': fields.String(),
+    }))),
+    'labels': fields.List(fields.String()),
+    'consolidation_dataset_id': fields.String(),
+    'versions': fields.List(fields.Nested(api.model('CatalogSchemaVersion', {
+        'version_name': fields.String(),
+        'schema_url': fields.String(),
+    }))),
+    'external_doc': fields.String(),
+    'external_tool': fields.String(description="Link to tools to create a file with this schema"),
+    'homepage': fields.String(),
+    'datapackage_title': fields.String(description="Only present if the schema is inside a datapackage"),
+    'datapackage_name': fields.String(description="Only present if the schema is inside a datapackage"),
+    'datapackage_description': fields.String(description="Only present if the schema is inside a datapackage"),
 })

udata/core/dataset/factories.py

@@ -81,25 +81,137 @@ def get_mock_data():
         return json.load(open(join(ROOT_DIR, 'tests', 'schemas.json')))
 
     @staticmethod
-    def get_expected_v1_result_from_mock_data():
-        return [
+    def get_all_schemas_from_mock_data(with_datapackage_info = True):
+        '''
+        with_datapackage_info is here to allow testing with or without marshalling (marshalling add None for inexistant datapackage_* fields)
+        '''
+        schemas = ResourceSchemaMockData.get_expected_assignable_schemas_from_mock_data(with_datapackage_info)
+
+        datapackage = {
+            "name": "etalab/schema-irve",
+            "title": "Infrastructures de recharges pour v\u00e9hicules \u00e9lectriques (IRVE)",
+            "description": "data package contenant 2 sch\u00e9mas : IRVE statique et IRVE dynamique",
+            "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve/latest/datapackage.json",
+            "schema_type": "datapackage",
+            "contact": "[email protected]",
+            "examples": [],
+            "labels": [
+                "Socle Commun des Donn\u00e9es Locales",
+                "transport.data.gouv.fr"
+            ],
+            "consolidation_dataset_id": "5448d3e0c751df01f85d0572",
+            "versions": [
+                {
+                    "version_name": "2.2.0",
+                    "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve/2.2.0/datapackage.json"
+                },
+                {
+                    "version_name": "2.2.1",
+                    "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve/2.2.1/datapackage.json"
+                },
+                {
+                    "version_name": "2.3.0",
+                    "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve/2.3.0/datapackage.json"
+                },
+                {
+                    "version_name": "2.3.1",
+                    "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve/2.3.1/datapackage.json"
+                }
+            ],
+            "external_doc": "https://doc.transport.data.gouv.fr/producteurs/infrastructures-de-recharge-de-vehicules-electriques-irve",
+            "external_tool": None,
+            "homepage": "https://github.com/etalab/schema-irve.git",
+        }
+
+        if with_datapackage_info:
+            datapackage["datapackage_title"] = None
+            datapackage["datapackage_name"] = None
+            datapackage["datapackage_description"] = None
+
+        return [datapackage] + schemas
+
+    @staticmethod
+    def get_expected_assignable_schemas_from_mock_data(with_datapackage_info = True):
+        '''
+        with_datapackage_info is here to allow testing with or without marshalling (marshalling add None for inexistant datapackage_* fields)
+        '''
+        schemas = [
             {
-                "id": "etalab/schema-irve-statique",
-                "label": "IRVE statique",
+                "name": "etalab/schema-irve-statique",
+                "title": "IRVE statique",
+                "description": "Sp\u00e9cification du fichier d'\u00e9change relatif aux donn\u00e9es concernant la localisation g\u00e9ographique et les caract\u00e9ristiques techniques des stations et des points de recharge pour v\u00e9hicules \u00e9lectriques",
+                "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve-statique/latest/schema-statique.json",
+                "schema_type": "tableschema",
+                "contact": "[email protected]",
+                "examples": [
+                    {
+                    "title": "Exemple de fichier IRVE valide",
+                    "path": "https://github.com/etalab/schema-irve/raw/v2.2.1/exemple-valide.csv"
+                    }
+                ],
+                "labels": [
+                    "Socle Commun des Donn\u00e9es Locales",
+                    "transport.data.gouv.fr"
+                ],
+                "consolidation_dataset_id": "5448d3e0c751df01f85d0572",
                 "versions": [
-                    "2.2.0",
-                    "2.2.1"
-                ]
+                    {
+                        "version_name": "2.2.0",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve-statique/2.2.0/schema-statique.json"
+                    },
+                    {
+                        "version_name": "2.2.1",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/etalab/schema-irve-statique/2.2.1/schema-statique.json"
+                    }
+                ],
+                "external_doc": "https://doc.transport.data.gouv.fr/producteurs/infrastructures-de-recharge-de-vehicules-electriques-irve",
+                "external_tool": None,
+                "homepage": "https://github.com/etalab/schema-irve.git",
+                "datapackage_title": "Infrastructures de recharges pour v\u00e9hicules \u00e9lectriques (IRVE)",
+                "datapackage_name": "etalab/schema-irve",
+                "datapackage_description": "data package contenant 2 sch\u00e9mas : IRVE statique et IRVE dynamique"
             },
             {
-                "id": "139bercy/format-commande-publique",
-                "label": "Données essentielles des marchés publics français",
+                "name": "139bercy/format-commande-publique",
+                "title": "Donn\u00e9es essentielles des march\u00e9s publics fran\u00e7ais",
+                "description": "Donn\u00e9es des attributions de march\u00e9s publics et de contrats de concessions sup\u00e9rieures \u00e0 40 000 euros.",
+                "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/latest/marches.json",
+                "schema_type": "jsonschema",
+                "contact": "[email protected]",
+                "examples": [],
+                "labels": [],
+                "consolidation_dataset_id": None,
                 "versions": [
-                    "1.3.0",
-                    "1.4.0",
-                    "1.5.0",
-                    "2.0.0",
-                    "2.0.1"
-                ]
+                    {
+                        "version_name": "1.3.0",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/1.3.0/sch\u00e9mas/json/contrats-concessions.json"
+                    },
+                    {
+                        "version_name": "1.4.0",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/1.4.0/marches.json"
+                    },
+                    {
+                        "version_name": "1.5.0",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/1.5.0/marches.json"
+                    },
+                    {
+                        "version_name": "2.0.0",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/2.0.0/marches.json"
+                    },
+                    {
+                        "version_name": "2.0.1",
+                        "schema_url": "https://schema.data.gouv.fr/schemas/139bercy/format-commande-publique/2.0.1/marches.json"
+                    }
+                ],
+                "external_doc": None,
+                "external_tool": None,
+                "homepage": "https://github.com/139bercy/format-commande-publique",
             }
-        ]
+        ]
+
+        if with_datapackage_info:
+            schemas[1]["datapackage_title"] = None
+            schemas[1]["datapackage_name"] = None
+            schemas[1]["datapackage_description"] = None
+
+        return schemas