From df5ee2c52233ec5c42837fb3305e3f9689412dac Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Thu, 29 Feb 2024 15:48:04 -0600 Subject: [PATCH 1/8] Add notebook for LAS to COPC Conversion --- copc/environment.yml | 12 + copc/lidar-las-to-copc.ipynb | 660 +++++++++++++++++++++++++++++++++++ 2 files changed, 672 insertions(+) create mode 100644 copc/environment.yml create mode 100644 copc/lidar-las-to-copc.ipynb diff --git a/copc/environment.yml b/copc/environment.yml new file mode 100644 index 0000000..a1eced7 --- /dev/null +++ b/copc/environment.yml @@ -0,0 +1,12 @@ +name: coguide-copc +channels: + - conda-forge +dependencies: + - python=3.11 + - earthaccess + - ipykernel + - jupyterlab + - matplotlib + - libgdal>=3.5 + - python-pdal + - pdal \ No newline at end of file diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb new file mode 100644 index 0000000..e080592 --- /dev/null +++ b/copc/lidar-las-to-copc.ipynb @@ -0,0 +1,660 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Converting LiDAR LAS Files to Cloud Optimized Point Clouds (COPCs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Environment\n", + "\n", + "The packages needed for this notebook can be installed with `conda` or `mamba`. Using the [`environment.yml` from this folder](./environment.yml) run:\n", + "\n", + "```bash\n", + "conda env create -f environment.yml\n", + "```\n", + "\n", + "or\n", + "\n", + "```bash\n", + "mamba env create -f environment.yml\n", + "```\n", + "\n", + "Finally, you may activate and select the kernel in the notebook (running in Jupyter)\n", + "\n", + "```bash\n", + "conda activate coguide-copc\n", + "```\n", + "\n", + "The notebook has been tested to work with the listed Conda environment." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "This tutorial will explore how to-\n", + "\n", + "1. Read a LiDAR LAS file using PDAL in Python\n", + "2. Convert the LiDAR LAS file to Cloud Optimized Point Cloud (COPC) format\n", + "2. Validate the generated COPC file\n", + "\n", + "## About the Dataset\n", + "\n", + "We will be using the [G-LiHT Lidar Point Cloud V001](http://doi.org/10.5067/Community/GLIHT/GLLIDARPC.001) from the NASA EarthData. To access NASA EarthData into Jupyter Notebook, you can create an account by visiting [NASA's Earthdata Login page](https://urs.earthdata.nasa.gov/users/new). This will enable you to register for an account and retrieve the datasets used in the notebook.\n", + "\n", + "We will use [earthaccess](https://github.com/nsidc/earthaccess) library to set up credentials to fetch data from NASA's EarthData catalog." + ] + }, + { + "cell_type": "code", + "execution_count": 86, + "metadata": {}, + "outputs": [], + "source": [ + "import earthaccess\n", + "import os\n", + "import pdal" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "earthaccess.login()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating a Data Directory for this Tutorial\n", + "\n", + "We are creating a data directory for downloading all the required files locally. " + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# set data directory path\n", + "data_dir = './data'\n", + "\n", + "# check if directory exists -> if directory doesn't exist, directory is created\n", + "if not os.path.exists(data_dir):\n", + " os.mkdir(data_dir)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Downloading the Dataset from EarthData\n", + "\n", + "We are using `search_data` method from the `earthaccess` module for searching the Granules from the selected collection. The `temporal` argument defines the temporal range for " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Granules found: 72\n" + ] + } + ], + "source": [ + "# Search Granules\n", + "short_name = 'GLLIDARPC'\n", + "version = '001'\n", + "\n", + "las_item_results = earthaccess.search_data(\n", + " short_name=short_name,\n", + " version=version,\n", + " temporal = (\"2020\"), \n", + " count=3\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Collection: {'EntryTitle': 'G-LiHT Lidar Point Cloud V001'}\n", + " Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'GPolygons': [{'Boundary': {'Points': [{'Longitude': -81.03452828650298, 'Latitude': 25.50220025425373}, {'Longitude': -81.01391715300757, 'Latitude': 25.50220365895999}, {'Longitude': -81.01391819492625, 'Latitude': 25.5112430715201}, {'Longitude': -81.03453087148995, 'Latitude': 25.511239665437053}, {'Longitude': -81.03452828650298, 'Latitude': 25.50220025425373}]}}]}}}\n", + " Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2020-03-11T04:00:00.000Z', 'EndingDateTime': '2020-03-12T03:59:59.000Z'}}\n", + " Size(MB): 238.623\n", + " Data: ['https://e4ftl01.cr.usgs.gov//GWELD1/COMMUNITY/GLLIDARPC.001/2020.03.11/GLLIDARPC_FL_20200311_FIA8_l0s47.las'],\n", + " Collection: {'EntryTitle': 'G-LiHT Lidar Point Cloud V001'}\n", + " Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'GPolygons': [{'Boundary': {'Points': [{'Longitude': -81.02242648723991, 'Latitude': 25.493163090615468}, {'Longitude': -80.99410838333016, 'Latitude': 25.49316468678571}, {'Longitude': -80.99410794242846, 'Latitude': 25.502204110708817}, {'Longitude': -81.02242816553566, 'Latitude': 25.50220251389295}, {'Longitude': -81.02242648723991, 'Latitude': 25.493163090615468}]}}]}}}\n", + " Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2020-03-11T04:00:00.000Z', 'EndingDateTime': '2020-03-12T03:59:59.000Z'}}\n", + " Size(MB): 248.383\n", + " Data: ['https://e4ftl01.cr.usgs.gov//GWELD1/COMMUNITY/GLLIDARPC.001/2020.03.11/GLLIDARPC_FL_20200311_FIA8_l0s46.las'],\n", + " Collection: {'EntryTitle': 'G-LiHT Lidar Point Cloud V001'}\n", + " Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'GPolygons': [{'Boundary': {'Points': [{'Longitude': -80.94099075054905, 'Latitude': 25.276201329530473}, {'Longitude': -80.9355627247816, 'Latitude': 25.276199059361314}, {'Longitude': -80.9355579494582, 'Latitude': 25.285238744206318}, {'Longitude': -80.94098637748567, 'Latitude': 25.285241015299494}, {'Longitude': -80.94099075054905, 'Latitude': 25.276201329530473}]}}]}}}\n", + " Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2020-03-11T04:00:00.000Z', 'EndingDateTime': '2020-03-12T03:59:59.000Z'}}\n", + " Size(MB): 91.0422\n", + " Data: ['https://e4ftl01.cr.usgs.gov//GWELD1/COMMUNITY/GLLIDARPC.001/2020.03.11/GLLIDARPC_FL_20200311_FIA8_l0s22.las']]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "las_item_results" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's use the file with size 91.04 MB and convert it to a COPC format. " + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Getting 1 granules, approx download size: 0.09 GB\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 1736.05it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "File GLLIDARPC_FL_20200311_FIA8_l0s22.las already downloaded\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 3788.89it/s]\n", + "COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 7096.96it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "data/GLLIDARPC_FL_20200311_FIA8_l0s22.las\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "# Download Data - Selecting the 3rd file from the `las_item_results` list\n", + "gliht_las_file = earthaccess.download(las_item_results[2], data_dir)\n", + "las_filename = f\"{gliht_las_file[0]}\"statssta\n", + "print(las_filename)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## A Brief Introduction to PDAL \n", + "\n", + "For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/2.6.0/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/2.6.0/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/2.6.0/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/2.6.0/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/2.6.0/pipeline.html#stage-object).\n", + "\n", + "#### PDAL Pipelines\n", + "A pdal pipeline is defined in a JSON format either as a JSON object or a JSON array. Below is an example of a pdal pipeline taking a `.las` file as input, generating `stats` and writing it to a COPC format. \n", + "\n", + "```json\n", + "{\n", + " \"pipeline\": [\n", + " {\n", + " \"filename\":las_filename,\n", + " \"type\":\"readers.las\"\n", + " },\n", + " {\n", + " \"type\":\"filters.stats\",\n", + " },\n", + " {\n", + " \"type\":\"writers.copc\",\n", + " \"filename\":copc_filename\n", + " }\n", + "]\n", + "}\n", + "```\n", + "\n", + "This pipeline can be executed using the `pdal pipeline ` from the command line for a pipeline saved as a local `JSON` file. \n", + "\n", + "#### Programmatic Pipeline Construction\n", + "\n", + "However, here we will explore a comparatively easier and Pythonic approach to define a pipeline and execute it. This is based on the [PDAL Python extension](https://pypi.org/project/pdal/) which provides a programmatic pipeline construction approach in addition to the simple pipeline construction approach discussed above. \n", + "\n", + "This approach utilizes the `|` operator to pipe various stages together representing a pipeline. For eg., the above pipeline can be represented as -\n", + "\n", + "```python\n", + "pipeline = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename) | pdal.Filter.stats()\n", + "```\n", + "This pipeline can be executed using `pipeline.execute`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## LAS to COPC Conversion\n", + "\n", + "Now, let's dive into converting the LAS file to a COPC format based on the programmatic pipeline construction. " + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": {}, + "outputs": [], + "source": [ + "# Defining output filename. Usually, COPC files are saved as .copc.laz\n", + "copc_filename = las_filename.split('.')[0]+'.copc.laz'" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3409439\n" + ] + } + ], + "source": [ + "# pipe = stage 1 | stage 2 | stage 3\n", + "# Or, pipeline = pipeline 1 | stage 2\n", + "\n", + "pipe = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename)\n", + "print(pipe.execute())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Validation\n", + "\n", + "As we can see from output of the below cell, the `.copc.laz` file is created in the destination directory." + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 253960\n", + "-rw-r--r-- 1 26M Feb 29 14:05 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n", + "-rw-r--r-- 1 91M Feb 29 11:27 GLLIDARPC_FL_20200311_FIA8_l0s22.las\n" + ] + } + ], + "source": [ + "# using -go for removing user details and h for getting memory size in MBs\n", + "!ls -goh {os.path.dirname(copc_filename)}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's read the created COPC file again and explore it's stats." + ] + }, + { + "cell_type": "code", + "execution_count": 83, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'metadata': {'filters.stats': {'bbox': {'EPSG:4326': {'bbox': {'maxx': -80.93555795, 'maxy': 25.28524102, 'maxz': 69.99, 'minx': -80.94099075, 'miny': 25.27619906, 'minz': -12.54}, 'boundary': {'type': 'Polygon', 'coordinates': [[[-80.94099075054905, 25.276201329530473, -12.54], [-80.94098637748567, 25.285241015299494, -12.54], [-80.9355579494582, 25.285238744206318, 69.99], [-80.9355627247816, 25.276199059361314, 69.99], [-80.94099075054905, 25.276201329530473, -12.54]]]}}, 'native': {'bbox': {'maxx': 506487.7363, 'maxy': 2796533.993, 'maxz': 69.99, 'minx': 505941.2263, 'miny': 2795533.003, 'minz': -12.54}, 'boundary': {'type': 'Polygon', 'coordinates': [[[505941.22630256764, 2795533.0032408433, -12.54], [505941.22630256764, 2796533.993240843, -12.54], [506487.73630256765, 2796533.993240843, 69.99], [506487.73630256765, 2795533.0032408433, 69.99], [505941.22630256764, 2795533.0032408433, -12.54]]]}}}, 'statistic': [{'average': 506237.8598, 'count': 3409439, 'maximum': 506487.7363, 'minimum': 505941.2263, 'name': 'X', 'position': 0, 'stddev': 101.3857552, 'variance': 10279.07135}, {'average': 2795977.637, 'count': 3409439, 'maximum': 2796533.993, 'minimum': 2795533.003, 'name': 'Y', 'position': 1, 'stddev': 274.313888, 'variance': 75248.10912}, {'average': 2.192797205, 'count': 3409439, 'maximum': 69.99, 'minimum': -12.54, 'name': 'Z', 'position': 2, 'stddev': 1.788122887, 'variance': 3.197383461}, {'average': 30205.96606, 'count': 3409439, 'maximum': 65535, 'minimum': 14789, 'name': 'Intensity', 'position': 3, 'stddev': 5497.346879, 'variance': 30220822.71}, {'average': 1.200336478, 'count': 3409439, 'maximum': 5, 'minimum': 1, 'name': 'ReturnNumber', 'position': 4, 'stddev': 0.4267302243, 'variance': 0.1820986843}, {'average': 1.400683808, 'count': 3409439, 'maximum': 5, 'minimum': 1, 'name': 'NumberOfReturns', 'position': 5, 'stddev': 0.5529822902, 'variance': 0.3057894133}, {'average': 0.5248757933, 'count': 3409439, 'maximum': 1, 'minimum': 0, 'name': 'ScanDirectionFlag', 'position': 6, 'stddev': 0.4993808847, 'variance': 0.249381268}, {'average': 0.002420339534, 'count': 3409439, 'maximum': 1, 'minimum': 0, 'name': 'EdgeOfFlightLine', 'position': 7, 'stddev': 0.04913738087, 'variance': 0.002414482199}, {'average': 1.239596602, 'count': 3409439, 'maximum': 2, 'minimum': 1, 'name': 'Classification', 'position': 8, 'stddev': 0.4268373506, 'variance': 0.1821901239}, {'average': 2.569738893, 'count': 3409439, 'maximum': 33, 'minimum': -31, 'name': 'ScanAngleRank', 'position': 9, 'stddev': 16.33805559, 'variance': 266.9320603}, {'average': 1.475124207, 'count': 3409439, 'maximum': 2, 'minimum': 1, 'name': 'UserData', 'position': 10, 'stddev': 0.4993808847, 'variance': 0.249381268}, {'average': 192.5227238, 'count': 3409439, 'maximum': 65535, 'minimum': 0, 'name': 'PointSourceId', 'position': 11, 'stddev': 680.5012031, 'variance': 463081.8874}, {'average': 310476.1839, 'count': 3409439, 'maximum': 310485.0477, 'minimum': 310469.0825, 'name': 'GpsTime', 'position': 12, 'stddev': 4.240892261, 'variance': 17.98516717}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Synthetic', 'position': 13, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'KeyPoint', 'position': 14, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Withheld', 'position': 15, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Overlap', 'position': 16, 'stddev': 0, 'variance': 0}]}, 'readers.las': {'comp_spatialreference': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'compressed': False, 'copc': False, 'count': 3409439, 'creation_doy': 28, 'creation_year': 2021, 'dataformat_id': 1, 'dataoffset': 399, 'filesource_id': 0, 'gtiff': 'Geotiff_Information:\\n Version: 1\\n Key_Revision: 1.0\\n Tagged_Information:\\n End_Of_Tags.\\n Keyed_Information:\\n Unknown-33922 (Double,1): 0 \\n GTModelTypeGeoKey (Short,1): ModelTypeProjected\\n GTRasterTypeGeoKey (Short,1): RasterPixelIsArea\\n GeogLinearUnitsGeoKey (Short,1): Linear_Meter\\n GeogAngularUnitsGeoKey (Short,1): Angular_Degree\\n ProjectedCSTypeGeoKey (Short,1): PCS_WGS84_UTM_zone_17N\\n End_Of_Keys.\\n End_Of_Geotiff.\\n', 'header_size': 227, 'major_version': 1, 'maxx': 506487.7363, 'maxy': 2796533.993, 'maxz': 69.99, 'minor_version': 1, 'minx': 505941.2263, 'miny': 2795533.003, 'minz': -12.54, 'offset_x': 458435.286302568, 'offset_y': 2776879.90324084, 'offset_z': 0, 'point_length': 28, 'project_id': '00000000-0000-0000-0000-000000000000', 'scale_x': 0.01, 'scale_y': 0.01, 'scale_z': 0.01, 'software_id': 'BCAL LidarTools, IDL 8.5.1', 'spatialreference': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'srs': {'compoundwkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'horizontal': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'isgeocentric': False, 'isgeographic': False, 'json': {'type': 'ProjectedCRS', 'name': 'WGS 84 / UTM zone 17N', 'base_crs': {'name': 'WGS 84', 'datum': {'type': 'GeodeticReferenceFrame', 'name': 'World Geodetic System 1984', 'ellipsoid': {'name': 'WGS 84', 'semi_major_axis': 6378137, 'inverse_flattening': 298.257223563}}, 'coordinate_system': {'subtype': 'ellipsoidal', 'axis': [{'name': 'Geodetic latitude', 'abbreviation': 'Lat', 'direction': 'north', 'unit': 'degree'}, {'name': 'Geodetic longitude', 'abbreviation': 'Lon', 'direction': 'east', 'unit': 'degree'}]}, 'id': {'authority': 'EPSG', 'code': 4326}}, 'conversion': {'name': 'UTM zone 17N', 'method': {'name': 'Transverse Mercator', 'id': {'authority': 'EPSG', 'code': 9807}}, 'parameters': [{'name': 'Latitude of natural origin', 'value': 0, 'unit': 'degree', 'id': {'authority': 'EPSG', 'code': 8801}}, {'name': 'Longitude of natural origin', 'value': -81, 'unit': 'degree', 'id': {'authority': 'EPSG', 'code': 8802}}, {'name': 'Scale factor at natural origin', 'value': 0.9996, 'unit': 'unity', 'id': {'authority': 'EPSG', 'code': 8805}}, {'name': 'False easting', 'value': 500000, 'unit': 'metre', 'id': {'authority': 'EPSG', 'code': 8806}}, {'name': 'False northing', 'value': 0, 'unit': 'metre', 'id': {'authority': 'EPSG', 'code': 8807}}]}, 'coordinate_system': {'subtype': 'Cartesian', 'axis': [{'name': 'Easting', 'abbreviation': '', 'direction': 'east', 'unit': 'metre'}, {'name': 'Northing', 'abbreviation': '', 'direction': 'north', 'unit': 'metre'}]}, 'id': {'authority': 'EPSG', 'code': 32617}}, 'prettycompoundwkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",\\n GEOGCS[\"WGS 84\",\\n DATUM[\"WGS_1984\",\\n SPHEROID[\"WGS 84\",6378137,298.257223563,\\n AUTHORITY[\"EPSG\",\"7030\"]],\\n AUTHORITY[\"EPSG\",\"6326\"]],\\n PRIMEM[\"Greenwich\",0,\\n AUTHORITY[\"EPSG\",\"8901\"]],\\n UNIT[\"degree\",0.0174532925199433,\\n AUTHORITY[\"EPSG\",\"9122\"]],\\n AUTHORITY[\"EPSG\",\"4326\"]],\\n PROJECTION[\"Transverse_Mercator\"],\\n PARAMETER[\"latitude_of_origin\",0],\\n PARAMETER[\"central_meridian\",-81],\\n PARAMETER[\"scale_factor\",0.9996],\\n PARAMETER[\"false_easting\",500000],\\n PARAMETER[\"false_northing\",0],\\n UNIT[\"metre\",1,\\n AUTHORITY[\"EPSG\",\"9001\"]],\\n AXIS[\"Easting\",EAST],\\n AXIS[\"Northing\",NORTH],\\n AUTHORITY[\"EPSG\",\"32617\"]]', 'prettywkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",\\n GEOGCS[\"WGS 84\",\\n DATUM[\"WGS_1984\",\\n SPHEROID[\"WGS 84\",6378137,298.257223563,\\n AUTHORITY[\"EPSG\",\"7030\"]],\\n AUTHORITY[\"EPSG\",\"6326\"]],\\n PRIMEM[\"Greenwich\",0,\\n AUTHORITY[\"EPSG\",\"8901\"]],\\n UNIT[\"degree\",0.0174532925199433,\\n AUTHORITY[\"EPSG\",\"9122\"]],\\n AUTHORITY[\"EPSG\",\"4326\"]],\\n PROJECTION[\"Transverse_Mercator\"],\\n PARAMETER[\"latitude_of_origin\",0],\\n PARAMETER[\"central_meridian\",-81],\\n PARAMETER[\"scale_factor\",0.9996],\\n PARAMETER[\"false_easting\",500000],\\n PARAMETER[\"false_northing\",0],\\n UNIT[\"metre\",1,\\n AUTHORITY[\"EPSG\",\"9001\"]],\\n AXIS[\"Easting\",EAST],\\n AXIS[\"Northing\",NORTH],\\n AUTHORITY[\"EPSG\",\"32617\"]]', 'proj4': '+proj=utm +zone=17 +datum=WGS84 +units=m +no_defs', 'units': {'horizontal': 'metre', 'vertical': ''}, 'vertical': '', 'wkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]'}, 'system_id': 'NASA G-LiHT v2.0, Riegl VQ-480', 'vlr_0': {'data': 'AQABAAAABgCChLCHAQAAAAAEAAABAAEAAQQAAAEAAQAECAAAAQApIwYIAAABAI4jAAwAAAEAaX8=', 'description': 'GeoKeyDirectoryTag', 'record_id': 34735, 'user_id': 'LASF_Projection'}, 'vlr_1': {'data': 'AAAAAAAAAAA=', 'description': 'GeoKeyDoubleParamsTag', 'record_id': 34736, 'user_id': 'LASF_Projection'}}}}\n" + ] + } + ], + "source": [ + "valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n", + "valid_pipe.execute()\n", + "print(valid_pipe.metadata)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Similarly, we can get `metadata`, `log` and `array values` from the executed pipeline using `valid_pipe.metadata`, `valid_pipe.log` and `valid_pipe.arrays`. The readers are encouraged to explore the results of these operations on their own. \n", + "\n", + "Additionally, we can run `pdal info ` on the command line for getting information from the generated COPC file." + ] + }, + { + "cell_type": "code", + "execution_count": 89, + "metadata": { + "vscode": { + "languageId": "shellscript" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"file_size\": 27367236,\n", + " \"filename\": \"data/GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\",\n", + " \"now\": \"2024-02-29T15:43:42-0600\",\n", + " \"pdal_version\": \"2.6.3 (git-version: Release)\",\n", + " \"reader\": \"readers.copc\",\n", + " \"stats\":\n", + " {\n", + " \"bbox\":\n", + " {\n", + " \"EPSG:4326\":\n", + " {\n", + " \"bbox\":\n", + " {\n", + " \"maxx\": -80.93555791,\n", + " \"maxy\": 25.28524099,\n", + " \"maxz\": 69.99,\n", + " \"minx\": -80.94099071,\n", + " \"miny\": 25.27619903,\n", + " \"minz\": -12.54\n", + " },\n", + " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ -80.94099071383971, 25.276201300248541, -12.54 ], [ -80.940986340773605, 25.285240986017602, -12.54 ], [ -80.935557912747441, 25.285238714923079, 69.99 ], [ -80.93556268807356, 25.276199030078029, 69.99 ], [ -80.94099071383971, 25.276201300248541, -12.54 ] ] ] }\n", + " },\n", + " \"native\":\n", + " {\n", + " \"bbox\":\n", + " {\n", + " \"maxx\": 506487.74,\n", + " \"maxy\": 2796533.99,\n", + " \"maxz\": 69.99,\n", + " \"minx\": 505941.23,\n", + " \"miny\": 2795533,\n", + " \"minz\": -12.54\n", + " },\n", + " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ 505941.230000000039581, 2795533.0, -12.54 ], [ 505941.230000000039581, 2796533.990000000223517, -12.54 ], [ 506487.74, 2796533.990000000223517, 69.99 ], [ 506487.74, 2795533.0, 69.99 ], [ 505941.230000000039581, 2795533.0, -12.54 ] ] ] }\n", + " }\n", + " },\n", + " \"statistic\":\n", + " [\n", + " {\n", + " \"average\": 506237.8635,\n", + " \"count\": 3409439,\n", + " \"maximum\": 506487.74,\n", + " \"minimum\": 505941.23,\n", + " \"name\": \"X\",\n", + " \"position\": 0,\n", + " \"stddev\": 101.3857552,\n", + " \"variance\": 10279.07135\n", + " },\n", + " {\n", + " \"average\": 2795977.634,\n", + " \"count\": 3409439,\n", + " \"maximum\": 2796533.99,\n", + " \"minimum\": 2795533,\n", + " \"name\": \"Y\",\n", + " \"position\": 1,\n", + " \"stddev\": 274.3138881,\n", + " \"variance\": 75248.10919\n", + " },\n", + " {\n", + " \"average\": 2.192797205,\n", + " \"count\": 3409439,\n", + " \"maximum\": 69.99,\n", + " \"minimum\": -12.54,\n", + " \"name\": \"Z\",\n", + " \"position\": 2,\n", + " \"stddev\": 1.788122887,\n", + " \"variance\": 3.197383461\n", + " },\n", + " {\n", + " \"average\": 30205.96606,\n", + " \"count\": 3409439,\n", + " \"maximum\": 65535,\n", + " \"minimum\": 14789,\n", + " \"name\": \"Intensity\",\n", + " \"position\": 3,\n", + " \"stddev\": 5497.346879,\n", + " \"variance\": 30220822.71\n", + " },\n", + " {\n", + " \"average\": 1.200336478,\n", + " \"count\": 3409439,\n", + " \"maximum\": 5,\n", + " \"minimum\": 1,\n", + " \"name\": \"ReturnNumber\",\n", + " \"position\": 4,\n", + " \"stddev\": 0.4267302243,\n", + " \"variance\": 0.1820986843\n", + " },\n", + " {\n", + " \"average\": 1.400683808,\n", + " \"count\": 3409439,\n", + " \"maximum\": 5,\n", + " \"minimum\": 1,\n", + " \"name\": \"NumberOfReturns\",\n", + " \"position\": 5,\n", + " \"stddev\": 0.5529822902,\n", + " \"variance\": 0.3057894133\n", + " },\n", + " {\n", + " \"average\": 0.5248757933,\n", + " \"count\": 3409439,\n", + " \"maximum\": 1,\n", + " \"minimum\": 0,\n", + " \"name\": \"ScanDirectionFlag\",\n", + " \"position\": 6,\n", + " \"stddev\": 0.4993808847,\n", + " \"variance\": 0.249381268\n", + " },\n", + " {\n", + " \"average\": 0.002420339534,\n", + " \"count\": 3409439,\n", + " \"maximum\": 1,\n", + " \"minimum\": 0,\n", + " \"name\": \"EdgeOfFlightLine\",\n", + " \"position\": 7,\n", + " \"stddev\": 0.04913738087,\n", + " \"variance\": 0.002414482199\n", + " },\n", + " {\n", + " \"average\": 1.239596602,\n", + " \"count\": 3409439,\n", + " \"maximum\": 2,\n", + " \"minimum\": 1,\n", + " \"name\": \"Classification\",\n", + " \"position\": 8,\n", + " \"stddev\": 0.4268373506,\n", + " \"variance\": 0.1821901239\n", + " },\n", + " {\n", + " \"average\": 2.569735542,\n", + " \"count\": 3409439,\n", + " \"maximum\": 33,\n", + " \"minimum\": -31.00200081,\n", + " \"name\": \"ScanAngleRank\",\n", + " \"position\": 9,\n", + " \"stddev\": 16.33805538,\n", + " \"variance\": 266.9320534\n", + " },\n", + " {\n", + " \"average\": 1.475124207,\n", + " \"count\": 3409439,\n", + " \"maximum\": 2,\n", + " \"minimum\": 1,\n", + " \"name\": \"UserData\",\n", + " \"position\": 10,\n", + " \"stddev\": 0.4993808847,\n", + " \"variance\": 0.249381268\n", + " },\n", + " {\n", + " \"average\": 192.5227238,\n", + " \"count\": 3409439,\n", + " \"maximum\": 65535,\n", + " \"minimum\": 0,\n", + " \"name\": \"PointSourceId\",\n", + " \"position\": 11,\n", + " \"stddev\": 680.5012031,\n", + " \"variance\": 463081.8874\n", + " },\n", + " {\n", + " \"average\": 310476.1839,\n", + " \"count\": 3409439,\n", + " \"maximum\": 310485.0477,\n", + " \"minimum\": 310469.0825,\n", + " \"name\": \"GpsTime\",\n", + " \"position\": 12,\n", + " \"stddev\": 4.240892261,\n", + " \"variance\": 17.98516717\n", + " },\n", + " {\n", + " \"average\": 0,\n", + " \"count\": 3409439,\n", + " \"maximum\": 0,\n", + " \"minimum\": 0,\n", + " \"name\": \"ScanChannel\",\n", + " \"position\": 13,\n", + " \"stddev\": 0,\n", + " \"variance\": 0\n", + " },\n", + " {\n", + " \"average\": 0,\n", + " \"count\": 3409439,\n", + " \"maximum\": 0,\n", + " \"minimum\": 0,\n", + " \"name\": \"Synthetic\",\n", + " \"position\": 14,\n", + " \"stddev\": 0,\n", + " \"variance\": 0\n", + " },\n", + " {\n", + " \"average\": 0,\n", + " \"count\": 3409439,\n", + " \"maximum\": 0,\n", + " \"minimum\": 0,\n", + " \"name\": \"KeyPoint\",\n", + " \"position\": 15,\n", + " \"stddev\": 0,\n", + " \"variance\": 0\n", + " },\n", + " {\n", + " \"average\": 0,\n", + " \"count\": 3409439,\n", + " \"maximum\": 0,\n", + " \"minimum\": 0,\n", + " \"name\": \"Withheld\",\n", + " \"position\": 16,\n", + " \"stddev\": 0,\n", + " \"variance\": 0\n", + " },\n", + " {\n", + " \"average\": 0,\n", + " \"count\": 3409439,\n", + " \"maximum\": 0,\n", + " \"minimum\": 0,\n", + " \"name\": \"Overlap\",\n", + " \"position\": 17,\n", + " \"stddev\": 0,\n", + " \"variance\": 0\n", + " }\n", + " ]\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "!pdal info {copc_filename}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the above metadata, it is important to note the line `\"reader\": \"readers.copc\"` which validates that the generated file is a valida COPC file. " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "base", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 23c00d36b6f42e5f9213e652a1d415c7a732f46c Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Fri, 22 Mar 2024 11:19:25 -0500 Subject: [PATCH 2/8] Adding cli based access info --- copc/lidar-las-to-copc.ipynb | 443 +++++++++++++++++++---------------- 1 file changed, 245 insertions(+), 198 deletions(-) diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb index e080592..23ca61b 100644 --- a/copc/lidar-las-to-copc.ipynb +++ b/copc/lidar-las-to-copc.ipynb @@ -55,9 +55,18 @@ }, { "cell_type": "code", - "execution_count": 86, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/opt/homebrew/anaconda3/envs/coguide-copc/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + } + ], "source": [ "import earthaccess\n", "import os\n", @@ -66,16 +75,16 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 3, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } @@ -95,7 +104,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -118,7 +127,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 7, "metadata": {}, "outputs": [ { @@ -144,7 +153,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "metadata": {}, "outputs": [ { @@ -167,7 +176,7 @@ " Data: ['https://e4ftl01.cr.usgs.gov//GWELD1/COMMUNITY/GLLIDARPC.001/2020.03.11/GLLIDARPC_FL_20200311_FIA8_l0s22.las']]" ] }, - "execution_count": 6, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -185,7 +194,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -199,7 +208,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 1736.05it/s]\n" + "QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 2236.96it/s]\n" ] }, { @@ -213,8 +222,8 @@ "name": "stderr", "output_type": "stream", "text": [ - "PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 3788.89it/s]\n", - "COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 7096.96it/s]" + "PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 3968.12it/s]\n", + "COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 12787.51it/s]" ] }, { @@ -235,7 +244,7 @@ "source": [ "# Download Data - Selecting the 3rd file from the `las_item_results` list\n", "gliht_las_file = earthaccess.download(las_item_results[2], data_dir)\n", - "las_filename = f\"{gliht_las_file[0]}\"statssta\n", + "las_filename = f\"{gliht_las_file[0]}\"\n", "print(las_filename)" ] }, @@ -245,166 +254,28 @@ "source": [ "## A Brief Introduction to PDAL \n", "\n", - "For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/2.6.0/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/2.6.0/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/2.6.0/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/2.6.0/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/2.6.0/pipeline.html#stage-object).\n", - "\n", - "#### PDAL Pipelines\n", - "A pdal pipeline is defined in a JSON format either as a JSON object or a JSON array. Below is an example of a pdal pipeline taking a `.las` file as input, generating `stats` and writing it to a COPC format. \n", - "\n", - "```json\n", - "{\n", - " \"pipeline\": [\n", - " {\n", - " \"filename\":las_filename,\n", - " \"type\":\"readers.las\"\n", - " },\n", - " {\n", - " \"type\":\"filters.stats\",\n", - " },\n", - " {\n", - " \"type\":\"writers.copc\",\n", - " \"filename\":copc_filename\n", - " }\n", - "]\n", - "}\n", - "```\n", - "\n", - "This pipeline can be executed using the `pdal pipeline ` from the command line for a pipeline saved as a local `JSON` file. \n", - "\n", - "#### Programmatic Pipeline Construction\n", - "\n", - "However, here we will explore a comparatively easier and Pythonic approach to define a pipeline and execute it. This is based on the [PDAL Python extension](https://pypi.org/project/pdal/) which provides a programmatic pipeline construction approach in addition to the simple pipeline construction approach discussed above. \n", - "\n", - "This approach utilizes the `|` operator to pipe various stages together representing a pipeline. For eg., the above pipeline can be represented as -\n", + "[PDAL](https://pdal.io/) (Point Data Abstraction Library) is a C/C++ based open-source library for processing point cloud data. It allows chained operations for processing point clouds. Additionally, it also has a PDAL-Python wrapper to work in a Pythonic environment. \n", "\n", - "```python\n", - "pipeline = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename) | pdal.Filter.stats()\n", - "```\n", - "This pipeline can be executed using `pipeline.execute`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## LAS to COPC Conversion\n", + "#### Accessing and Getting Metadata Information\n", "\n", - "Now, let's dive into converting the LAS file to a COPC format based on the programmatic pipeline construction. " - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "metadata": {}, - "outputs": [], - "source": [ - "# Defining output filename. Usually, COPC files are saved as .copc.laz\n", - "copc_filename = las_filename.split('.')[0]+'.copc.laz'" + "PDAL CLI provides multiple [applications](https://pdal.io/en/2.7.0/apps/index.html) for processing point clouds. Similar to `gdal info` for TIFFs, we can run `pdal info ` on the command line for getting metadata from a point cloud file without reading it in memory. " ] }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 37, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "3409439\n" - ] - } - ], - "source": [ - "# pipe = stage 1 | stage 2 | stage 3\n", - "# Or, pipeline = pipeline 1 | stage 2\n", - "\n", - "pipe = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename)\n", - "print(pipe.execute())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Validation\n", - "\n", - "As we can see from output of the below cell, the `.copc.laz` file is created in the destination directory." - ] - }, - { - "cell_type": "code", - "execution_count": 73, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 253960\n", - "-rw-r--r-- 1 26M Feb 29 14:05 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n", - "-rw-r--r-- 1 91M Feb 29 11:27 GLLIDARPC_FL_20200311_FIA8_l0s22.las\n" - ] - } - ], - "source": [ - "# using -go for removing user details and h for getting memory size in MBs\n", - "!ls -goh {os.path.dirname(copc_filename)}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's read the created COPC file again and explore it's stats." - ] - }, - { - "cell_type": "code", - "execution_count": 83, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'metadata': {'filters.stats': {'bbox': {'EPSG:4326': {'bbox': {'maxx': -80.93555795, 'maxy': 25.28524102, 'maxz': 69.99, 'minx': -80.94099075, 'miny': 25.27619906, 'minz': -12.54}, 'boundary': {'type': 'Polygon', 'coordinates': [[[-80.94099075054905, 25.276201329530473, -12.54], [-80.94098637748567, 25.285241015299494, -12.54], [-80.9355579494582, 25.285238744206318, 69.99], [-80.9355627247816, 25.276199059361314, 69.99], [-80.94099075054905, 25.276201329530473, -12.54]]]}}, 'native': {'bbox': {'maxx': 506487.7363, 'maxy': 2796533.993, 'maxz': 69.99, 'minx': 505941.2263, 'miny': 2795533.003, 'minz': -12.54}, 'boundary': {'type': 'Polygon', 'coordinates': [[[505941.22630256764, 2795533.0032408433, -12.54], [505941.22630256764, 2796533.993240843, -12.54], [506487.73630256765, 2796533.993240843, 69.99], [506487.73630256765, 2795533.0032408433, 69.99], [505941.22630256764, 2795533.0032408433, -12.54]]]}}}, 'statistic': [{'average': 506237.8598, 'count': 3409439, 'maximum': 506487.7363, 'minimum': 505941.2263, 'name': 'X', 'position': 0, 'stddev': 101.3857552, 'variance': 10279.07135}, {'average': 2795977.637, 'count': 3409439, 'maximum': 2796533.993, 'minimum': 2795533.003, 'name': 'Y', 'position': 1, 'stddev': 274.313888, 'variance': 75248.10912}, {'average': 2.192797205, 'count': 3409439, 'maximum': 69.99, 'minimum': -12.54, 'name': 'Z', 'position': 2, 'stddev': 1.788122887, 'variance': 3.197383461}, {'average': 30205.96606, 'count': 3409439, 'maximum': 65535, 'minimum': 14789, 'name': 'Intensity', 'position': 3, 'stddev': 5497.346879, 'variance': 30220822.71}, {'average': 1.200336478, 'count': 3409439, 'maximum': 5, 'minimum': 1, 'name': 'ReturnNumber', 'position': 4, 'stddev': 0.4267302243, 'variance': 0.1820986843}, {'average': 1.400683808, 'count': 3409439, 'maximum': 5, 'minimum': 1, 'name': 'NumberOfReturns', 'position': 5, 'stddev': 0.5529822902, 'variance': 0.3057894133}, {'average': 0.5248757933, 'count': 3409439, 'maximum': 1, 'minimum': 0, 'name': 'ScanDirectionFlag', 'position': 6, 'stddev': 0.4993808847, 'variance': 0.249381268}, {'average': 0.002420339534, 'count': 3409439, 'maximum': 1, 'minimum': 0, 'name': 'EdgeOfFlightLine', 'position': 7, 'stddev': 0.04913738087, 'variance': 0.002414482199}, {'average': 1.239596602, 'count': 3409439, 'maximum': 2, 'minimum': 1, 'name': 'Classification', 'position': 8, 'stddev': 0.4268373506, 'variance': 0.1821901239}, {'average': 2.569738893, 'count': 3409439, 'maximum': 33, 'minimum': -31, 'name': 'ScanAngleRank', 'position': 9, 'stddev': 16.33805559, 'variance': 266.9320603}, {'average': 1.475124207, 'count': 3409439, 'maximum': 2, 'minimum': 1, 'name': 'UserData', 'position': 10, 'stddev': 0.4993808847, 'variance': 0.249381268}, {'average': 192.5227238, 'count': 3409439, 'maximum': 65535, 'minimum': 0, 'name': 'PointSourceId', 'position': 11, 'stddev': 680.5012031, 'variance': 463081.8874}, {'average': 310476.1839, 'count': 3409439, 'maximum': 310485.0477, 'minimum': 310469.0825, 'name': 'GpsTime', 'position': 12, 'stddev': 4.240892261, 'variance': 17.98516717}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Synthetic', 'position': 13, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'KeyPoint', 'position': 14, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Withheld', 'position': 15, 'stddev': 0, 'variance': 0}, {'average': 0, 'count': 3409439, 'maximum': 0, 'minimum': 0, 'name': 'Overlap', 'position': 16, 'stddev': 0, 'variance': 0}]}, 'readers.las': {'comp_spatialreference': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'compressed': False, 'copc': False, 'count': 3409439, 'creation_doy': 28, 'creation_year': 2021, 'dataformat_id': 1, 'dataoffset': 399, 'filesource_id': 0, 'gtiff': 'Geotiff_Information:\\n Version: 1\\n Key_Revision: 1.0\\n Tagged_Information:\\n End_Of_Tags.\\n Keyed_Information:\\n Unknown-33922 (Double,1): 0 \\n GTModelTypeGeoKey (Short,1): ModelTypeProjected\\n GTRasterTypeGeoKey (Short,1): RasterPixelIsArea\\n GeogLinearUnitsGeoKey (Short,1): Linear_Meter\\n GeogAngularUnitsGeoKey (Short,1): Angular_Degree\\n ProjectedCSTypeGeoKey (Short,1): PCS_WGS84_UTM_zone_17N\\n End_Of_Keys.\\n End_Of_Geotiff.\\n', 'header_size': 227, 'major_version': 1, 'maxx': 506487.7363, 'maxy': 2796533.993, 'maxz': 69.99, 'minor_version': 1, 'minx': 505941.2263, 'miny': 2795533.003, 'minz': -12.54, 'offset_x': 458435.286302568, 'offset_y': 2776879.90324084, 'offset_z': 0, 'point_length': 28, 'project_id': '00000000-0000-0000-0000-000000000000', 'scale_x': 0.01, 'scale_y': 0.01, 'scale_z': 0.01, 'software_id': 'BCAL LidarTools, IDL 8.5.1', 'spatialreference': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'srs': {'compoundwkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'horizontal': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]', 'isgeocentric': False, 'isgeographic': False, 'json': {'type': 'ProjectedCRS', 'name': 'WGS 84 / UTM zone 17N', 'base_crs': {'name': 'WGS 84', 'datum': {'type': 'GeodeticReferenceFrame', 'name': 'World Geodetic System 1984', 'ellipsoid': {'name': 'WGS 84', 'semi_major_axis': 6378137, 'inverse_flattening': 298.257223563}}, 'coordinate_system': {'subtype': 'ellipsoidal', 'axis': [{'name': 'Geodetic latitude', 'abbreviation': 'Lat', 'direction': 'north', 'unit': 'degree'}, {'name': 'Geodetic longitude', 'abbreviation': 'Lon', 'direction': 'east', 'unit': 'degree'}]}, 'id': {'authority': 'EPSG', 'code': 4326}}, 'conversion': {'name': 'UTM zone 17N', 'method': {'name': 'Transverse Mercator', 'id': {'authority': 'EPSG', 'code': 9807}}, 'parameters': [{'name': 'Latitude of natural origin', 'value': 0, 'unit': 'degree', 'id': {'authority': 'EPSG', 'code': 8801}}, {'name': 'Longitude of natural origin', 'value': -81, 'unit': 'degree', 'id': {'authority': 'EPSG', 'code': 8802}}, {'name': 'Scale factor at natural origin', 'value': 0.9996, 'unit': 'unity', 'id': {'authority': 'EPSG', 'code': 8805}}, {'name': 'False easting', 'value': 500000, 'unit': 'metre', 'id': {'authority': 'EPSG', 'code': 8806}}, {'name': 'False northing', 'value': 0, 'unit': 'metre', 'id': {'authority': 'EPSG', 'code': 8807}}]}, 'coordinate_system': {'subtype': 'Cartesian', 'axis': [{'name': 'Easting', 'abbreviation': '', 'direction': 'east', 'unit': 'metre'}, {'name': 'Northing', 'abbreviation': '', 'direction': 'north', 'unit': 'metre'}]}, 'id': {'authority': 'EPSG', 'code': 32617}}, 'prettycompoundwkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",\\n GEOGCS[\"WGS 84\",\\n DATUM[\"WGS_1984\",\\n SPHEROID[\"WGS 84\",6378137,298.257223563,\\n AUTHORITY[\"EPSG\",\"7030\"]],\\n AUTHORITY[\"EPSG\",\"6326\"]],\\n PRIMEM[\"Greenwich\",0,\\n AUTHORITY[\"EPSG\",\"8901\"]],\\n UNIT[\"degree\",0.0174532925199433,\\n AUTHORITY[\"EPSG\",\"9122\"]],\\n AUTHORITY[\"EPSG\",\"4326\"]],\\n PROJECTION[\"Transverse_Mercator\"],\\n PARAMETER[\"latitude_of_origin\",0],\\n PARAMETER[\"central_meridian\",-81],\\n PARAMETER[\"scale_factor\",0.9996],\\n PARAMETER[\"false_easting\",500000],\\n PARAMETER[\"false_northing\",0],\\n UNIT[\"metre\",1,\\n AUTHORITY[\"EPSG\",\"9001\"]],\\n AXIS[\"Easting\",EAST],\\n AXIS[\"Northing\",NORTH],\\n AUTHORITY[\"EPSG\",\"32617\"]]', 'prettywkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",\\n GEOGCS[\"WGS 84\",\\n DATUM[\"WGS_1984\",\\n SPHEROID[\"WGS 84\",6378137,298.257223563,\\n AUTHORITY[\"EPSG\",\"7030\"]],\\n AUTHORITY[\"EPSG\",\"6326\"]],\\n PRIMEM[\"Greenwich\",0,\\n AUTHORITY[\"EPSG\",\"8901\"]],\\n UNIT[\"degree\",0.0174532925199433,\\n AUTHORITY[\"EPSG\",\"9122\"]],\\n AUTHORITY[\"EPSG\",\"4326\"]],\\n PROJECTION[\"Transverse_Mercator\"],\\n PARAMETER[\"latitude_of_origin\",0],\\n PARAMETER[\"central_meridian\",-81],\\n PARAMETER[\"scale_factor\",0.9996],\\n PARAMETER[\"false_easting\",500000],\\n PARAMETER[\"false_northing\",0],\\n UNIT[\"metre\",1,\\n AUTHORITY[\"EPSG\",\"9001\"]],\\n AXIS[\"Easting\",EAST],\\n AXIS[\"Northing\",NORTH],\\n AUTHORITY[\"EPSG\",\"32617\"]]', 'proj4': '+proj=utm +zone=17 +datum=WGS84 +units=m +no_defs', 'units': {'horizontal': 'metre', 'vertical': ''}, 'vertical': '', 'wkt': 'PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]'}, 'system_id': 'NASA G-LiHT v2.0, Riegl VQ-480', 'vlr_0': {'data': 'AQABAAAABgCChLCHAQAAAAAEAAABAAEAAQQAAAEAAQAECAAAAQApIwYIAAABAI4jAAwAAAEAaX8=', 'description': 'GeoKeyDirectoryTag', 'record_id': 34735, 'user_id': 'LASF_Projection'}, 'vlr_1': {'data': 'AAAAAAAAAAA=', 'description': 'GeoKeyDoubleParamsTag', 'record_id': 34736, 'user_id': 'LASF_Projection'}}}}\n" - ] - } - ], - "source": [ - "valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n", - "valid_pipe.execute()\n", - "print(valid_pipe.metadata)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Similarly, we can get `metadata`, `log` and `array values` from the executed pipeline using `valid_pipe.metadata`, `valid_pipe.log` and `valid_pipe.arrays`. The readers are encouraged to explore the results of these operations on their own. \n", - "\n", - "Additionally, we can run `pdal info ` on the command line for getting information from the generated COPC file." - ] - }, - { - "cell_type": "code", - "execution_count": 89, - "metadata": { - "vscode": { - "languageId": "shellscript" - } - }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", - " \"file_size\": 27367236,\n", - " \"filename\": \"data/GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\",\n", - " \"now\": \"2024-02-29T15:43:42-0600\",\n", + " \"file_size\": 95464691,\n", + " \"filename\": \"data/GLLIDARPC_FL_20200311_FIA8_l0s22.las\",\n", + " \"now\": \"2024-03-20T12:30:57-0500\",\n", " \"pdal_version\": \"2.6.3 (git-version: Release)\",\n", - " \"reader\": \"readers.copc\",\n", + " \"reader\": \"readers.las\",\n", " \"stats\":\n", " {\n", " \"bbox\":\n", @@ -413,50 +284,50 @@ " {\n", " \"bbox\":\n", " {\n", - " \"maxx\": -80.93555791,\n", - " \"maxy\": 25.28524099,\n", + " \"maxx\": -80.93555795,\n", + " \"maxy\": 25.28524102,\n", " \"maxz\": 69.99,\n", - " \"minx\": -80.94099071,\n", - " \"miny\": 25.27619903,\n", + " \"minx\": -80.94099075,\n", + " \"miny\": 25.27619906,\n", " \"minz\": -12.54\n", " },\n", - " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ -80.94099071383971, 25.276201300248541, -12.54 ], [ -80.940986340773605, 25.285240986017602, -12.54 ], [ -80.935557912747441, 25.285238714923079, 69.99 ], [ -80.93556268807356, 25.276199030078029, 69.99 ], [ -80.94099071383971, 25.276201300248541, -12.54 ] ] ] }\n", + " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ -80.940990750549048, 25.276201329530473, -12.54 ], [ -80.940986377485672, 25.285241015299494, -12.54 ], [ -80.9355579494582, 25.285238744206318, 69.99 ], [ -80.935562724781605, 25.276199059361314, 69.99 ], [ -80.940990750549048, 25.276201329530473, -12.54 ] ] ] }\n", " },\n", " \"native\":\n", " {\n", " \"bbox\":\n", " {\n", - " \"maxx\": 506487.74,\n", - " \"maxy\": 2796533.99,\n", + " \"maxx\": 506487.7363,\n", + " \"maxy\": 2796533.993,\n", " \"maxz\": 69.99,\n", - " \"minx\": 505941.23,\n", - " \"miny\": 2795533,\n", + " \"minx\": 505941.2263,\n", + " \"miny\": 2795533.003,\n", " \"minz\": -12.54\n", " },\n", - " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ 505941.230000000039581, 2795533.0, -12.54 ], [ 505941.230000000039581, 2796533.990000000223517, -12.54 ], [ 506487.74, 2796533.990000000223517, 69.99 ], [ 506487.74, 2795533.0, 69.99 ], [ 505941.230000000039581, 2795533.0, -12.54 ] ] ] }\n", + " \"boundary\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ 505941.226302567636594, 2795533.003240843303502, -12.54 ], [ 505941.226302567636594, 2796533.993240843061358, -12.54 ], [ 506487.736302567645907, 2796533.993240843061358, 69.99 ], [ 506487.736302567645907, 2795533.003240843303502, 69.99 ], [ 505941.226302567636594, 2795533.003240843303502, -12.54 ] ] ] }\n", " }\n", " },\n", " \"statistic\":\n", " [\n", " {\n", - " \"average\": 506237.8635,\n", + " \"average\": 506237.8598,\n", " \"count\": 3409439,\n", - " \"maximum\": 506487.74,\n", - " \"minimum\": 505941.23,\n", + " \"maximum\": 506487.7363,\n", + " \"minimum\": 505941.2263,\n", " \"name\": \"X\",\n", " \"position\": 0,\n", " \"stddev\": 101.3857552,\n", " \"variance\": 10279.07135\n", " },\n", " {\n", - " \"average\": 2795977.634,\n", + " \"average\": 2795977.637,\n", " \"count\": 3409439,\n", - " \"maximum\": 2796533.99,\n", - " \"minimum\": 2795533,\n", + " \"maximum\": 2796533.993,\n", + " \"minimum\": 2795533.003,\n", " \"name\": \"Y\",\n", " \"position\": 1,\n", - " \"stddev\": 274.3138881,\n", - " \"variance\": 75248.10919\n", + " \"stddev\": 274.313888,\n", + " \"variance\": 75248.10912\n", " },\n", " {\n", " \"average\": 2.192797205,\n", @@ -529,14 +400,14 @@ " \"variance\": 0.1821901239\n", " },\n", " {\n", - " \"average\": 2.569735542,\n", + " \"average\": 2.569738893,\n", " \"count\": 3409439,\n", " \"maximum\": 33,\n", - " \"minimum\": -31.00200081,\n", + " \"minimum\": -31,\n", " \"name\": \"ScanAngleRank\",\n", " \"position\": 9,\n", - " \"stddev\": 16.33805538,\n", - " \"variance\": 266.9320534\n", + " \"stddev\": 16.33805559,\n", + " \"variance\": 266.9320603\n", " },\n", " {\n", " \"average\": 1.475124207,\n", @@ -573,18 +444,8 @@ " \"count\": 3409439,\n", " \"maximum\": 0,\n", " \"minimum\": 0,\n", - " \"name\": \"ScanChannel\",\n", - " \"position\": 13,\n", - " \"stddev\": 0,\n", - " \"variance\": 0\n", - " },\n", - " {\n", - " \"average\": 0,\n", - " \"count\": 3409439,\n", - " \"maximum\": 0,\n", - " \"minimum\": 0,\n", " \"name\": \"Synthetic\",\n", - " \"position\": 14,\n", + " \"position\": 13,\n", " \"stddev\": 0,\n", " \"variance\": 0\n", " },\n", @@ -594,7 +455,7 @@ " \"maximum\": 0,\n", " \"minimum\": 0,\n", " \"name\": \"KeyPoint\",\n", - " \"position\": 15,\n", + " \"position\": 14,\n", " \"stddev\": 0,\n", " \"variance\": 0\n", " },\n", @@ -604,7 +465,7 @@ " \"maximum\": 0,\n", " \"minimum\": 0,\n", " \"name\": \"Withheld\",\n", - " \"position\": 16,\n", + " \"position\": 15,\n", " \"stddev\": 0,\n", " \"variance\": 0\n", " },\n", @@ -614,7 +475,7 @@ " \"maximum\": 0,\n", " \"minimum\": 0,\n", " \"name\": \"Overlap\",\n", - " \"position\": 17,\n", + " \"position\": 16,\n", " \"stddev\": 0,\n", " \"variance\": 0\n", " }\n", @@ -625,14 +486,200 @@ } ], "source": [ - "!pdal info {copc_filename}" + "!pdal info {las_filename}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### PDAL Pipelines\n", + "\n", + "For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/2.6.0/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/2.6.0/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/2.6.0/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/2.6.0/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/2.6.0/pipeline.html#stage-object).\n", + "\n", + "A pdal pipeline is defined in a JSON format either as a JSON object or a JSON array. Below is an example of a pdal pipeline taking a `.las` file as input, generating `stats` and writing it to a COPC format. \n", + "\n", + "```json\n", + "{\n", + " \"pipeline\": [\n", + " {\n", + " \"filename\":las_filename,\n", + " \"type\":\"readers.las\"\n", + " },\n", + " {\n", + " \"type\":\"filters.stats\",\n", + " },\n", + " {\n", + " \"type\":\"writers.copc\",\n", + " \"filename\":copc_filename\n", + " }\n", + "]\n", + "}\n", + "```\n", + "\n", + "This pipeline can be executed using the `pdal pipeline ` from the command line for a pipeline saved as a local `JSON` file. \n", + "\n", + "#### Programmatic Pipeline Construction\n", + "\n", + "However, here we will explore a comparatively easier and Pythonic approach to define a pipeline and execute it. This is based on the [PDAL Python extension](https://pypi.org/project/pdal/) which provides a programmatic pipeline construction approach in addition to the simple pipeline construction approach discussed above. \n", + "\n", + "This approach utilizes the `|` operator to pipe various stages together representing a pipeline. For eg., the above pipeline can be represented as -\n", + "\n", + "```python\n", + "pipeline = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename) | pdal.Filter.stats()\n", + "```\n", + "This pipeline can be executed using `pipeline.execute`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## LAS to COPC Conversion\n", + "\n", + "Now, let's dive into converting the LAS file to a COPC format based on the programmatic pipeline construction. " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "# Defining output filename. Usually, COPC files are saved as .copc.laz\n", + "copc_filename = las_filename.split('.')[0]+'.copc.laz'" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "3409439" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# pipe = stage 1 | stage 2 | stage 3\n", + "# Or, pipeline = pipeline 1 | stage 2\n", + "\n", + "# Once the pipeline is executed successfully, it prints the count of number of points\n", + "pipe = pdal.Reader.las(filename=las_filename) | pdal.Writer.copc(filename=copc_filename)\n", + "pipe.execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Validation\n", + "\n", + "As we can see from output of the below cell, the `.copc.laz` file is created in the destination directory." + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 253960\n", + "-rw-r--r-- 1 26M Feb 29 14:05 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n", + "-rw-r--r-- 1 91M Feb 29 11:27 GLLIDARPC_FL_20200311_FIA8_l0s22.las\n" + ] + } + ], + "source": [ + "# using -go for removing user details and h for getting memory size in MBs\n", + "!ls -goh {os.path.dirname(copc_filename)}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's read the created COPC file again and explore it's stats." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "metadata\n", + "dict_keys(['filters.stats', 'readers.las'])\n" + ] + } + ], + "source": [ + "valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n", + "valid_pipe.execute()\n", + "\n", + "# Get top key from the nested metadata dictionary\n", + "metadata = valid_pipe.metadata\n", + "top_key = list(metadata.keys())[0]\n", + "print(top_key)\n", + "\n", + "# Get top 2 keys from the next level metadata\n", + "print(metadata[top_key].keys())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Accessing Data\n", + "\n", + "The `array values` can be accessed from the executed pipeline using `valid_pipe.arrays`. " + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[(506097.87630257, 2795614.96324084, 3.26, 27371, 2, 2, 0, 0, 1, 0, 0, 0, 0, -28., 2, 279, 310470.74349771)\n", + " (506098.70630257, 2795614.83324084, 5.61, 29075, 1, 2, 0, 0, 1, 0, 0, 0, 0, -29., 2, 506, 310470.74350105)\n", + " (506095.80630257, 2795615.06324084, 0.3 , 26934, 2, 2, 0, 0, 2, 0, 0, 0, 0, -29., 2, 0, 310470.74350105)\n", + " ...\n", + " (506311.74630257, 2796533.98324084, 1.68, 27306, 1, 2, 1, 0, 1, 0, 0, 0, 0, 28., 1, 117, 310484.58526748)\n", + " (506311.99630257, 2796533.98324084, 2.1 , 32701, 1, 1, 1, 0, 1, 0, 0, 0, 0, 28., 1, 164, 310484.58527103)\n", + " (506296.84630257, 2796533.72324084, 0.34, 36306, 1, 1, 1, 0, 2, 0, 0, 0, 0, 26., 1, 0, 310484.59846769)]\n" + ] + } + ], + "source": [ + "arr_values = valid_pipe.arrays\n", + "\n", + "# Print the array values as a dataframe\n", + "print(arr_values[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "From the above metadata, it is important to note the line `\"reader\": \"readers.copc\"` which validates that the generated file is a valida COPC file. " + "Similarly, we can get `metadata` and `log` from the executed pipeline using `valid_pipe.metadata` and `valid_pipe.log`. The readers are encouraged to explore the results of these operations on their own. " ] } ], From 422a500376626c3ad500715bee92cc2f099d78ae Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Wed, 27 Mar 2024 14:04:53 -0500 Subject: [PATCH 3/8] Addressing review comments --- copc/lidar-las-to-copc.ipynb | 200 +++++++++++++++++++++++++++++------ 1 file changed, 165 insertions(+), 35 deletions(-) diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb index 23ca61b..3bf77c5 100644 --- a/copc/lidar-las-to-copc.ipynb +++ b/copc/lidar-las-to-copc.ipynb @@ -48,7 +48,7 @@ "\n", "## About the Dataset\n", "\n", - "We will be using the [G-LiHT Lidar Point Cloud V001](http://doi.org/10.5067/Community/GLIHT/GLLIDARPC.001) from the NASA EarthData. To access NASA EarthData into Jupyter Notebook, you can create an account by visiting [NASA's Earthdata Login page](https://urs.earthdata.nasa.gov/users/new). This will enable you to register for an account and retrieve the datasets used in the notebook.\n", + "We will be using the [G-LiHT Lidar Point Cloud V001](http://doi.org/10.5067/Community/GLIHT/GLLIDARPC.001) from the NASA EarthData. To access NASA EarthData in Jupyter you need to register for an [Earthdata account](https://urs.earthdata.nasa.gov/users/new).\n", "\n", "We will use [earthaccess](https://github.com/nsidc/earthaccess) library to set up credentials to fetch data from NASA's EarthData catalog." ] @@ -140,12 +140,10 @@ ], "source": [ "# Search Granules\n", - "short_name = 'GLLIDARPC'\n", - "version = '001'\n", "\n", "las_item_results = earthaccess.search_data(\n", - " short_name=short_name,\n", - " version=version,\n", + " short_name=\"GLLIDARPC\",\n", + " version=\"001\",\n", " temporal = (\"2020\"), \n", " count=3\n", ")" @@ -194,7 +192,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 39, "metadata": {}, "outputs": [ { @@ -208,7 +206,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 2236.96it/s]\n" + "QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 1869.12it/s]\n" ] }, { @@ -222,8 +220,8 @@ "name": "stderr", "output_type": "stream", "text": [ - "PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 3968.12it/s]\n", - "COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 12787.51it/s]" + "PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 16131.94it/s]\n", + "COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 33554.43it/s]" ] }, { @@ -244,7 +242,7 @@ "source": [ "# Download Data - Selecting the 3rd file from the `las_item_results` list\n", "gliht_las_file = earthaccess.download(las_item_results[2], data_dir)\n", - "las_filename = f\"{gliht_las_file[0]}\"\n", + "las_filename = gliht_las_file[0]\n", "print(las_filename)" ] }, @@ -254,11 +252,11 @@ "source": [ "## A Brief Introduction to PDAL \n", "\n", - "[PDAL](https://pdal.io/) (Point Data Abstraction Library) is a C/C++ based open-source library for processing point cloud data. It allows chained operations for processing point clouds. Additionally, it also has a PDAL-Python wrapper to work in a Pythonic environment. \n", + "[PDAL](https://pdal.io/) (Point Data Abstraction Library) is a C/C++ based open-source library for processing point cloud data. Additionally, it also has a PDAL-Python wrapper to work in a Pythonic environment. \n", "\n", "#### Accessing and Getting Metadata Information\n", "\n", - "PDAL CLI provides multiple [applications](https://pdal.io/en/2.7.0/apps/index.html) for processing point clouds. Similar to `gdal info` for TIFFs, we can run `pdal info ` on the command line for getting metadata from a point cloud file without reading it in memory. " + "PDAL CLI provides multiple [applications](https://pdal.io/en/2.7.0/apps/index.html) for processing point clouds. Also, it allows chaining of these applications for processing point clouds. Similar to `gdal info` for TIFFs, we can run `pdal info ` on the command line for getting metadata from a point cloud file without reading it in memory. " ] }, { @@ -497,7 +495,7 @@ "\n", "#### PDAL Pipelines\n", "\n", - "For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/2.6.0/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/2.6.0/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/2.6.0/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/2.6.0/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/2.6.0/pipeline.html#stage-object).\n", + "For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/latest/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/latest/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/latest/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/latest/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/latest/pipeline.html#stage-object).\n", "\n", "A pdal pipeline is defined in a JSON format either as a JSON object or a JSON array. Below is an example of a pdal pipeline taking a `.las` file as input, generating `stats` and writing it to a COPC format. \n", "\n", @@ -544,12 +542,12 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "# Defining output filename. Usually, COPC files are saved as .copc.laz\n", - "copc_filename = las_filename.split('.')[0]+'.copc.laz'" + "copc_filename = las_filename.replace('.laz', '.copc.laz')" ] }, { @@ -588,56 +586,188 @@ }, { "cell_type": "code", - "execution_count": 73, + "execution_count": 48, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "total 253960\n", - "-rw-r--r-- 1 26M Feb 29 14:05 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n", + "total 239888\n", + "-rw-r--r-- 1 26M Mar 20 11:55 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n", "-rw-r--r-- 1 91M Feb 29 11:27 GLLIDARPC_FL_20200311_FIA8_l0s22.las\n" ] } ], "source": [ "# using -go for removing user details and h for getting memory size in MBs\n", - "!ls -goh {os.path.dirname(copc_filename)}" + "!ls -goh ./data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Let's read the created COPC file again and explore it's stats." + "Let's read the created COPC file again and explore the point cloud statistics." ] }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 51, "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "metadata\n", - "dict_keys(['filters.stats', 'readers.las'])\n" - ] + "data": { + "text/plain": [ + "[{'average': 506237.8598,\n", + " 'count': 3409439,\n", + " 'maximum': 506487.7363,\n", + " 'minimum': 505941.2263,\n", + " 'name': 'X',\n", + " 'position': 0,\n", + " 'stddev': 101.3857552,\n", + " 'variance': 10279.07135},\n", + " {'average': 2795977.637,\n", + " 'count': 3409439,\n", + " 'maximum': 2796533.993,\n", + " 'minimum': 2795533.003,\n", + " 'name': 'Y',\n", + " 'position': 1,\n", + " 'stddev': 274.313888,\n", + " 'variance': 75248.10912},\n", + " {'average': 2.192797205,\n", + " 'count': 3409439,\n", + " 'maximum': 69.99,\n", + " 'minimum': -12.54,\n", + " 'name': 'Z',\n", + " 'position': 2,\n", + " 'stddev': 1.788122887,\n", + " 'variance': 3.197383461},\n", + " {'average': 30205.96606,\n", + " 'count': 3409439,\n", + " 'maximum': 65535,\n", + " 'minimum': 14789,\n", + " 'name': 'Intensity',\n", + " 'position': 3,\n", + " 'stddev': 5497.346879,\n", + " 'variance': 30220822.71},\n", + " {'average': 1.200336478,\n", + " 'count': 3409439,\n", + " 'maximum': 5,\n", + " 'minimum': 1,\n", + " 'name': 'ReturnNumber',\n", + " 'position': 4,\n", + " 'stddev': 0.4267302243,\n", + " 'variance': 0.1820986843},\n", + " {'average': 1.400683808,\n", + " 'count': 3409439,\n", + " 'maximum': 5,\n", + " 'minimum': 1,\n", + " 'name': 'NumberOfReturns',\n", + " 'position': 5,\n", + " 'stddev': 0.5529822902,\n", + " 'variance': 0.3057894133},\n", + " {'average': 0.5248757933,\n", + " 'count': 3409439,\n", + " 'maximum': 1,\n", + " 'minimum': 0,\n", + " 'name': 'ScanDirectionFlag',\n", + " 'position': 6,\n", + " 'stddev': 0.4993808847,\n", + " 'variance': 0.249381268},\n", + " {'average': 0.002420339534,\n", + " 'count': 3409439,\n", + " 'maximum': 1,\n", + " 'minimum': 0,\n", + " 'name': 'EdgeOfFlightLine',\n", + " 'position': 7,\n", + " 'stddev': 0.04913738087,\n", + " 'variance': 0.002414482199},\n", + " {'average': 1.239596602,\n", + " 'count': 3409439,\n", + " 'maximum': 2,\n", + " 'minimum': 1,\n", + " 'name': 'Classification',\n", + " 'position': 8,\n", + " 'stddev': 0.4268373506,\n", + " 'variance': 0.1821901239},\n", + " {'average': 2.569738893,\n", + " 'count': 3409439,\n", + " 'maximum': 33,\n", + " 'minimum': -31,\n", + " 'name': 'ScanAngleRank',\n", + " 'position': 9,\n", + " 'stddev': 16.33805559,\n", + " 'variance': 266.9320603},\n", + " {'average': 1.475124207,\n", + " 'count': 3409439,\n", + " 'maximum': 2,\n", + " 'minimum': 1,\n", + " 'name': 'UserData',\n", + " 'position': 10,\n", + " 'stddev': 0.4993808847,\n", + " 'variance': 0.249381268},\n", + " {'average': 192.5227238,\n", + " 'count': 3409439,\n", + " 'maximum': 65535,\n", + " 'minimum': 0,\n", + " 'name': 'PointSourceId',\n", + " 'position': 11,\n", + " 'stddev': 680.5012031,\n", + " 'variance': 463081.8874},\n", + " {'average': 310476.1839,\n", + " 'count': 3409439,\n", + " 'maximum': 310485.0477,\n", + " 'minimum': 310469.0825,\n", + " 'name': 'GpsTime',\n", + " 'position': 12,\n", + " 'stddev': 4.240892261,\n", + " 'variance': 17.98516717},\n", + " {'average': 0,\n", + " 'count': 3409439,\n", + " 'maximum': 0,\n", + " 'minimum': 0,\n", + " 'name': 'Synthetic',\n", + " 'position': 13,\n", + " 'stddev': 0,\n", + " 'variance': 0},\n", + " {'average': 0,\n", + " 'count': 3409439,\n", + " 'maximum': 0,\n", + " 'minimum': 0,\n", + " 'name': 'KeyPoint',\n", + " 'position': 14,\n", + " 'stddev': 0,\n", + " 'variance': 0},\n", + " {'average': 0,\n", + " 'count': 3409439,\n", + " 'maximum': 0,\n", + " 'minimum': 0,\n", + " 'name': 'Withheld',\n", + " 'position': 15,\n", + " 'stddev': 0,\n", + " 'variance': 0},\n", + " {'average': 0,\n", + " 'count': 3409439,\n", + " 'maximum': 0,\n", + " 'minimum': 0,\n", + " 'name': 'Overlap',\n", + " 'position': 16,\n", + " 'stddev': 0,\n", + " 'variance': 0}]" + ] + }, + "execution_count": 51, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ "valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n", "valid_pipe.execute()\n", "\n", - "# Get top key from the nested metadata dictionary\n", - "metadata = valid_pipe.metadata\n", - "top_key = list(metadata.keys())[0]\n", - "print(top_key)\n", - "\n", - "# Get top 2 keys from the next level metadata\n", - "print(metadata[top_key].keys())" + "# Getting statistic from the metadata\n", + "valid_pipe.metadata[\"metadata\"][\"filters.stats\"][\"statistic\"]" ] }, { @@ -646,7 +776,7 @@ "source": [ "## Accessing Data\n", "\n", - "The `array values` can be accessed from the executed pipeline using `valid_pipe.arrays`. " + "The data values can be accessed from the executed pipeline using `valid_pipe.arrays`. " ] }, { From 4398f994ed80adc47e38c4d460ad4aee5c0f0c10 Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Fri, 29 Mar 2024 09:15:42 -0500 Subject: [PATCH 4/8] update navigation link for COPC notebook --- _quarto.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/_quarto.yml b/_quarto.yml index 2026659..6ee6b79 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -53,6 +53,7 @@ website: - section: Cloud-Optimized Point Clouds (COPC) contents: - copc/index.qmd + - copc/lidar-las-to-copc.ipynb - section: GeoParquet contents: - geoparquet/index.qmd From 6a11a9aab6aa0ad9212bd0763990842e7b456185 Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Fri, 29 Mar 2024 09:19:47 -0500 Subject: [PATCH 5/8] Minor updates for consistency in COPC full form --- copc/lidar-las-to-copc.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb index 3bf77c5..f68e63d 100644 --- a/copc/lidar-las-to-copc.ipynb +++ b/copc/lidar-las-to-copc.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Converting LiDAR LAS Files to Cloud Optimized Point Clouds (COPCs)" + "# Converting LiDAR LAS Files to Cloud-Optimized Point Clouds (COPCs)" ] }, { @@ -43,7 +43,7 @@ "This tutorial will explore how to-\n", "\n", "1. Read a LiDAR LAS file using PDAL in Python\n", - "2. Convert the LiDAR LAS file to Cloud Optimized Point Cloud (COPC) format\n", + "2. Convert the LiDAR LAS file to Cloud-Optimized Point Cloud (COPC) format\n", "2. Validate the generated COPC file\n", "\n", "## About the Dataset\n", From 4f58ef050770e95be81b8dada65a59e20daedff2 Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Fri, 29 Mar 2024 10:16:07 -0500 Subject: [PATCH 6/8] Update environment file --- copc/environment.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/copc/environment.yml b/copc/environment.yml index a1eced7..b66210d 100644 --- a/copc/environment.yml +++ b/copc/environment.yml @@ -8,5 +8,5 @@ dependencies: - jupyterlab - matplotlib - libgdal>=3.5 - - python-pdal - - pdal \ No newline at end of file + - python-pdal=3.3.0 + - pdal=2.6.3 \ No newline at end of file From d71854f2304515263d9c06b0e9ab8d21769ac641 Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Fri, 29 Mar 2024 10:29:38 -0500 Subject: [PATCH 7/8] update pdal reader for copc --- copc/lidar-las-to-copc.ipynb | 84 ++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 32 deletions(-) diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb index f68e63d..2aa26ff 100644 --- a/copc/lidar-las-to-copc.ipynb +++ b/copc/lidar-las-to-copc.ipynb @@ -542,17 +542,29 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 64, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "'data/GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz'" + ] + }, + "execution_count": 64, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Defining output filename. Usually, COPC files are saved as .copc.laz\n", - "copc_filename = las_filename.replace('.laz', '.copc.laz')" + "copc_filename = las_filename.replace('.las', '.copc.laz')\n", + "copc_filename" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 65, "metadata": {}, "outputs": [ { @@ -561,7 +573,7 @@ "3409439" ] }, - "execution_count": 12, + "execution_count": 65, "metadata": {}, "output_type": "execute_result" } @@ -613,28 +625,28 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 70, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[{'average': 506237.8598,\n", + "[{'average': 506237.8635,\n", " 'count': 3409439,\n", - " 'maximum': 506487.7363,\n", - " 'minimum': 505941.2263,\n", + " 'maximum': 506487.74,\n", + " 'minimum': 505941.23,\n", " 'name': 'X',\n", " 'position': 0,\n", " 'stddev': 101.3857552,\n", " 'variance': 10279.07135},\n", - " {'average': 2795977.637,\n", + " {'average': 2795977.634,\n", " 'count': 3409439,\n", - " 'maximum': 2796533.993,\n", - " 'minimum': 2795533.003,\n", + " 'maximum': 2796533.99,\n", + " 'minimum': 2795533,\n", " 'name': 'Y',\n", " 'position': 1,\n", " 'stddev': 274.313888,\n", - " 'variance': 75248.10912},\n", + " 'variance': 75248.10915},\n", " {'average': 2.192797205,\n", " 'count': 3409439,\n", " 'maximum': 69.99,\n", @@ -691,14 +703,14 @@ " 'position': 8,\n", " 'stddev': 0.4268373506,\n", " 'variance': 0.1821901239},\n", - " {'average': 2.569738893,\n", + " {'average': 2.569735542,\n", " 'count': 3409439,\n", " 'maximum': 33,\n", - " 'minimum': -31,\n", + " 'minimum': -31.00200081,\n", " 'name': 'ScanAngleRank',\n", " 'position': 9,\n", - " 'stddev': 16.33805559,\n", - " 'variance': 266.9320603},\n", + " 'stddev': 16.33805538,\n", + " 'variance': 266.9320534},\n", " {'average': 1.475124207,\n", " 'count': 3409439,\n", " 'maximum': 2,\n", @@ -721,13 +733,13 @@ " 'minimum': 310469.0825,\n", " 'name': 'GpsTime',\n", " 'position': 12,\n", - " 'stddev': 4.240892261,\n", - " 'variance': 17.98516717},\n", + " 'stddev': 4.240892259,\n", + " 'variance': 17.98516715},\n", " {'average': 0,\n", " 'count': 3409439,\n", " 'maximum': 0,\n", " 'minimum': 0,\n", - " 'name': 'Synthetic',\n", + " 'name': 'ScanChannel',\n", " 'position': 13,\n", " 'stddev': 0,\n", " 'variance': 0},\n", @@ -735,7 +747,7 @@ " 'count': 3409439,\n", " 'maximum': 0,\n", " 'minimum': 0,\n", - " 'name': 'KeyPoint',\n", + " 'name': 'Synthetic',\n", " 'position': 14,\n", " 'stddev': 0,\n", " 'variance': 0},\n", @@ -743,7 +755,7 @@ " 'count': 3409439,\n", " 'maximum': 0,\n", " 'minimum': 0,\n", - " 'name': 'Withheld',\n", + " 'name': 'KeyPoint',\n", " 'position': 15,\n", " 'stddev': 0,\n", " 'variance': 0},\n", @@ -751,19 +763,27 @@ " 'count': 3409439,\n", " 'maximum': 0,\n", " 'minimum': 0,\n", - " 'name': 'Overlap',\n", + " 'name': 'Withheld',\n", " 'position': 16,\n", " 'stddev': 0,\n", + " 'variance': 0},\n", + " {'average': 0,\n", + " 'count': 3409439,\n", + " 'maximum': 0,\n", + " 'minimum': 0,\n", + " 'name': 'Overlap',\n", + " 'position': 17,\n", + " 'stddev': 0,\n", " 'variance': 0}]" ] }, - "execution_count": 51, + "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n", + "valid_pipe = pdal.Reader.copc(filename=copc_filename) | pdal.Filter.stats()\n", "valid_pipe.execute()\n", "\n", "# Getting statistic from the metadata\n", @@ -781,20 +801,20 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 69, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "[(506097.87630257, 2795614.96324084, 3.26, 27371, 2, 2, 0, 0, 1, 0, 0, 0, 0, -28., 2, 279, 310470.74349771)\n", - " (506098.70630257, 2795614.83324084, 5.61, 29075, 1, 2, 0, 0, 1, 0, 0, 0, 0, -29., 2, 506, 310470.74350105)\n", - " (506095.80630257, 2795615.06324084, 0.3 , 26934, 2, 2, 0, 0, 2, 0, 0, 0, 0, -29., 2, 0, 310470.74350105)\n", + "[(506245.56, 2796471.44, 0.24, 40740, 1, 1, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75227621, 0)\n", + " (506247.16, 2796471.58, 0.27, 35541, 2, 2, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75229014, 0)\n", + " (506247.95, 2796471.65, 0.24, 17716, 2, 2, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75229699, 0)\n", " ...\n", - " (506311.74630257, 2796533.98324084, 1.68, 27306, 1, 2, 1, 0, 1, 0, 0, 0, 0, 28., 1, 117, 310484.58526748)\n", - " (506311.99630257, 2796533.98324084, 2.1 , 32701, 1, 1, 1, 0, 1, 0, 0, 0, 0, 28., 1, 164, 310484.58527103)\n", - " (506296.84630257, 2796533.72324084, 0.34, 36306, 1, 1, 1, 0, 2, 0, 0, 0, 0, 26., 1, 0, 310484.59846769)]\n" + " (506066.58, 2796032.75, 2.34, 31587, 1, 1, 0, 0, 1, 0, 0, 0, 0, -24. , 2, 203, 310477.36925451, 0)\n", + " (506067.37, 2796033.29, 2.52, 32876, 1, 1, 0, 0, 1, 0, 0, 0, 0, -22.998, 2, 216, 310477.37590641, 0)\n", + " (506062.6 , 2796033.27, 1.4 , 27393, 1, 1, 0, 0, 1, 0, 0, 0, 0, -24. , 2, 108, 310477.38259945, 0)]\n" ] } ], From 20bcfc0011dc34e10b91b0ceb5b9376a5d837b57 Mon Sep 17 00:00:00 2001 From: Rajat Shinde Date: Mon, 1 Apr 2024 16:10:28 -0500 Subject: [PATCH 8/8] update copc:true check for validation --- copc/lidar-las-to-copc.ipynb | 123 ++++++++++++++++++++--------------- 1 file changed, 72 insertions(+), 51 deletions(-) diff --git a/copc/lidar-las-to-copc.ipynb b/copc/lidar-las-to-copc.ipynb index 2aa26ff..ab5d4b7 100644 --- a/copc/lidar-las-to-copc.ipynb +++ b/copc/lidar-las-to-copc.ipynb @@ -620,12 +620,78 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's read the created COPC file again and explore the point cloud statistics." + "Let's read the created COPC file again and check the value of `copc` flag from the [metadata](https://pdal.io/en/latest/development/metadata.html). If the generated LiDAR file is a valid COPC file, then this flag should be set to `True`." ] }, { "cell_type": "code", - "execution_count": 70, + "execution_count": 81, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True\n" + ] + } + ], + "source": [ + "valid_pipe = pdal.Reader.copc(filename=copc_filename) | pdal.Filter.stats()\n", + "valid_pipe.execute()\n", + "\n", + "# Getting value for the \"copc\" key under the metadata\n", + "# Output is True for a valid COPC\n", + "value = valid_pipe.metadata[\"metadata\"][\"readers.copc\"].get(\"copc\")\n", + "print(value)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Accessing Data\n", + "\n", + "The data values can be accessed from the executed pipeline using `valid_pipe.arrays`. The values in the arrays represent the LiDAR point cloud attributes such as `X`, `Y`, `Z`, and `Intensity`, etc." + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[array([(506245.56, 2796471.44, 0.24, 40740, 1, 1, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75227621, 0),\n", + " (506247.16, 2796471.58, 0.27, 35541, 2, 2, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75229014, 0),\n", + " (506247.95, 2796471.65, 0.24, 17716, 2, 2, 1, 0, 2, 0, 0, 0, 0, 16.998, 1, 0, 310483.75229699, 0),\n", + " ...,\n", + " (506066.58, 2796032.75, 2.34, 31587, 1, 1, 0, 0, 1, 0, 0, 0, 0, -24. , 2, 203, 310477.36925451, 0),\n", + " (506067.37, 2796033.29, 2.52, 32876, 1, 1, 0, 0, 1, 0, 0, 0, 0, -22.998, 2, 216, 310477.37590641, 0),\n", + " (506062.6 , 2796033.27, 1.4 , 27393, 1, 1, 0, 0, 1, 0, 0, 0, 0, -24. , 2, 108, 310477.38259945, 0)],\n", + " dtype=[('X', '