On a large, highly frequently updated dataset, the row count may change before geopetl finishes extracting, leading to assertion error

[jrmidkiff]

[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 0: Trying to connect to postgres...
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 1: Connected to postgres.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 2: Connected to table cco.asset_history
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 3: cco.asset_history current row count: 1,347,556
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 4: ********************************************************************************
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 5: cco.asset_history current row count: 1,347,556
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 6: Starting extract from cco.asset_history
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 7: Rows to extract: 1347556
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 8: Note: petl can cause log messages to seemingly come out of order.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 9: Initializing data var with etl.frompostgis()..
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 10: Database object type: table.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 11: sde register check:  []
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 12: Dataset appears to be non-geometric, returning geom_field as None.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 13: Dataset appears to be non-geometric, returning geom_field as None.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 14: Database object type: table.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 15: Asserting counts match between db and extracted csv
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 16: 1347556 == 1347556
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 17: Extracting csv...
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 18: 89837 rows in 3.65s (24591 row/s); batch in 3.65s (24591 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 19: 179674 rows in 4.75s (37786 row/s); batch in 1.10s (81539 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 20: 269511 rows in 5.86s (45986 row/s); batch in 1.11s (81244 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 21: 359348 rows in 6.96s (51599 row/s); batch in 1.10s (81411 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 22: 449185 rows in 8.07s (55680 row/s); batch in 1.10s (81451 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 23: 539022 rows in 9.18s (58746 row/s); batch in 1.11s (81058 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 24: 628859 rows in 10.28s (61162 row/s); batch in 1.11s (81200 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 25: 718696 rows in 11.38s (63130 row/s); batch in 1.10s (81489 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 26: 808533 rows in 12.49s (64733 row/s); batch in 1.11s (81237 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 27: 898370 rows in 13.60s (66076 row/s); batch in 1.11s (81238 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 28: 988207 rows in 14.70s (67205 row/s); batch in 1.11s (81053 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 29: 1078044 rows in 15.82s (68163 row/s); batch in 1.11s (80847 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 30: 1167881 rows in 16.93s (68997 row/s); batch in 1.11s (80861 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 31: 1257718 rows in 18.04s (69720 row/s); batch in 1.11s (80729 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 32: 1347555 rows in 19.15s (70362 row/s); batch in 1.11s (80768 row/s)
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 33: 1347562 rows in 19.17s (70285 row/s); batches in 1.28 +/- 0.64s [1.10-3.65] (77381 +/- 14111 rows/s [24591-81539])
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 34: Dataset appears to be non-geometric, returning geom_field as None.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 35: Dataset appears to be non-geometric, returning geom_field as None.
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 36: cco.asset_history current row count: 1,347,562
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 37: Asserting counts match between current db count and extracted csv
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 38: 1347562 == 1347556
[2024-10-29, 10:25:16 EDT] {get_batch_logs.py:41} INFO - 39: Workflow failed... rolling back database transactions.