NEOS-1708: optimize sql disconnections and busy signals during sync

[nickzelei]

• Adds low-level wrapper around database connections for handling retries reads and writes (except query row)
• Removes bespoke deadlock code from benthos input/updates as it is redundant to the above
• Adds more error codes to backoff strategy to handle busy signals and unexpected eof, along with some other temporary object errors for postgres.

[linear]

NEOS-1708 Optimize SQL Disconnections and Busy signals during sync

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
neosync-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 23, 2025 6:07pm

[codecov]

Codecov Report

Attention: Patch coverage is 59.64912% with 46 lines in your changes missing coverage. Please review.

Project coverage is 27.85%. Comparing base (982a282) to head (104ca96).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
backend/pkg/sqlretry/dbtx_retry.go	61.22%	35 Missing and 3 partials ⚠️
backend/pkg/sqlconnect/sql-connector.go	50.00%	5 Missing ⚠️
worker/pkg/benthos/sql/output_sql_insert.go	33.33%	2 Missing ⚠️
worker/pkg/benthos/sql/output_sql_update.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3182      +/-   ##
==========================================
+ Coverage   27.82%   27.85%   +0.02%     
==========================================
  Files         378      378              
  Lines       43259    43270      +11     
==========================================
+ Hits        12037    12051      +14     
+ Misses      29772    29771       -1     
+ Partials     1450     1448       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Benchstat Geomean Results

-0.46% sec/op, -0.03% B/op, 0.00% allocs/op

Benchstat results

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │       sec/op       │    sec/op     vs base                │
CleanPostgresType-4                        41.12n ± ∞ ¹   42.69n ±  4%       ~ (p=0.286 n=1+6)
Runner_Single-4                            7.942µ ± ∞ ¹   7.848µ ±  3%       ~ (p=0.571 n=1+6)
GenerateBool-4                             11.76n ± ∞ ¹   11.44n ± 54%       ~ (p=0.571 n=1+6)
GenerateBusinessName-4                     334.8n ± ∞ ¹   315.0n ±  3%       ~ (p=0.286 n=1+6)
GenerateCardNumber-4                       28.64n ± ∞ ¹   28.60n ±  0%       ~ (p=0.571 n=1+6)
GenerateCategorical-4                      102.3n ± ∞ ¹   104.4n ±  0%       ~ (p=0.286 n=1+6)
GenerateCity-4                             173.1n ± ∞ ¹   174.8n ±  1%       ~ (p=0.286 n=1+6)
GenerateCountry-4                          105.2n ± ∞ ¹   104.7n ±  1%       ~ (p=0.857 n=1+6)
GenerateEmail-4                            1.383µ ± ∞ ¹   1.345µ ±  1%       ~ (p=0.286 n=1+6)
GenerateFirstName-4                        236.7n ± ∞ ¹   206.4n ±  7%       ~ (p=0.286 n=1+6)
GenerateFloat64-4                          28.72n ± ∞ ¹   29.54n ±  1%       ~ (p=0.286 n=1+6)
GenerateFullAddress-4                      1.430µ ± ∞ ¹   1.451µ ±  1%       ~ (p=0.286 n=1+6)
GenerateFullName-4                         1.299µ ± ∞ ¹   1.263µ ±  6%       ~ (p=0.571 n=1+6)
GenerateGender-4                           41.98n ± ∞ ¹   41.75n ±  1%       ~ (p=0.571 n=1+6)
GenerateInt64-4                            28.00n ± ∞ ¹   28.32n ±  1%       ~ (p=0.286 n=1+6)
GenerateInt64PhoneNumber-4                 51.76n ± ∞ ¹   51.64n ±  1%       ~ (p=0.857 n=1+6)
GenerateInternationalPhoneNumber-4         176.1n ± ∞ ¹   177.5n ±  6%       ~ (p=0.286 n=1+6)
GenerateIpAddress-4                        568.1n ± ∞ ¹   548.2n ±  1%       ~ (p=0.286 n=1+6)
GenerateLastName-4                         196.7n ± ∞ ¹   191.4n ±  1%       ~ (p=0.286 n=1+6)
GenerateRandomString-4                    1007.0n ± ∞ ¹   996.6n ±  1%       ~ (p=0.286 n=1+6)
GenerateSHA256Hash-4                       1.007µ ± ∞ ¹   1.007µ ±  0%       ~ (p=0.857 n=1+6)
GenerateSSN-4                              301.8n ± ∞ ¹   304.4n ±  1%       ~ (p=0.286 n=1+6)
GenerateState-4                            104.9n ± ∞ ¹   105.3n ±  0%       ~ (p=0.286 n=1+6)
GenerateStreetAddress-4                    205.3n ± ∞ ¹   205.8n ±  1%       ~ (p=0.857 n=1+6)
GenerateStringPhoneNumber-4                173.9n ± ∞ ¹   180.7n ±  1%       ~ (p=0.286 n=1+6)
GenerateUnixTimestamp-4                    81.14n ± ∞ ¹   77.34n ±  4%  -4.68% (n=1+6)
GenerateUsername-4                         354.3n ± ∞ ¹   355.3n ±  0%       ~ (p=0.571 n=1+6)
GenerateUTCTimestamp-4                     107.1n ± ∞ ¹   104.2n ±  1%       ~ (p=0.286 n=1+6)
GenerateUUID-4                             738.8n ± ∞ ¹   742.8n ±  0%       ~ (p=0.286 n=1+6)
GenerateZipcode-4                          105.5n ± ∞ ¹   106.1n ±  0%       ~ (p=0.286 n=1+6)
TransformCharacterScramble-4               173.7n ± ∞ ¹   172.9n ±  1%       ~ (p=0.571 n=1+6)
TransformE164PhoneNumber-4                 180.9n ± ∞ ¹   181.8n ±  5%       ~ (p=0.286 n=1+6)
TransformEmail-4                           1.791µ ± ∞ ¹   1.829µ ±  1%       ~ (p=0.286 n=1+6)
TransformFirstName-4                       239.7n ± ∞ ¹   241.2n ±  0%       ~ (p=0.286 n=1+6)
TransformFloat64-4                         111.5n ± ∞ ¹   111.5n ±  0%       ~ (p=1.000 n=1+6)
TransformFullName-4                        1.304µ ± ∞ ¹   1.328µ ±  5%       ~ (p=0.857 n=1+6)
TransformInt64-4                           33.45n ± ∞ ¹   33.76n ±  0%       ~ (p=0.286 n=1+6)
TransformInt64PhoneNumber-4                58.74n ± ∞ ¹   58.17n ±  0%       ~ (p=0.286 n=1+6)
TransformLastName-4                        197.3n ± ∞ ¹   193.4n ±  1%       ~ (p=0.286 n=1+6)
TransformString-4                          1.026µ ± ∞ ¹   1.032µ ±  1%       ~ (p=0.571 n=1+6)
TransformStringPhoneNumber-4               205.1n ± ∞ ¹   211.4n ±  1%       ~ (p=0.286 n=1+6)
TransformUuid-4                            36.66n ± ∞ ¹   36.89n ±  0%       ~ (p=0.286 n=1+6)
geomean                                    197.0n         196.1n        -0.46%
¹ need >= 6 samples for confidence interval at level 0.95

                                   │ main-benchmark.txt │          branch-benchmark.txt          │
                                   │        B/op        │     B/op      vs base                  │
CleanPostgresType-4                         0.000 ± ∞ ¹     0.000 ± 0%       ~ (p=1.000 n=1+6) ²
Runner_Single-4                           2.318Ki ± ∞ ¹   2.292Ki ± 2%       ~ (p=0.571 n=1+6)
GenerateBool-4                              0.000 ± ∞ ¹     0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBusinessName-4                      304.0 ± ∞ ¹     304.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCardNumber-4                        8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCategorical-4                       64.00 ± ∞ ¹     64.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCity-4                              160.0 ± ∞ ¹     160.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCountry-4                           24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateEmail-4                             385.0 ± ∞ ¹     385.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFirstName-4                         208.0 ± ∞ ¹     208.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFloat64-4                           8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullAddress-4                       512.0 ± ∞ ¹     512.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullName-4                          418.0 ± ∞ ¹     418.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateGender-4                            16.00 ± ∞ ¹     16.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64-4                             7.000 ± ∞ ¹     7.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64PhoneNumber-4                  8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInternationalPhoneNumber-4          40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateIpAddress-4                         31.00 ± ∞ ¹     31.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateLastName-4                          176.0 ± ∞ ¹     176.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateRandomString-4                      186.0 ± ∞ ¹     186.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSHA256Hash-4                        288.0 ± ∞ ¹     288.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSSN-4                               46.00 ± ∞ ¹     46.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateState-4                             24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStreetAddress-4                     240.0 ± ∞ ¹     240.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStringPhoneNumber-4                 40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUnixTimestamp-4                     8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUsername-4                          205.0 ± ∞ ¹     205.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUTCTimestamp-4                      24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUUID-4                              80.00 ± ∞ ¹     80.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateZipcode-4                           24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformCharacterScramble-4                40.00 ± ∞ ¹     40.00 ± 3%       ~ (p=0.286 n=1+6)
TransformE164PhoneNumber-4                  40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformEmail-4                            489.0 ± ∞ ¹     489.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFirstName-4                        224.0 ± ∞ ¹     224.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFloat64-4                          88.00 ± ∞ ¹     88.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFullName-4                         418.0 ± ∞ ¹     418.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64-4                            8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64PhoneNumber-4                 8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformLastName-4                         176.0 ± ∞ ¹     176.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformString-4                           202.0 ± ∞ ¹     202.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformStringPhoneNumber-4                56.00 ± ∞ ¹     56.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformUuid-4                             16.00 ± ∞ ¹     16.00 ± 0%       ~ (p=1.000 n=1+6) ²
geomean                                               ³                 -0.03%                 ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │     allocs/op      │ allocs/op   vs base                  │
CleanPostgresType-4                         0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
Runner_Single-4                             24.00 ± ∞ ¹   24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBool-4                              0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBusinessName-4                      2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCardNumber-4                        1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCategorical-4                       2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCity-4                              2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCountry-4                           2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateEmail-4                             8.000 ± ∞ ¹   8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFirstName-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFloat64-4                           1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullAddress-4                       10.00 ± ∞ ¹   10.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullName-4                          6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateGender-4                            1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64-4                             0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64PhoneNumber-4                  1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInternationalPhoneNumber-4          3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateIpAddress-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateLastName-4                          2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateRandomString-4                      3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSHA256Hash-4                        7.000 ± ∞ ¹   7.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSSN-4                               3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateState-4                             2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStreetAddress-4                     2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStringPhoneNumber-4                 3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUnixTimestamp-4                     1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUsername-4                          6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUTCTimestamp-4                      1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUUID-4                              3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateZipcode-4                           2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformCharacterScramble-4                2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformE164PhoneNumber-4                  3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformEmail-4                            14.00 ± ∞ ¹   14.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFirstName-4                        3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFloat64-4                          3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFullName-4                         6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64-4                            1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64PhoneNumber-4                 1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformLastName-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformString-4                           4.000 ± ∞ ¹   4.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformStringPhoneNumber-4                4.000 ± ∞ ¹   4.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformUuid-4                             1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
geomean                                               ³               +0.00%                 ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean