Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow BLOB columns with utf8mb3 charset #144

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

juan-ferrer-toribio
Copy link

Fixes #143

@rhwinter
Copy link

I was running into this issue when using @rodrigogs/mysql-events. Since I don't have access to configure the MySQL server, which is likely the culprit of the issue (wrong charset encoding settings).

The following error would be thrown when something changed in certain databases/tables/columns:

Error: Encoding not recognized: 'utf8mb3' (searched as: 'utf8mb3')
      at Object.getCodec (_____/node_modules/iconv-lite/lib/index.js:106:23)
      at Object.getDecoder (_____/node_modules/iconv-lite/lib/index.js:127:23)
      at Object.decode (_____/node_modules/iconv-lite/lib/index.js:40:25)
      at exports.readMysqlValue (_____/node_modules/@rodrigogs/zongji/lib/common.js:461:24)
      at readRow (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:111:16)
      at RowsEvent._fetchOneRow (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:93:10)
      at DeleteRows.RowsEvent (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:62:27)
      at new DeleteRows (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:128:13)
      at BinlogHeader.parse (_____/node_modules/@rodrigogs/zongji/lib/packet/binlog_header.js:47:23)
      at Protocol._parsePacket (_____/node_modules/mysql/lib/protocol/Protocol.js:272:12)

which would be followed by a mysql-events connection error:

Error: Packets out of order. Got: 161 Expected: 42
      at Parser._tryReadPacketHeader (_____/node_modules/mysql/lib/protocol/Parser.js:470:15)
      at Parser.write (_____/node_modules/mysql/lib/protocol/Parser.js:33:29)
      at Protocol.write (_____/node_modules/mysql/lib/protocol/Protocol.js:38:16)
      at Socket.<anonymous> (_____/node_modules/mysql/lib/Connection.js:88:28)
      at Socket.<anonymous> (_____/node_modules/mysql/lib/Connection.js:526:10)
      at Socket.emit (node:events:511:28)
      at Socket.emit (node:domain:489:12)
      at Readable.read (node:internal/streams/readable:547:10)
      at Socket.read (node:net:765:39)
      at flow (node:internal/streams/readable:1031:34)
      --------------------
      at Protocol._enqueue (_____/node_modules/mysql/lib/protocol/Protocol.js:144:48)
      at Immediate._start (_____/node_modules/@rodrigogs/zongji/index.js:240:31)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'PROTOCOL_PACKETS_OUT_OF_ORDER',
    fatal: true
  }

I patched @rodrigogs/zongji's common.js as follows:

diff --git a/node_modules/@rodrigogs/zongji/lib/common.js b/node_modules/@rodrigogs/zongji/lib/common.js
index 89bf7ed..1cb9c6f 100644
--- a/node_modules/@rodrigogs/zongji/lib/common.js
+++ b/node_modules/@rodrigogs/zongji/lib/common.js
@@ -440,7 +440,7 @@ exports.readMysqlValue = function(
       } else {
         if (column.charset !== null) {
           // Javascript UTF8 always allows up to 4 bytes per character
-          column.charset = column.charset === 'utf8mb4' ? 'utf8' : column.charset;
+          column.charset = column.charset.startsWith('utf8mb') ? 'utf8' : column.charset;
           parser._encoding = column.charset;
         }
         result = parser.parseString(size);
@@ -457,7 +457,7 @@ exports.readMysqlValue = function(
       // e.g. TINYTEXT, MEDIUMTEXT, LONGTEXT, TEXT data types
       if (column.charset !== null) {
         // Javascript UTF8 always allows up to 4 bytes per character
-        column.charset = column.charset === 'utf8mb4' ? 'utf8' : column.charset;
+        column.charset = column.charset.startsWith('utf8mb') ? 'utf8' : column.charset;
         result = iconv.decode(result, column.charset);
       }
       break;

Maybe this can be of help to someone out there (I couldn't find any information about these errors and issue).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash when parsing utf8mb3 BLOB columns (MariaDB 10.7) Tag: v0.5.1
2 participants