What happened?
After picking up @op-engineering/op-sqlite@16.2.1, I started seeing persisted JSON become invalid after writing parsed sync data into SQLite.
The server response was valid JSON and the parsed JS objects were correct, but after writing those objects through db.execute(..., params), non-ASCII characters were stored as the low byte of their UTF-16 code unit. In some cases that turns valid JSON into invalid JSON.
I validated this with a small repro in a React Native app using op-sqlite@16.2.1.
Repro:
const value = JSON.stringify({
bullet: 'Kimball Wildlife Refuge • Burlingame State Park',
smartQuotes: 'John “Jack” Doe'
})
await db.execute('CREATE TABLE IF NOT EXISTS op_sqlite_unicode_repro (value TEXT)')
await db.execute('DELETE FROM op_sqlite_unicode_repro')
await db.execute('INSERT INTO op_sqlite_unicode_repro (value) VALUES (?)', [value])
const result = await db.execute('SELECT value FROM op_sqlite_unicode_repro')
console.log(result.rows[0]?.value)Expected stored/selected value:
{"bullet":"Kimball Wildlife Refuge • Burlingame State Park","smartQuotes":"John “Jack” Doe"}Actual value stored in SQLite:
{"bullet":"Kimball Wildlife Refuge " Burlingame State Park","smartQuotes":"John <0x1c>Jack<0x1d> Doe"}
Actual SQLite hex for the stored value:
7B2262756C6C6574223A224B696D62616C6C2057696C646C696665205265667567652022204275726C696E67616D65205374617465205061726B222C22736D61727451756F746573223A224A6F686E201C4A61636B1D20446F65227D
The specific byte transformations are:
U+2022 BULLET should be UTF-8 e2 80 a2, but is stored as 22
U+201C LEFT QUOTE should be UTF-8 e2 80 9c, but is stored as 1c
U+201D RIGHT QUOTE should be UTF-8 e2 80 9d, but is stored as 1d
That looks like UTF-16 code units are being truncated to char instead of encoded as UTF-8.
I only see this with 16.2.1. The same app's lockfile had 16.2.0, and that version still appears to use value.asString(rt).utf8(rt) in cpp/utils.cpp.
This looks related to the string conversion change introduced in #415 and released in 16.2.1.
op-sqlite version
16.2.1
React Native version
0.85.3
Reproducible Example
https://github.com/rwjblue/op-sqlite-string-roundtrip-issue
What happened?
After picking up
@op-engineering/op-sqlite@16.2.1, I started seeing persisted JSON become invalid after writing parsed sync data into SQLite.The server response was valid JSON and the parsed JS objects were correct, but after writing those objects through
db.execute(..., params), non-ASCII characters were stored as the low byte of their UTF-16 code unit. In some cases that turns valid JSON into invalid JSON.I validated this with a small repro in a React Native app using
op-sqlite@16.2.1.Repro:
Expected stored/selected value:
{"bullet":"Kimball Wildlife Refuge • Burlingame State Park","smartQuotes":"John “Jack” Doe"}Actual value stored in SQLite:
Actual SQLite hex for the stored value:
The specific byte transformations are:
That looks like UTF-16 code units are being truncated to
charinstead of encoded as UTF-8.I only see this with
16.2.1. The same app's lockfile had16.2.0, and that version still appears to usevalue.asString(rt).utf8(rt)incpp/utils.cpp.This looks related to the string conversion change introduced in #415 and released in
16.2.1.op-sqlite version
16.2.1
React Native version
0.85.3
Reproducible Example
https://github.com/rwjblue/op-sqlite-string-roundtrip-issue