Types: Add support for BINARY columns by amotl · Pull Request #12 · crate/sqlalchemy-cratedb

amotl · 2023-12-23T12:47:27Z

About

Improvements to be mainlined from https://github.com/pyveci/supertask/blob/19316e2/supertask/store/cratedb.py. The support for binary data types is being emulated by serializing them to STRING using base64. In that way, pickled data can be stored and retrieved without further ado.

matriv

Keep in mind that CrateDB also supports BitString type, but I don't think that's what we want here, right?

amotl · 2023-12-24T00:30:11Z

Keep in mind that CrateDB also supports BitString type, but I don't think that's what we want here, right?

Right. Here, it is about providing an emulation for a BLOB-type column. However, expanding type support to include the BIT type will be nice to have for the sake of completeness. So far, I didn't need it, but I am sure others will.

amotl · 2023-12-24T00:33:27Z

+    def bind_processor(self, dialect):
+        if dialect.dbapi is None:
+            return None
+
+        # TODO: DBAPIBinary = dialect.dbapi.Binary
+
+        def process(value):
+            if value is not None:
+                # TODO: return DBAPIBinary(value)
+                return base64.b64encode(value).decode()
+            else:
+                return None
+
+        return process


Note to self, or others who want to pick this up:

Review that detail about returning a DBAPIBinary, or not.

@amotl Did you review it?

No. I just staged the patch directly from its origin, where the code is in use and works well. However, it can still be wrong from a generic perspective.

Let's toggle this patch back into draft mode: Better safe than sorry, thanks! The other commit 465b55b has been removed here and diverged to a separate patch.

Types: Improve support for FLOAT type, converging to FLOAT vs. DOUBLE #265

Hi @bgunebakan,

Review that detail about returning a DBAPIBinary, or not.

No matter how the downstream supertask package currently consumes this API, I think we should aim for making it standards-compliant and then adjust supertask correspondingly, when possible. In this spirit, I think this comment is the most prominent example what we are referring to:

# TODO: return DBAPIBinary(value)

Only because we "unlocked" this feature by harmonizing sqlalchemy-cratedb with supertask in a bare-bones way to satisfy some basic explorations the other day doesn't automatically mean that the currently implemented interface is correct yet.

With kind regards,
Andreas.

amotl · 2023-12-24T00:35:32Z

+    # Python 3 has native bytes() type
+    # both sqlite3 and pg8000 seem to return it,
+    # psycopg2 as of 2.5 returns 'memoryview'
+    def result_processor(self, dialect, coltype):
+        if dialect.returns_native_bytes:
+            return None
+
+        def process(value):
+            if value is not None:
+                return base64.b64decode(value)
+            return value
+
+        return process


Note to self, or others who want to pick this up:

Review that detail about dialect.returns_native_bytes: Should it be handled differently, because, well, base64.b64decode actually returns native bytes already?

@amotl Did you review it?

Thanks for your comment. I've responded more extensively above. Yes, I agree this patch should get more love before considering it ready, optimally in harmony with its consumer site in supertask.

coderabbitai · 2026-06-02T20:38:30Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5c41c1ac-be88-4447-a012-55aa02a3e31e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch amo/more-types

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

kneth

We need tests for BLOBs.

kneth · 2026-06-04T11:55:30Z

+        if dialect.dbapi is None:
+            return None
+
+        # TODO: DBAPIBinary = dialect.dbapi.Binary


Can it be removed?

... or activated, to be more standards-compliant. Let's investigate within another iteration before getting things wrong. Thanks!

kneth · 2026-06-04T11:56:03Z

+class LargeBinary(sa.String):
+    """A type for large binary byte data.
+
+    The :class:`.LargeBinary` type corresponds to a large and/or unlengthed


I suggest that we also mention CrateDB' BLOB type here

CrateDB does not provide a BLOB type, does it?

BLOB might require some tweaks to add it to our SQL Alchemy variant. SinceBLOB is only working over HTTP(s), it could be natural to add support for it.

Thanks. The CrateDB BLOB support you are mentioning is not made through a data type, but a table variant instead. In this spirit, it is a totally different animal.

kneth · 2026-06-04T11:57:24Z

+    def bind_processor(self, dialect):
+        if dialect.dbapi is None:
+            return None
+
+        # TODO: DBAPIBinary = dialect.dbapi.Binary
+
+        def process(value):
+            if value is not None:
+                # TODO: return DBAPIBinary(value)
+                return base64.b64encode(value).decode()
+            else:
+                return None
+
+        return process


@amotl Did you review it?

kneth · 2026-06-04T11:57:36Z

+    # Python 3 has native bytes() type
+    # both sqlite3 and pg8000 seem to return it,
+    # psycopg2 as of 2.5 returns 'memoryview'
+    def result_processor(self, dialect, coltype):
+        if dialect.returns_native_bytes:
+            return None
+
+        def process(value):
+            if value is not None:
+                return base64.b64decode(value)
+            return value
+
+        return process


@amotl Did you review it?

amotl changed the title ~~Types: Add support for BINARY columns and improve support for FLOATs~~ [TYPES] Add support for BINARY columns and improve support for FLOATs Dec 23, 2023

amotl requested review from hlcianfagna, matriv and surister December 23, 2023 12:50

matriv approved these changes Dec 23, 2023

View reviewed changes

amotl commented Dec 24, 2023

View reviewed changes

amotl changed the base branch from amo/postgresql-async to amo/fix-inspector January 15, 2024 21:34

amotl force-pushed the amo/more-types branch 2 times, most recently from 9941c5d to 2a34862 Compare January 15, 2024 21:36

amotl force-pushed the amo/fix-inspector branch from 2f08b2a to 43c45fb Compare January 15, 2024 21:50

amotl force-pushed the amo/more-types branch from 2a34862 to 2800d5d Compare January 15, 2024 21:51

amotl changed the title ~~[TYPES] Add support for BINARY columns and improve support for FLOATs~~ Types: Add support for BINARY columns and improve support for FLOATs Jan 15, 2024

amotl force-pushed the amo/fix-inspector branch 5 times, most recently from 6ac0a22 to d2c7613 Compare June 13, 2024 14:33

Base automatically changed from amo/fix-inspector to main June 13, 2024 14:38

amotl force-pushed the amo/more-types branch 3 times, most recently from 8f01308 to 73bfe8e Compare June 25, 2024 14:51

amotl mentioned this pull request Mar 29, 2025

Default type map float64 -> float is wrong #212

Closed

amotl force-pushed the amo/more-types branch 3 times, most recently from 2707929 to de288ab Compare March 30, 2025 02:20

amotl mentioned this pull request Mar 30, 2025

Preview: Bundle a few improvements #215

Closed

amotl force-pushed the amo/more-types branch 2 times, most recently from 09c55fc to 9d1fc10 Compare March 30, 2025 03:54

github-advanced-security AI found potential problems Mar 30, 2025

View reviewed changes

Comment thread tests/test_support_pandas.py Fixed

amotl force-pushed the amo/more-types branch from 9d1fc10 to a8dc4ea Compare March 30, 2025 03:57

amotl mentioned this pull request Aug 15, 2025

Timestamp columns with time zone map to UserDefinedType rather than TIMESTAMP #226

Closed

amotl force-pushed the amo/more-types branch from a8dc4ea to 465b55b Compare June 2, 2026 20:38

amotl requested review from bgunebakan and removed request for hlcianfagna and surister June 2, 2026 20:42

amotl marked this pull request as ready for review June 2, 2026 20:42

amotl requested a review from kneth June 3, 2026 10:02

kneth requested changes Jun 4, 2026

View reviewed changes

amotl marked this pull request as draft June 4, 2026 16:30

amotl changed the title ~~Types: Add support for BINARY columns and improve support for FLOATs~~ Types: Add support for BINARY columns Jun 4, 2026

amotl force-pushed the amo/more-types branch from 465b55b to 4fd1f77 Compare June 4, 2026 16:35

Types: Add support for BLOB type, per base64 encoding

b504343

amotl force-pushed the amo/more-types branch from 4fd1f77 to b504343 Compare June 4, 2026 21:09

bgunebakan self-assigned this Jun 12, 2026

bgunebakan added 2 commits June 12, 2026 15:20

Merge branch 'main' into amo/more-types

576a0ca

Merge branch 'main' into amo/more-types

4dafb65

bgunebakan added the enhancement New feature or request label Jun 17, 2026

Fix formatting by removing unnecessary whitespace

08157ed

Conversation

amotl commented Dec 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Uh oh!

matriv left a comment • edited by amotl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl commented Dec 24, 2023

Uh oh!

amotl Dec 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Dec 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

kneth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

amotl commented Dec 23, 2023 •

edited

Loading

matriv left a comment •

edited by amotl

Loading

amotl Dec 24, 2023 •

edited

Loading

amotl Jun 4, 2026 •

edited

Loading

amotl Dec 24, 2023 •

edited

Loading

coderabbitai Bot commented Jun 2, 2026 •

edited

Loading

amotl Jun 8, 2026 •

edited

Loading