diff --git a/docs/_static/custom.css b/docs/_static/custom.css new file mode 100644 index 000000000..87902d727 --- /dev/null +++ b/docs/_static/custom.css @@ -0,0 +1,7 @@ +/* The sphinx_rtd_theme forces `white-space: nowrap` on every data-table cell. + For the Supported KIPs table (class `kip-table`) we want every column + except the KIP number/link (column 1) to wrap onto multiple lines. */ +.wy-table-responsive table.kip-table td:not(:first-child), +.wy-table-responsive table.kip-table th:not(:first-child) { + white-space: normal; +} diff --git a/docs/compatibility.rst b/docs/compatibility.rst index 3db062d19..471e04ff9 100644 --- a/docs/compatibility.rst +++ b/docs/compatibility.rst @@ -13,7 +13,8 @@ Because the kafka server protocol is backwards compatible, kafka-python is expected to work with newer broker releases as well. Although kafka-python is tested and expected to work on recent broker versions, -not all features are supported. Please see github open issues for feature tracking. +not all features are supported. See :doc:`kips` for per-feature coverage of +Kafka Improvement Proposals, or the github open issues for feature tracking. PRs welcome! kafka-python is tested on python 3.8-3.14. diff --git a/docs/conf.py b/docs/conf.py index 1ea662f93..9b7175412 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -134,7 +134,10 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -#html_static_path = ['_static'] +html_static_path = ['_static'] + +# Custom stylesheets, loaded after the theme's own CSS. +html_css_files = ['custom.css'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied diff --git a/docs/index.rst b/docs/index.rst index 050b293ed..3a4fa7271 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -261,6 +261,7 @@ See https://docs.python.org/3/howto/logging.html for overview / howto. install tests compatibility + Supported KIPs support license changelog diff --git a/docs/kips.rst b/docs/kips.rst new file mode 100644 index 000000000..2e30de0b2 --- /dev/null +++ b/docs/kips.rst @@ -0,0 +1,900 @@ +Supported KIPs +============== + +`Kafka Improvement Proposals (KIPs) `_ +describe the protocol and feature changes that ship in each Apache Kafka +release. This page tracks how kafka-python covers the client-facing KIPs. The +KIP set and the Kafka-release column follow the excellent list maintained by +the `franz-go `_ project. + +Because the wire protocol is generated from the upstream Kafka JSON schemas +(``kafka/protocol/schemas/resources/``), the protocol classes track many KIPs +automatically. The **Status** column reflects client-facing behavior in +kafka-python -- whether the feature is actually usable through +``KafkaProducer``, ``KafkaConsumer``, or ``KafkaAdminClient`` -- not merely +whether the wire format exists. Note that kafka-python is a *client* library: +broker- and controller-internal protocols (KRaft, inter-broker replication, +etc.) are intentionally out of scope. + +This table layout was borrowed from `franz-go `__ +because we liked it so much! It was originally constructed here by Claude, +but it is maintained with a caring human touch. + +Status legend +------------- + +* **Supported** -- usable through the kafka-python client or admin API. +* **Partial** -- partially implemented; the status links to the notes below. +* **Protocol only** -- wire/protocol classes exist under ``kafka/protocol/`` + but the client does not yet drive or expose the feature. +* ``--`` -- not implemented / not supported. + +KIPs +---- + +.. list-table:: + :header-rows: 1 + :widths: 12 38 14 24 12 + :class: kip-table + + * - KIP + - Title + - Kafka release + - Status + - kafka-python release + + * - `KIP-1 `__ + - Disallow acks > 1 + - 0.8.3 + - Supported + - 1.0.0 + * - `KIP-8 `__ + - Flush method on Producer + - 0.8.3 + - Supported + - 1.0.0 + * - `KIP-4 `__ + - Basic Admin APIs + - 0.9.0 + - Supported + - 1.0.0 + * - `KIP-12 `__ + - SASL & SSL + - 0.9.0 + - Supported + - 1.1.0 + * - `KIP-13 `__ + - Throttling (on broker) + - 0.9.0 + - Supported + - 2.1.0 + * - `KIP-15 `__ + - Producer close with a timeout + - 0.9.0 + - Supported + - 1.0.0 + * - `KIP-19 `__ + - Request timeouts + - 0.9.0 + - Supported + - 1.0.0 + * - `KIP-22 `__ + - Custom partitioners + - 0.9.0 + - Supported + - 1.0.0 + * - `KIP-40 `__ + - ListGroups and DescribeGroups + - 0.9.0 + - Supported + - 1.4.4 + * - `KIP-31 `__ + - Relative offsets in message sets + - 0.10.0 + - Supported + - 1.2.0 + * - `KIP-32 `__ + - Timestamps in message set v1 + - 0.10.0 + - Supported + - 1.2.0 + * - `KIP-35 `__ + - ApiVersion + - 0.10.0 + - Supported + - 1.3.0 + * - `KIP-41 `__ + - max.poll.records + - 0.10.0 + - Supported + - 1.0.0 + * - `KIP-42 `__ + - Producer & consumer interceptors + - 0.10.0 + - -- + - -- + * - `KIP-43 `__ + - SASL PLAIN & handshake + - 0.10.0 + - Supported + - 1.3.0 + * - `KIP-57 `__ + - Interoperable lz4 framing + - 0.10.0 + - Supported + - 1.2.0 + * - `KIP-62 `__ + - background heartbeats & improvements + - 0.10.1 + - Supported + - 1.4.0 + * - `KIP-70 `__ + - On{Assigned,Revoked} + - 0.10.1 + - Supported + - 2.1.3 + * - `KIP-74 `__ + - Fetch response size limits + - 0.10.1 + - Supported + - 2.1.3 + * - `KIP-78 `__ + - ClusterID in Metadata + - 0.10.1 + - Supported + - 1.3.3 + * - `KIP-79 `__ + - List offsets for times + - 0.10.1 + - Supported + - 1.3.4 + * - `KIP-81 `__ + - Bound fetch memory usage + - 0.10.1 + - :ref:`Partial ` + - 1.0.0 + * - `KIP-84 `__ + - SASL SCRAM + - 0.10.2 + - Supported + - 2.0.0 + * - `KIP-86 `__ + - SASL Callbacks + - 0.10.2 + - :ref:`Partial ` + - 1.4.6 + * - `KIP-88 `__ + - OffsetFetch for admins + - 0.10.2 + - Supported + - 1.3.3 + * - `KIP-97 `__ + - Backwards compat for old brokers + - 0.10.2 + - Supported + - 1.0.0 + * - `KIP-102 `__ + - Consumer close timeouts + - 0.10.2 + - Supported + - 1.0.0 + * - `KIP-108 `__ + - CreateTopic validate only field + - 0.10.2 + - Supported + - 2.0.0 + * - `KIP-54 `__ + - Sticky partitioning + - 0.11.0 + - Supported + - 2.0.2 + * - `KIP-82 `__ + - Record headers + - 0.11.0 + - Supported + - 1.4.0 + * - `KIP-98 `__ + - EOS + - 0.11.0 + - Supported + - 2.2.0 + * - `KIP-101 `__ + - OffsetForLeaderEpoch v0 + - 0.11.0 + - Supported + - 1.4.0 + * - `KIP-107 `__ + - DeleteRecords + - 0.11.0 + - Supported + - 2.1.0 + * - `KIP-117 `__ + - Admin client + - 0.11.0 + - Supported + - 1.4.4 + * - `KIP-124 `__ + - Request rate quotas + - 0.11.0 + - Supported + - 2.1.0 + * - `KIP-126 `__ + - Ensure proper batch size after compression + - 0.11.0 + - Supported + - 1.0.0 + * - `KIP-133 `__ + - Describe & Alter configs + - 0.11.0 + - Supported + - 2.0.0 + * - `KIP-140 `__ + - ACLs + - 0.11.0 + - Supported + - 1.4.7 + * - `KIP-144 `__ + - Broker reconnect backoff + - 0.11.0 + - Supported + - 1.3.4 + * - `KIP-112 `__ + - Broker request protocol changes + - 1.0 + - Supported + - 1.4.4 + * - `KIP-113 `__ + - LogDir requests + - 1.0 + - Supported + - 1.4.4 + * - `KIP-152 `__ + - More SASL; SASLAuthenticate + - 1.0 + - Supported + - 2.1.0 + * - `KIP-185 `__ + - Idempotency is default + - 1.0 + - Supported + - 2.2.0 + * - `KIP-192 `__ + - Cleaner idempotence semantics + - 1.0 + - Supported + - 2.2.0 + * - `KIP-195 `__ + - CreatePartitions + - 1.0 + - Supported + - 1.4.4 + * - `KIP-48 `__ + - Delegation tokens + - 1.1 + - -- + - -- + * - `KIP-204 `__ + - DeleteRecords via admin API + - 1.1 + - Supported + - 2.1.0 + * - `KIP-226 `__ + - Describe configs v1 + - 1.1 + - Supported + - 2.0.0 + * - `KIP-227 `__ + - Incremental fetch + - 1.1 + - Supported + - 2.1.0 + * - `KIP-229 `__ + - DeleteGroups + - 1.1 + - Supported + - 2.0.2 + * - `KIP-219 `__ + - Client-side throttling + - 2.0 + - Supported + - 2.1.0 + * - `KIP-222 `__ + - Group operations via admin API + - 2.0 + - Supported + - 1.4.4 + * - `KIP-249 `__ + - Delegation tokens in admin API + - 2.0 + - -- + - -- + * - `KIP-255 `__ + - SASL OAUTHBEARER + - 2.0 + - Supported + - 1.4.6 + * - `KIP-266 `__ + - Fix indefinite consumer timeouts + - 2.0 + - :ref:`Partial ` + - 1.0.0 + * - `KIP-279 `__ + - OffsetForLeaderEpoch bump + - 2.0 + - Supported + - 3.0.0 + * - `KIP-294 `__ + - TLS verification + - 2.0 + - Supported + - 1.4.6 + * - `KIP-91 `__ + - Intuitive producer timeouts + - 2.1 + - Supported + - 2.2.0 + * - `KIP-110 `__ + - zstd + - 2.1 + - Supported + - 2.0.2 + * - `KIP-302 `__ + - Use multiple addrs for resolved hostnames + - 2.1 + - Supported + - 2.0.0 + * - `KIP-320 `__ + - Fetcher: detect log truncation + - 2.1 + - Supported + - 3.0.0 + * - `KIP-322 `__ + - DeleteTopics disabled error code + - 2.1 + - Supported + - 2.1.0 + * - `KIP-342 `__ + - OAUTHBEARER extensions + - 2.1 + - Supported + - 1.4.6 + * - `KIP-357 `__ + - List ACLs per principal via admin API + - 2.1 + - Supported + - 1.4.7 + * - `KIP-390 `__ + - Configurable compression level + - 2.1 + - -- + - -- + * - `KIP-183 `__ + - Elect preferred leaders + - 2.2 + - Supported + - 2.1.0 + * - `KIP-207 `__ + - New error in ListOffsets + - 2.2 + - Supported + - 2.3.0 + * - `KIP-289 `__ + - Default group.id to null + - 2.2 + - Supported + - 2.2.0 + * - `KIP-368 `__ + - Periodically reauthenticate SASL + - 2.2 + - Supported + - 3.0.0 + * - `KIP-389 `__ + - Group max size error + - 2.2 + - Supported + - 2.1.0 + * - `KIP-394 `__ + - Require member.id for initial join request + - 2.2 + - Supported + - 2.2.0 + * - `KIP-339 `__ + - IncrementalAlterConfigs + - 2.3 + - Supported + - 3.0.0 + * - `KIP-341 `__ + - Sticky group bugfix + - 2.3 + - Supported + - 2.0.3 + * - `KIP-361 `__ + - Allow disable auto topic creation + - 2.3 + - Supported + - 2.1.0 + * - `KIP-430 `__ + - Authorized ops in DescribeGroups + - 2.3 + - Supported + - 2.3.0 + * - `KIP-345 `__ + - Static group membership + - 2.4 + - Supported + - 2.3.0 + * - `KIP-369 `__ + - An always round robin produce partitioner + - 2.4 + - -- + - -- + * - `KIP-392 `__ + - Closest replica fetching w/ rack + - 2.4 + - Supported + - 3.0.0 + * - `KIP-396 `__ + - Commit offsets manually + - 2.4 + - Supported + - 2.0.0 + * - `KIP-412 `__ + - Dynamic log levels w/ IncrementalAlterConfigs + - 2.4 + - Supported + - 3.0.0 + * - `KIP-429 `__ + - Consumer incremental rebalance + - 2.4 + - Supported + - 3.0.0 + * - `KIP-455 `__ + - Replica reassignment API + - 2.4 + - Supported + - 3.0.0 + * - `KIP-460 `__ + - Leader election API + - 2.4 + - Supported + - 2.1.0 + * - `KIP-464 `__ + - CreateTopic defaults + - 2.4 + - Supported + - 2.0.0 + * - `KIP-467 `__ + - Per-record error codes when producing + - 2.4 + - Supported + - 2.3.0 + * - `KIP-480 `__ + - Sticky partition producing + - 2.4 + - Supported + - 3.0.0 + * - `KIP-482 `__ + - Tagged fields (KAFKA-8885) + - 2.4 + - Supported + - 3.0.0 + * - `KIP-496 `__ + - OffsetDelete admin command + - 2.4 + - Supported + - 3.0.0 + * - `KIP-511 `__ + - Client name/version in ApiVersions request + - 2.4 + - Supported + - 2.1.3 + * - `KIP-514 `__ + - Bounded Flush + - 2.4 + - Supported + - 1.0.0 + * - `KIP-525 `__ + - CreateTopics v5 returns configs + - 2.4 + - Supported + - 2.0.0 + * - `KIP-360 `__ + - Safe epoch bumping for ``UNKNOWN_PRODUCER_ID`` + - 2.5 + - Supported + - 3.0.0 + * - `KIP-447 `__ + - Producer scalability for EOS + - 2.5 + - Supported + - 3.0.0 + * - `KIP-526 `__ + - Reduce metadata lookups + - 2.5 + - Supported + - 2.0.0 + * - `KIP-533 `__ + - Default API timeout (total time, not per request) + - 2.5 + - :ref:`Partial ` + - 1.0.0 + * - `KIP-546 `__ + - Client Quota APIs + - 2.5 + - Protocol only + - -- + * - `KIP-559 `__ + - Protocol info in sync/join + - 2.5 + - Supported + - 3.0.0 + * - `KIP-518 `__ + - List groups by state + - 2.6 + - Supported + - 3.0.0 + * - `KIP-519 `__ + - Configurable SSL "engine" + - 2.6 + - Supported + - 2.0.0 + * - `KIP-568 `__ + - Explicit rebalance triggering on the consumer + - 2.6 + - -- + - -- + * - `KIP-569 `__ + - Docs & type in DescribeConfigs + - 2.6 + - Supported + - 3.0.0 + * - `KIP-580 `__ + - Exponential backoff + - 2.6 + - Supported + - 3.0.0 + * - `KIP-584 `__ + - Versioning scheme for features + - 2.6 + - Supported + - 3.0.0 + * - `KIP-602 `__ + - Use all resolved addrs by default + - 2.6 + - Supported + - 2.0.0 + * - `KIP-554 `__ + - Broker side SCRAM APIs + - 2.7 + - Supported + - 3.0.0 + * - `KIP-588 `__ + - Producer recovery from txn timeout + - 2.7 + - Supported + - 3.0.0 + * - `KIP-595 `__ + - New APIs for raft protocol + - 2.7 + - Supported + - 3.0.0 + * - `KIP-599 `__ + - Throttling on create/delete topic/partition + - 2.7 + - Supported + - 2.1.0 + * - `KIP-601 `__ + - Configurable socket connection timeout + - 2.7 + - Supported + - 3.0.0 + * - `KIP-651 `__ + - Support PEM + - 2.7 + - Supported + - 2.0.0 + * - `KIP-654 `__ + - Aborted txns with unflushed data is not fatal + - 2.7 + - Supported + - 2.3.0 + * - `KIP-516 `__ + - Topic IDs + - 2.8 + - Supported + - 3.0.0 + * - `KIP-700 `__ + - DescribeCluster + - 2.8 + - Supported + - 3.0.0 + * - `KIP-664 `__ + - Admin API for DescribeProducers + - 3.0 + - Supported + - 3.0.0 + * - `KIP-679 `__ + - Strongest producer guarantee by default + - 3.0 + - Supported + - 3.0.0 + * - `KIP-699 `__ + - Batch FindCoordinators + - 3.0 + - Supported + - 3.0.0 + * - `KIP-709 `__ + - Batch OffsetFetch + - 3.0 + - Supported + - 3.0.0 + * - `KIP-734 `__ + - Support MaxTimestamp in ListOffsets + - 3.0 + - Supported + - 3.0.0 + * - `KIP-735 `__ + - Bump default session timeout + - 3.0 + - Supported + - 3.0.0 + * - `KIP-768 `__ + - Extend SASL/OAUTHBEARER support for OIDC + - 3.1 + - :ref:`Partial ` + - 1.4.6 + * - `KIP-784 `__ + - Add ErrorCode to DescribeLogDirs response + - 3.1 + - Supported + - 3.0.0 + * - `KIP-800 `__ + - Reason in Join/Leave group + - 3.1 + - Supported + - 3.0.0 + * - `KIP-814 `__ + - SkipAssignment for static group leaders + - 3.1 + - :ref:`Partial ` + - 3.0.0 + * - `KIP-373 `__ + - Users can create delegation tokens for others + - 3.3 + - -- + - -- + * - `KIP-794 `__ + - Better sticky partitioning + - 3.3 + - :ref:`Partial ` + - 3.0.0 + * - `KIP-827 `__ + - ``DescribeLogDirs.{Total,Usable}Bytes`` + - 3.3 + - Supported + - 3.0.0 + * - `KIP-836 `__ + - ``DescribeQuorum`` voter lag info + - 3.3 + - Supported + - 3.0.0 + * - `KIP-851 `__ + - ``RequireStable`` on OffsetFetch + - 3.3 + - :ref:`Partial ` + - 3.0.0 + * - `KIP-792 `__ + - Generation field in consumer group protocol + - 3.4 + - Supported + - 3.0.0 + * - `KIP-405 `__ + - Kafka Tiered Storage + - 3.5 + - Supported + - 3.0.0 + * - `KIP-881 `__ + - Rack-aware consumer partition assignment + - 3.5 + - -- + - -- + * - `KIP-893 `__ + - Nullable structs in the protocol + - 3.5 + - Supported + - 3.0.0 + * - `KIP-903 `__ + - Stale broker epoch fencing + - 3.5 + - Protocol only + - -- + * - `KIP-714 `__ + - Client Metrics + - 3.7 + - -- + - -- + * - `KIP-848 `__ + - Next gen consumer rebalance protocol + - 3.7 + - -- + - -- + * - `KIP-919 `__ + - Admin client to KRaft, Controller registration + - 3.7 + - -- + - -- + * - `KIP-951 `__ + - Leader discovery optimizations + - 3.7 + - Protocol only + - -- + * - `KIP-966 `__ + - Eligible leader replicas (protocol) + - 3.7 + - Supported + - 3.0.0 + * - `KIP-1000 `__ + - ListClientMetricsResources + - 3.7 + - -- + - -- + * - `KIP-890 `__ + - Transactions server side defense + - 3.8, 4.0 + - :ref:`Partial ` + - -- + * - `KIP-899 `__ + - Allow clients to rebootstrap + - 3.8 + - -- + - -- + * - `KIP-994 `__ + - List/Describe transactions enhancements + - 3.8 + - -- + - -- + * - `KIP-853 `__ + - Add replica directory ID for replica fetchers + - 3.9 + - Supported + - 3.0.0 + * - `KIP-1005 `__ + - ListOffsets w. Timestamp -5 + - 3.9 + - Supported + - 3.0.0 + * - `KIP-1025 `__ + - URL-encode clientID/secret in OAuth auth header + - 3.9 + - :ref:`Partial ` + - 1.4.6 + * - `KIP-1022 `__ + - Formatting changes for features + - 4.0 + - Supported + - 3.0.0 + * - `KIP-1043 `__ + - Administration of groups + - 4.0 + - Supported + - 3.0.0 + * - `KIP-1073 `__ + - DescribeCluster.IsFenced + - 4.0 + - Protocol only + - -- + * - `KIP-1075 `__ + - TimeoutMillis on ListOffsets + - 4.0 + - Protocol only + - -- + * - `KIP-1076 `__ + - User provided client metrics + - 4.0 + - -- + - -- + * - `KIP-1082 `__ + - ClientID in the next-gen rebalancer + - 4.0 + - -- + - -- + * - `KIP-1102 `__ + - RebootstrapRequired + - 4.0 + - -- + - -- + * - `KIP-1139 `__ + - Oauth JWT bearer grant + - 4.0 + - :ref:`Partial ` + - 1.4.6 + * - `KIP-860 `__ + - Client side AlterPartitionAssignments RF change guard + - 4.1 + - Supported + - 3.0.0 + * - `KIP-932 `__ + - Share groups (queues) + - 4.1 + - -- + - -- + * - `KIP-1142 `__ + - ListConfigResources + - 4.1 + - Supported + - 3.0.0 + * - `KIP-1152 `__ + - ListTransactions.TransactionalIDPattern + - 4.1 + - -- + - -- + * - `KIP-1023 `__ + - ListOffsets earliest pending upload offset + - 4.2 + - -- + - -- + * - `KIP-1071 `__ + - Streams group protocol + - 4.2 + - -- + - -- + * - `KIP-1160 `__ + - Per-broker supported features in ApiVersions + - 4.2 + - Protocol only + - -- + * - `KIP-1206 `__ + - ShareFetch strict record limit + - 4.2 + - -- + - -- + * - `KIP-1222 `__ + - Share consumer renew acknowledgements + - 4.2 + - -- + - -- + * - `KIP-1226 `__ + - Share partition lag in DescribeShareGroupOffsets + - 4.2 + - -- + - -- + * - `KIP-1227 `__ + - Rack ID in (Share)MemberDescription + - 4.2 + - -- + - -- + * - `KIP-1258 `__ + - OAuth client assertion in client_credentials grant + - 4.3 + - :ref:`Partial ` + - 1.4.6 + * - `KIP-498 `__ + - Max bound on reads + - WIP + - Supported + - 3.0.0 + + +Notes +----- + +.. _partial-entries: + +* **Partial entries** -- KIP-81 (fetch memory is bounded by ``fetch_max_bytes`` + / ``max_partition_fetch_bytes`` rather than a global buffer); KIP-86 and + KIP-768 / KIP-1025 / KIP-1139 / KIP-1258 (SASL/OAUTHBEARER is driven by a + user-supplied ``AbstractTokenProvider``, so OIDC, JWT-bearer, and client + assertion grants are the application's responsibility); KIP-266 / KIP-533 + (per-call timeouts exist, but there is no ``default.api.timeout.ms``); + KIP-794 / KIP-814 (the sticky producer partitioner and cooperative-sticky + assignor are implemented, but not these specific refinements); KIP-890 + (transaction RPCs are capped at + ``AddPartitionsToTxn``/``EndTxn``/``TxnOffsetCommit`` v3 and ``Produce`` v9, + below the KIP-890 versions). +* **Not supported** (shown as ``--`` in the table) -- notable client-facing gaps + include KIP-42 (interceptors), KIP-48 / KIP-249 / KIP-373 (delegation tokens), + KIP-369 (round-robin partitioner), KIP-568 (explicit rebalance trigger), + KIP-714 / KIP-1000 / KIP-1076 (client telemetry), KIP-848 / KIP-1082 (next-generation consumer + group protocol), KIP-881 (rack-aware assignment), KIP-899 / KIP-1102 (client + re-bootstrap), and KIP-932 / KIP-1071 / KIP-1206 / KIP-1222 (share and streams + groups).