Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 88 additions & 1 deletion docs/apidoc/KafkaAdminClient.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,90 @@
KafkaAdminClient
================

.. autoclass:: kafka.KafkaAdminClient
:members:
:inherited-members:

:class:`~kafka.KafkaAdminClient` is composed of functional mixins. The
sections below group its methods by area.


Topics
------

.. automethod:: kafka.KafkaAdminClient.list_topics
.. automethod:: kafka.KafkaAdminClient.describe_topics
.. automethod:: kafka.KafkaAdminClient.create_topics
.. automethod:: kafka.KafkaAdminClient.wait_for_topics
.. automethod:: kafka.KafkaAdminClient.delete_topics


Partitions
----------

.. automethod:: kafka.KafkaAdminClient.create_partitions
.. automethod:: kafka.KafkaAdminClient.delete_records
.. automethod:: kafka.KafkaAdminClient.elect_leaders
.. automethod:: kafka.KafkaAdminClient.alter_partition_reassignments
.. automethod:: kafka.KafkaAdminClient.list_partition_reassignments
.. automethod:: kafka.KafkaAdminClient.describe_topic_partitions
.. automethod:: kafka.KafkaAdminClient.list_partition_offsets


Configs
-------

.. automethod:: kafka.KafkaAdminClient.describe_configs
.. automethod:: kafka.KafkaAdminClient.list_config_resources
.. automethod:: kafka.KafkaAdminClient.alter_configs
.. automethod:: kafka.KafkaAdminClient.reset_configs


Consumer Groups
---------------

.. automethod:: kafka.KafkaAdminClient.list_groups
.. automethod:: kafka.KafkaAdminClient.describe_groups
.. automethod:: kafka.KafkaAdminClient.list_group_offsets
.. automethod:: kafka.KafkaAdminClient.alter_group_offsets
.. automethod:: kafka.KafkaAdminClient.reset_group_offsets
.. automethod:: kafka.KafkaAdminClient.delete_group_offsets
.. automethod:: kafka.KafkaAdminClient.delete_groups
.. automethod:: kafka.KafkaAdminClient.remove_group_members


ACLs
----

.. automethod:: kafka.KafkaAdminClient.describe_acls
.. automethod:: kafka.KafkaAdminClient.create_acls
.. automethod:: kafka.KafkaAdminClient.delete_acls


Cluster
-------

.. automethod:: kafka.KafkaAdminClient.describe_cluster
.. automethod:: kafka.KafkaAdminClient.describe_log_dirs
.. automethod:: kafka.KafkaAdminClient.alter_replica_log_dirs
.. automethod:: kafka.KafkaAdminClient.describe_metadata_quorum
.. automethod:: kafka.KafkaAdminClient.get_broker_version_data
.. automethod:: kafka.KafkaAdminClient.api_versions
.. automethod:: kafka.KafkaAdminClient.describe_features
.. automethod:: kafka.KafkaAdminClient.update_features


Transactions
------------

.. automethod:: kafka.KafkaAdminClient.list_transactions
.. automethod:: kafka.KafkaAdminClient.describe_transactions
.. automethod:: kafka.KafkaAdminClient.describe_producers
.. automethod:: kafka.KafkaAdminClient.abort_transaction
.. automethod:: kafka.KafkaAdminClient.find_hanging_transactions


Users
-----

.. automethod:: kafka.KafkaAdminClient.alter_user_scram_credentials
.. automethod:: kafka.KafkaAdminClient.describe_user_scram_credentials
2 changes: 0 additions & 2 deletions docs/apidoc/KafkaConnection.rst

This file was deleted.

2 changes: 0 additions & 2 deletions docs/apidoc/KafkaConnectionManager.rst

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
ClusterMetadata
===============

.. autoclass:: kafka.cluster.ClusterMetadata
:members:
:undoc-members:
10 changes: 10 additions & 0 deletions docs/apidoc/misc/ConsumerRebalanceListener.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
ConsumerRebalanceListener
=========================

.. autoclass:: kafka.ConsumerRebalanceListener
:members:
:undoc-members:

.. autoclass:: kafka.AsyncConsumerRebalanceListener
:members:
:undoc-members:
6 changes: 6 additions & 0 deletions docs/apidoc/misc/IsolationLevel.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
IsolationLevel
==============

.. autoclass:: kafka.IsolationLevel
:members:
:undoc-members:
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
OffsetAndMetadata
=================

.. autoclass:: kafka.OffsetAndMetadata
:members:
6 changes: 6 additions & 0 deletions docs/apidoc/misc/OffsetSpec.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
OffsetSpec
==========

.. autoclass:: kafka.OffsetSpec
:members:
:undoc-members:
6 changes: 6 additions & 0 deletions docs/apidoc/misc/Partitioner.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Partitioner
===========

.. autoclass:: kafka.partitioner.Partitioner
:members:
:undoc-members:
18 changes: 18 additions & 0 deletions docs/apidoc/misc/Serializer.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Serializer
==========

.. autoclass:: kafka.Serializer
:members:
:undoc-members:

.. autoclass:: kafka.Deserializer
:members:
:undoc-members:

.. autoclass:: kafka.JsonSerializer
:members:
:undoc-members:

.. autoclass:: kafka.DefaultSerializer
:members:
:undoc-members:
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
TopicPartition
==============

.. autoclass:: kafka.TopicPartition
:members:
103 changes: 95 additions & 8 deletions docs/apidoc/modules.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,101 @@
kafka-python API
****************

The :mod:`kafka` package exposes a small set of public classes for working
with Apache Kafka. The following sections group them by role; each class
links to its own API reference page.


kafka
=====

Application code typically interacts with one of three top-level clients.
Each owns a background IO thread and a shared async networking layer.

- :class:`~kafka.KafkaConsumer` - high-level, group-aware message consumer.
Iterable, with manual or automatic offset commits, cooperative rebalance,
pluggable deserializers, and transactional-read isolation.
- :class:`~kafka.KafkaProducer` - high-level, asynchronous message producer.
Batches records into a background sender thread, with optional
idempotence, transactions, compression, and pluggable serializers.
- :class:`~kafka.KafkaAdminClient` - admin operations: topic, ACL, config,
consumer group, partition, quota, log-directory, and quorum management.

.. toctree::
:maxdepth: 1
:hidden:

KafkaConsumer <KafkaConsumer>
KafkaProducer <KafkaProducer>
KafkaAdminClient <KafkaAdminClient>


kafka.net
=========

The clients share a single async networking layer (``kafka.net``). These
classes are exposed for advanced use cases - embedding the connection
pool, building a custom client on top of the kafka.net event loop, or
driving the protocol layer directly from the REPL.

- :mod:`~kafka.net.manager` - connection pool and
high-level facade over the shared IO event loop. Each top-level client
owns one.
- :mod:`~kafka.net.connection` - per-broker async
connection: state machine, request/response correlation, and SASL
handshake.
- :mod:`~kafka.net.transport` - Async socket I/O with write buffering,
pause/resume hooks, and the asyncio-shaped protocol callback surface.
- :mod:`~kafka.net.inet` - DNS lookup + non-blocking connect, plus a
URL-scheme registry that resolves ``proxy_url`` to socket factories.
- :mod:`~kafka.net.http_connect` - Tunnels broker connections through
an HTTP CONNECT proxy (RFC 7231).
- :mod:`~kafka.net.socks5` - SOCKS5 client with optional username/password
authentication.

.. toctree::
:maxdepth: 1
:hidden:

manager <net/manager>
connection <net/connection>
transport <net/transport>
inet <net/inet>
http_connect <net/http_connect>
socks5 <net/socks5>


other / misc
============

Lightweight data types used throughout the client APIs (and useful when
working with the lower-level protocol layer).

- :class:`~kafka.cluster.ClusterMetadata` - in-memory cache of brokers,
topics, partitions, and the active controller. Refreshes itself on the
shared IO thread.
- :class:`~kafka.TopicPartition` - namedtuple identifying a partition as
``(topic, partition)``.
- :class:`~kafka.OffsetAndMetadata` - committed-offset record
``(offset, metadata, leader_epoch)``.
- :class:`~kafka.OffsetSpec` - enum for partition offset queries.
- :class:`~kafka.IsolationLevel` - enum for transactional isolation.
- :class:`~kafka.Serializer` - base class for serialization / deserialization
of key and value bytes. Includes helper classes `~kafka.DefaultSerializer`
and `~kafka.JsonSerializer`.
- :class:`~kafka.ConsumerRebalanceListener` - base class for consumer
class to receive join/rebalance group hooks. Also includes async
interface `~kafka.AsyncConsumerRebalanceListener`.

.. toctree::
:maxdepth: 1
:hidden:

KafkaConsumer
KafkaProducer
KafkaAdminClient
KafkaConnectionManager
KafkaConnection
ClusterMetadata
OffsetAndMetadata
TopicPartition
ClusterMetadata <misc/ClusterMetadata>
TopicPartition <misc/TopicPartition>
OffsetAndMetadata <misc/OffsetAndMetadata>
OffsetSpec <misc/OffsetSpec>
IsolationLevel <misc/IsolationLevel>
Serializer <misc/Serializer>
ConsumerRebalanceListener <misc/ConsumerRebalanceListener>
Partitioner <misc/Partitioner>
8 changes: 8 additions & 0 deletions docs/apidoc/net/connection.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
kafka.net.connection
====================

.. module:: kafka.net.connection

.. autoclass:: kafka.net.connection.KafkaConnection
:members:
:undoc-members:
8 changes: 8 additions & 0 deletions docs/apidoc/net/http_connect.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
kafka.net.http_connect
======================

.. module:: kafka.net.http_connect

.. autoclass:: kafka.net.http_connect.HttpConnectProxy
:members:
:undoc-members:
10 changes: 10 additions & 0 deletions docs/apidoc/net/inet.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
kafka.net.inet
==============

.. module:: kafka.net.inet

.. autoclass:: kafka.net.inet.KafkaNetSocket
:members:
:undoc-members:

.. autofunction:: kafka.net.inet.create_connection
8 changes: 8 additions & 0 deletions docs/apidoc/net/manager.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
kafka.net.manager
=================

.. module:: kafka.net.manager

.. autoclass:: kafka.net.manager.KafkaConnectionManager
:members:
:undoc-members:
8 changes: 8 additions & 0 deletions docs/apidoc/net/socks5.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
kafka.net.socks5
================

.. module:: kafka.net.socks5

.. autoclass:: kafka.net.socks5.Socks5Proxy
:members:
:undoc-members:
12 changes: 12 additions & 0 deletions docs/apidoc/net/transport.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
kafka.net.transport
===================

.. module:: kafka.net.transport

.. autoclass:: kafka.net.transport.KafkaTCPTransport
:members:
:undoc-members:

.. autoclass:: kafka.net.transport.KafkaSSLTransport
:members:
:undoc-members:
4 changes: 3 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,9 @@
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
html_theme_options = {
'navigation_depth': -1,
}

# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
Expand Down
9 changes: 4 additions & 5 deletions kafka/admin/_partitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -510,7 +510,7 @@ def _list_partition_offsets_process_response(response):
)
return results

async def _async_list_partition_offsets(self, topic_partition_specs, isolation_level='read_uncommitted', timeout_ms=None):
async def _async_list_partition_offsets(self, topic_partition_specs, isolation_level=IsolationLevel.READ_UNCOMMITTED, timeout_ms=None):
isolation_level = IsolationLevel.build_from(isolation_level)
timer = Timer(self._validate_timeout(timeout_ms))
backoff_secs = self.config['retry_backoff_ms'] / 1000
Expand Down Expand Up @@ -539,7 +539,7 @@ async def _async_list_partition_offsets(self, topic_partition_specs, isolation_l
await self._net.sleep(min(backoff_secs, max(0.0, timer.timeout_secs or 0.0)))
return results

def list_partition_offsets(self, topic_partition_specs, isolation_level='read_uncommitted', timeout_ms=None):
def list_partition_offsets(self, topic_partition_specs, isolation_level=IsolationLevel.READ_UNCOMMITTED, timeout_ms=None):
"""Look up offsets for the given partitions by spec.

Partitions are routed to their respective leader brokers via cluster
Expand All @@ -554,9 +554,8 @@ def list_partition_offsets(self, topic_partition_specs, isolation_level='read_un
wire-level sentinel).

Keyword Arguments:
isolation_level (str, optional): One of ``'read_uncommitted'``
(default) or ``'read_committed'``. ``read_committed`` requires
broker support for ListOffsets v2+.
isolation_level (IsolationLevel, optional): Requires broker support
for ListOffsets v2+. Default: IsolationLevel.READ_UNCOMMITTED.
timeout_ms (int, optional): Maximum time to spend retrying
NotLeaderForPartitionError. Default: ``request_timeout_ms``.

Expand Down
Loading
Loading