docs: Add Command-Line docs with sphinx-argparse

.unicode_files

@@ -1,7 +1,7 @@
 # Files that have known unicode chars; all others should be ascii-only
 .github/workflows/codeql-analysis.yml
 AUTHORS.md
-docs/usage.rst
+docs/high_level.rst
 test/protocol/old/test_compact.py
 test/protocol/old/test_types.py
 test/protocol/schemas/test_codec_types.py

docs/cli/admin.rst

@@ -0,0 +1,113 @@
+kafka-python admin
+******************
+
+``kafka-python admin`` exposes :class:`~kafka.KafkaAdminClient`
+operations as a command-line tool. Commands are grouped by the kind of
+resource they act on (topics, partitions, configs, ...). Each group
+contains one or more subcommands.
+
+Output is printed to stdout. ``--format raw`` (the default) uses
+``pprint``; ``--format json`` emits a single JSON document, useful for
+piping to ``jq`` or other tooling.
+
+.. code-block:: bash
+
+    kafka-python admin -b localhost:9092 cluster describe
+    kafka-python admin -b localhost:9092 --format json topics list
+    python -m kafka.admin -b localhost:9092 topics create -t foo --num-partitions 3
+
+
+Global options
+==============
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :nosubcommands:
+
+
+acls
+====
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: acls
+
+
+cluster
+=======
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: cluster
+
+
+configs
+=======
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: configs
+
+
+topics
+======
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: topics
+
+
+partitions
+==========
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: partitions
+
+
+groups
+======
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: groups
+
+
+transactions
+============
+
+KIP-664 administrative tools for inspecting and recovering from hanging
+transactions. ``list``, ``describe``, and ``describe-producers`` require
+broker >= 3.0 (``describe-producers`` works against broker >= 2.8).
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: transactions
+
+
+users
+=====
+
+SCRAM credential management. See KIP-554.
+
+.. argparse::
+   :module: kafka.cli.admin
+   :func: main_parser
+   :prog: kafka-python admin
+   :path: users

docs/cli/consumer.rst

@@ -0,0 +1,33 @@
+kafka-python consumer
+*********************
+
+A line-oriented console consumer. Subscribes to one or more topics and
+prints each received record to standard output.
+
+Examples
+========
+
+.. code-block:: bash
+
+    # Read every record on a topic until interrupted
+    kafka-python consumer -b localhost:9092 -t my-topic
+
+    # Join a consumer group and commit offsets automatically
+    kafka-python consumer -b localhost:9092 -g my-group -t my-topic
+
+    # Read from the beginning, then exit after 1s of idle
+    kafka-python consumer -b localhost:9092 -t my-topic \
+        -C auto_offset_reset=earliest \
+        -C consumer_timeout_ms=1000
+
+    # Print full ConsumerRecord (topic, partition, offset, key, value, ...)
+    kafka-python consumer -b localhost:9092 -t my-topic -f full
+
+
+Reference
+=========
+
+.. argparse::
+   :module: kafka.cli.consumer
+   :func: main_parser
+   :prog: kafka-python consumer

docs/cli/index.rst

@@ -0,0 +1,83 @@
+Command-Line Interface
+**********************
+
+kafka-python ships simple command-line interfaces for consumer, producer,
+and admin clients. They can be invoked either as the ``kafka-python``
+console script or as module entry points:
+
+.. code-block:: bash
+
+    kafka-python consumer -b localhost:9092 -t my-topic
+    kafka-python producer -b localhost:9092 -t my-topic
+    kafka-python admin    -b localhost:9092 cluster describe
+
+    # equivalent module invocations
+    python -m kafka.consumer -b localhost:9092 -t my-topic
+    python -m kafka.producer -b localhost:9092 -t my-topic
+    python -m kafka.admin    -b localhost:9092 cluster describe
+
+The ``kafka-python admin`` command, in particular, is a convenient
+alternative to the apache kafka ``bin/`` scripts when a compatible JVM is
+not available.
+
+.. toctree::
+   :maxdepth: 2
+
+   consumer
+   producer
+   admin
+
+
+Common Options
+==============
+
+All three commands share a common set of connection, logging, and
+configuration options. They are documented in full on the individual
+command pages; the summary below highlights the most commonly used
+flags.
+
+Connection
+----------
+
+``-b/--bootstrap-servers HOST:PORT``
+    One or more bootstrap servers used to discover the rest of the
+    cluster. May be supplied multiple times.
+
+``-S/--security-protocol``
+    One of ``PLAINTEXT``, ``SSL``, ``SASL_PLAINTEXT``, ``SASL_SSL``.
+    Defaults to ``PLAINTEXT``.
+
+``-M/--sasl-mechanism``
+    One of ``PLAIN``, ``GSSAPI``, ``OAUTHBEARER``, ``SCRAM-SHA-256``,
+    ``SCRAM-SHA-512``. Defaults to ``PLAIN``.
+
+``-U/--sasl-user`` / ``-P/--sasl-password``
+    Credentials for SASL ``PLAIN`` and ``SCRAM-*`` mechanisms.
+
+Logging
+-------
+
+``-l/--log-level``
+    Python ``logging`` level (``DEBUG``, ``INFO``, ...). Defaults to
+    ``CRITICAL`` so the CLI is quiet by default.
+
+``-L/--enable-logger`` / ``-D/--disable-logger``
+    Selectively turn on or off a single logger by name. Both flags may
+    be supplied multiple times.
+
+Extended Configuration
+----------------------
+
+``-C/--extra-config key=value``
+    Pass arbitrary keyword arguments through to the underlying client
+    constructor (:class:`~kafka.KafkaConsumer`,
+    :class:`~kafka.KafkaProducer`, or
+    :class:`~kafka.KafkaAdminClient`). Values that parse as ``int``,
+    ``True``, ``False``, or ``None`` are converted; everything else is
+    passed through as a string. May be supplied multiple times.
+
+    .. code-block:: bash
+
+        kafka-python consumer -b localhost:9092 -t foo \
+            -C auto_offset_reset=earliest \
+            -C consumer_timeout_ms=1000

docs/cli/producer.rst

@@ -0,0 +1,25 @@
+kafka-python producer
+*********************
+
+A line-oriented console producer. Reads lines from standard input and
+publishes each one as a record to the configured topic.
+
+Examples
+========
+
+.. code-block:: bash
+
+    # Publish a single message
+    echo "hello kafka" | kafka-python producer -b localhost:9092 -t my-topic
+
+    # Stream lines from a file
+    kafka-python producer -b localhost:9092 -t my-topic < messages.txt
+
+
+Reference
+=========
+
+.. argparse::
+   :module: kafka.cli.producer
+   :func: main_parser
+   :prog: kafka-python producer

docs/conf.py

@@ -20,6 +20,10 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('../'))
 
+# Suppress ANSI color escapes from argparse 3.14+ help/usage output captured
+# by sphinx-argparse.
+os.environ['NO_COLOR'] = '1'
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -34,6 +38,7 @@
     'sphinx.ext.viewcode',
     'sphinx.ext.napoleon',
     'sphinx_rtd_theme',
+    'sphinxarg.ext',
 ]
 
 # Add any paths that contain templates here, relative to this directory.