Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .unicode_files
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Files that have known unicode chars; all others should be ascii-only
.github/workflows/codeql-analysis.yml
AUTHORS.md
docs/usage.rst
test/protocol/old/test_compact.py
test/protocol/old/test_types.py
test/protocol/schemas/test_codec_types.py
72 changes: 24 additions & 48 deletions docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,75 +19,51 @@ Bleeding-Edge

.. code:: bash

git clone https://github.com/dpkp/kafka-python
pip install ./kafka-python
pip install git+https://github.com/dpkp/kafka-python.git


Optional crc32c install
***********************
Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python`
uses a new message protocol version, that requires calculation of `crc32c`,
which differs from the `zlib.crc32` hash implementation. By default `kafka-python`
calculates it in pure python, which is quite slow. To speed it up we optionally
support https://pypi.python.org/pypi/crc32c package if it's installed.
Optional Installs
*****************

crc32c
======
Highly recommended for performance optimization. By default `kafka-python`
calculates record checksums in pure python, but the calculation is somewhat
CPU intensive. As throughput increases this can become a bottleneck. Installing
the optional ``crc32c`` dependency reduces the CPU cost of each check using
an optimized C library. See https://pypi.python.org/pypi/crc32c .

.. code:: bash

pip install 'kafka-python[crc32c]'


Optional ZSTD install
*********************
zstd
====

To enable ZSTD compression/decompression, install python-zstandard:
To enable ZSTD compression/decompression, install `python-zstandard`:

>>> pip install 'kafka-python[zstd]'


Optional LZ4 install
********************
lz4
===

To enable LZ4 compression/decompression, install python-lz4:
To enable LZ4 compression/decompression, install `python-lz4`:

>>> pip install 'kafka-python[lz4]'


Optional Snappy install
***********************

Install Development Libraries
=============================

Download and build Snappy from https://google.github.io/snappy/

Ubuntu:

.. code:: bash

apt-get install libsnappy-dev

OSX:
snappy
======

.. code:: bash

brew install snappy

From Source:
To enable Snappy compression/decompression, install `python-snappy`:

.. code:: bash

wget https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
tar xzvf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make
sudo make install

Install Python Module
=====================

Install the `python-snappy` module
pip install 'kafka-python[snappy]'

.. code:: bash

pip install 'kafka-python[snappy]'
Note that python-snappy generally does not publish pre-compiled wheels,
so installation may require building the snappy library from source.
See https://google.github.io/snappy/ .
10 changes: 4 additions & 6 deletions docs/tests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,17 @@ Tests
.. image:: https://img.shields.io/github/actions/workflow/status/dpkp/kafka-python/python-package.yml
:target: https://github.com/dpkp/kafka-python/actions/workflows/python-package.yml

The test suite is run via pytest.

Linting is run via pylint.

Test coverage details are currently published as an html build artifact.
Testing uses pytest and pylint.

The test suite includes unit tests that mock network interfaces, mock broker tests
that simulate request/receive network messaging, as well as integration tests that
setup and teardown kafka broker (and zookeeper where required) fixtures.

Test coverage details are currently published as an html build artifact.


Unit tests
------------------
----------

To run the tests locally, install test dependencies:

Expand Down
Loading
Loading