From ad935549f4d4eb15b6b150eb83b4f2ab5eaeec14 Mon Sep 17 00:00:00 2001 From: William Woodruff Date: Fri, 17 Apr 2026 12:20:04 -0400 Subject: [PATCH 1/3] Draft for freezing the HTML simple repository API Signed-off-by: William Woodruff Lint Signed-off-by: William Woodruff More lint Signed-off-by: William Woodruff Deprecation -> freezing Signed-off-by: William Woodruff Mention optimization problems Signed-off-by: William Woodruff Tweak conneg language Signed-off-by: William Woodruff Update spon/delegate Signed-off-by: William Woodruff Precision Signed-off-by: William Woodruff --- peps/pep-9999.rst | 233 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..b2e5fe1b9f3 --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,233 @@ +PEP: 9999 +Title: Freezing the HTML simple repository API +Author: William Woodruff +Sponsor: Donald Stufft +PEP-Delegate: Donald Stufft +Discussions-To: Pending +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 16-Apr-2026 +Post-History: `13-Apr-2026 `__ + + +Abstract +======== + +This PEP proposes freezing the +:ref:`standard HTML representation ` +of the simple repository API, as originally specified in :pep:`503` +and updated over subsequent PEPs. + +In this context of this PEP, "freezing" means that the HTML representation +is considered complete from the perspective of the standards process, +and **SHOULD NOT** be updated by future PEPs. Future PEPs **SHOULD** instead +target the +:ref:`standard JSON representation `, +as originally specified in :pep:`691`. + +Similarly, this PEP's freezing of the HTML representation does **not** stipulate +that installers should remove support for the HTML representation, or that +indices (like PyPI) will or should stop providing an HTML representation. + +Rationale and Motivation +======================== + +The use of an HTML representation for Python package indices predates +efforts to standardize Python packaging. Consequently, the HTML representation +standardized with :pep:`503` represents a *formalization* of +existing practices (particularly those of PyPI), rather than a *design*. + +The HTML representation of a Python package index has served the Python +packaging ecosystem admirably: it has acted as the baseline representation +that all indices and installers support, and has allowed PyPI to incrementally +modernize its index presentation while maintaining backwards compatibility +with installers and mirrors. :pep:`629`, :pep:`714`, :pep:`740`, +:pep:`792`, and many others demonstrate the viability of this approach. + +At the same time, the HTML representation has several limitations that +have become increasingly apparent and salient as Python packaging as a whole +has modernized: + +- The HTML representation is *rigid*, for backwards compatibility reasons. + This rigidity makes it difficult to represent new pieces of metadata, + and PEPs that attempt to do so typically need to shoehorn their changes + into ```` tags or ``data-`` attributes to avoid interfering with + assumptions that existing consumers make about the structure of the HTML. + + This shoehorning process also requires PEPs that modify the HTML index + to invent syntax for encoding structured data. For example, :pep:`792` + adds meta tags named ``pypi:project-status`` and + ``pypi:project-status-reason``, effectively flattening an object + representation that appears naturally in the JSON representation. + + Similarly, the HTML representation's rigidity makes it an optimization + barrier: :pep:`658` allows indices to serve distribution metadata via + the simple repository API, but the absence of a straightforward and + backwards-compatible way to encode that metadata within the HTML + representation means that installers must incur an additional HTTP round-trip + to fetch relatively small amounts of information. :pep:`740` adopts a + similar approach, with similar overhead repercussions. + + In practice, some index PEPs have chosen not to modify the HTML representation + at all, and instead focus solely on the JSON representation. :pep:`700` + for example introduces both per-distribution metadata *and* a top-level + ``versions`` key to the JSON representation, but does not modify the HTML + representation. The original rationale for this was that HTML consumers + would be unlikely to need the new metadata, + +- Relatedly, third-party consumption of the HTML representation is often + *brittle*: even syntactically valid, non-semantic changes to PyPI's HTML + representation are + `known to cause breakage `__ + due to unsound assumptions about the exact structure of the HTML, including + its whitespace. + + Consumption of the JSON representation, by contrast, is more robust to + non-semantic changes thanks to the prevalence of robust JSON parsing + libraries. Robust handling of HTML is naturally possible, but consumers + are often *tempted* to avoid the perceived complexity and generality + of HTML parsing in favor of brittle approaches involving regular expressions + and similar ad-hoc parsing techniques. + +- In practice, *adoption* of incremental improvements to the HTML representation + is limited: PyPI itself typically adopts new features, but third-party + indices (particularly those sold as corporate offerings) frequently provide + only the absolute minimum representation originally defined in :pep:`503`. + + As a result, *even when* the HTML representation is improved, many consumers + do not benefit from those improvements. + +Put together, these limitations mean that the HTML representation is (1) +often difficult to extend in a robust way, (2) *de facto* frozen with +respect to how many consumers interact with Python packaging, even +when standards processes work to modernize it. + +The purpose of this PEP is to formalize this status quo. + +Specification +============= + +The HTML representation of the simple repository API is frozen +for the purposes of Python packaging standards processes. Future +Python packaging PEPs **SHOULD NOT** modify the HTML representation of the +simple repository API, and **MUST** instead modify the JSON representation. + +This PEP does not alter the status of the HTML representation on PyPI +and does not prescribe any behavioral changes for installers. + +One functional consequence of this freeze is that future changes +to the simple repository API will be +:ref:`versioned ` as they are +currently, but that only the JSON representation will receive changes +to its versioning marker. For example, if a future PEP introduces +version 1.5 of the simple repository API, the HTML representation will retain +the following versioning marker: + +.. code-block:: html + + + +Future Considerations +===================== + +This PEP does not stipulate any changes to how indices and installers should +handle the HTML representation. + +As of April 2026, the prospect of *fully* removing support for the HTML +representation from either indices or installers is unrealistic: it is simply +too critical to the ecosystem, and efforts to remove it would be extremely +and unreasonably disruptive. + +However, it is not *inconceivable* that the HTML representation could be +fully removed (or relegated to legacy/default-disabled flows) in the future. +This PEP does not preclude such a future, but does not propose it either. + +The Python packaging community has made several valuable observations +around behaviors that make outright removal of the HTML representation +difficult or infeasible, including: + +- By virtue of being the default, the HTML representation is extremely + easy to adopt internally: it doesn't require any (explicit) content + negotiation, and can often be served trivially by a CDN or a minimal + HTTP server (like ``python -m http.server``). + + The JSON representation does not technically require content negotiation + either, but in practice clients that consume it expect to perform + explicit content negotiation due to the assumption that the same URL + provides both representations. Consequently, any future efforts to remove the + HTML representation will likely require a simpler adoption story for the JSON + representation. + +- The HTML representation is currently easier for installers like ``pip`` + to parse incrementally, as the Python standard library includes + ``html.parser`` for incremental HTML parsing. This helps mitigate + the memory overhead of large HTML index responses, e.g. detail responses + for packages that have hundreds or thousands of distributions. + + By contrast, Python's standard library currently lacks an incremental + JSON parser. Incremental JSON parsing is not impractical (and is strictly + less complex than incremental HTML parsing), but the absence of a + standard library solution presents an adoption barrier. + Future efforts to remove the HTML representation will likely require a robust + standard library (or acceptably vendorable third-party) solution for + incremental JSON parsing within ``pip``. + +Security Implications +===================== + +This PEP does not identify and positive or negative security implications +associated with freezing the HTML representation of the simple repository +API. + +How to Teach This +================= + +Because this PEP only freezes the HTML representation of the simple repository +API for the purposes of Python packaging standards processes, the end user +implications of this PEP are limited. + +However, for third-party indices that wish to modernize their index +representations, this PEP proposes the following if accepted: + +- The authors of this PEP will coordinate with the maintainers + of PyPI on appropriate public-facing documentation and communication, + including an announcement on the `PyPI blog `_ + if deemed appropriate. + +- The authors of this PEP will make appropriate changes to the + :ref:`living standard ` for the simple + repository API, including admonitions and callouts where appropriate + to indicate that the HTML representation will not receive future updates. + +Rejected Ideas +============== + +Doing nothing +------------- + +Doing nothing is always an option. Per above, this would be a continuation +of the status quo, wherein the HTML representation is updated on paper +(and on PyPI), but is frozen in practice in third-party settings. + +The authors of this PEP believe that being explicit about the status +of the HTML representation is valuable, and would benefit future standards +efforts by diverting design effort away from shoehorning new features +into the HTML representation. + + +Aggressively removing the HTML representation +--------------------------------------------- + +Encouraging indices and installers to aggressively remove support for the HTML +representation is another option. However, as noted above, this is unrealistic +in the near term, and would be disruptive to the ecosystem. + +The authors of this PEP believe that freezing is a more gradual and +pragmatic approach that better reflects the ecosystem's reality. + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. From 1ab5a02ccb810510bb255d4116a2c99b48c2f12b Mon Sep 17 00:00:00 2001 From: William Woodruff Date: Mon, 20 Apr 2026 12:33:59 -0400 Subject: [PATCH 2/3] Assign PEP 833 Signed-off-by: William Woodruff --- .github/CODEOWNERS | 1 + peps/{pep-9999.rst => pep-0833.rst} | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) rename peps/{pep-9999.rst => pep-0833.rst} (99%) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e3585e55f60..f3cf311daf1 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -708,6 +708,7 @@ peps/pep-0829.rst @warsaw peps/pep-0830.rst @gpshead peps/pep-0831.rst @pablogsal @Fidget-Spinner @savannahostrowski peps/pep-0832.rst @brettcannon +peps/pep-0833.rst @dstufft # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-9999.rst b/peps/pep-0833.rst similarity index 99% rename from peps/pep-9999.rst rename to peps/pep-0833.rst index b2e5fe1b9f3..8278a16e641 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-0833.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 833 Title: Freezing the HTML simple repository API Author: William Woodruff Sponsor: Donald Stufft From 6030e4a7848ad0ebf480a2aa97644a89d7520cd9 Mon Sep 17 00:00:00 2001 From: William Woodruff Date: Mon, 20 Apr 2026 12:35:04 -0400 Subject: [PATCH 3/3] Revert "Assign PEP 833" This reverts commit 1ab5a02ccb810510bb255d4116a2c99b48c2f12b. --- .github/CODEOWNERS | 1 - peps/{pep-0833.rst => pep-9999.rst} | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) rename peps/{pep-0833.rst => pep-9999.rst} (99%) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f3cf311daf1..e3585e55f60 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -708,7 +708,6 @@ peps/pep-0829.rst @warsaw peps/pep-0830.rst @gpshead peps/pep-0831.rst @pablogsal @Fidget-Spinner @savannahostrowski peps/pep-0832.rst @brettcannon -peps/pep-0833.rst @dstufft # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0833.rst b/peps/pep-9999.rst similarity index 99% rename from peps/pep-0833.rst rename to peps/pep-9999.rst index 8278a16e641..b2e5fe1b9f3 100644 --- a/peps/pep-0833.rst +++ b/peps/pep-9999.rst @@ -1,4 +1,4 @@ -PEP: 833 +PEP: 9999 Title: Freezing the HTML simple repository API Author: William Woodruff Sponsor: Donald Stufft