Skip to content

feat: add %%sql --limit and --no-display support#1883

Open
djanand wants to merge 1 commit into
apache:mainfrom
djanand:feat/add-limit-no-display-support
Open

feat: add %%sql --limit and --no-display support#1883
djanand wants to merge 1 commit into
apache:mainfrom
djanand:feat/add-limit-no-display-support

Conversation

@djanand

@djanand djanand commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

The %%sql Jupyter cell magic advertises two options in its help text that were never actually implemented:
--no-display - Don't display results
--limit N - Limit displayed rows (default: 50)
The argument-parsing loop only extracted a variable name, it scanned all tokens and kept the last non-flag, never breaking, silently dropping any --flag. So the documented options were ignored. This PR makes the behavior match
the documented contract, including the (default: 50) already promised in the help text.

What changes are included in this PR?

In python/python/ballista/jupyter.py:

  • Added _parse_cell_magic_args(), which parses --limit N and --no-display plus the optional variable name, using the same hand-rolled line.split() convention as the other magics in this module (%ballista, %register). Invalid --limit values (missing / non-integer / non-positive) return a clear message instead of raising.
  • --limit N caps the display to N rows via datafusion's configure_formatter(max_rows=N, min_rows=N). This is display-only — it never truncates the underlying data, so an in-query LIMIT always takes effect. When --limit is omitted, the display falls back to a default of 50.
  • --no-display runs the query, stores the result in the variable if one is given, and renders nothing.
  • Updated the sql() docstring and the %ballista help text.

Docs: added a short --limit / --no-display example to the "SQL Magic Commands" section of python/README.md.

Tests in python/python/tests/test_jupyter.py:

  • A cluster-free unit test for the argument parser (valid cases + invalid --limit rejection).
  • An integration test covering --limit (display cap set, data not truncated), the default cap, --no-display, and the invalid---limit error path.

Are there any user-facing changes?

Yes, the previously documented --limit and --no-display options on the %%sql cell magic now work. The README is updated accordingly.
The only behavioral change to existing usage: a %%sql cell now renders up to 50 rows by default (the value the help text already documented here) instead of datafusion's built-in default of 10. This affects rendered rows only. Query results are never truncated, and in-query LIMIT is unaffected.

Note: --limit is implemented via datafusion's global HTML formatter, so each %%sql cell re-sets the display cap. The cap also applies to subsequent non-%%sql
DataFrame displays in the same kernel until changed

@djanand

djanand commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

@coderfender @martin-g - ptal

@djanand

djanand commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Design note: why not use @magic_arguments?

IPython's @magic_arguments / parse_argstring is the conventional way to parse magic options (and would give --limit=5 and type validation for free). I deliberately avoided it here:

  • This module is written to import and be unit-tested without IPython — it provides stub decorators for the IPYTHON_AVAILABLE == False path, ipython is only an optional [jupyter] extra (not a base or dev dependency), and the test suite imports BallistaMagics with no IPython installed. magic_arguments/parse_argstring are not stubbed and would be imported unconditionally, breaking that IPython-less path.
  • The hand-rolled line.split() parsing matches the existing convention used by the other magics in this file (%ballista, %register); there is no argparse/magic_arguments usage anywhere in the module today.

If the project would prefer the @magic_arguments style, that'd be better done as a separate refactor converting all the magics at once IMO.

@djanand

djanand commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Why python release build fails ?

python/ is a separate Cargo workspace (excluded from the main one) with its own Cargo.lock, and it path-depends on the main ballista crates. But:

  • Dependabot only manages directory: "/" (the main workspace) there's no /python entry. So every dep bump updates /Cargo.lock and never python/Cargo.lock.
  • CI does not catches the drift early, it only surfaces later as the cryptic --locked maturin error.

So the lock goes stale on main itself every time deps move, and the next contributor inherits a red build. Kindly let me know your thoughts

edit: Raised #1887

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant