Add Ollama provider and streaming translation output support#461
Open
Shungy wants to merge 9 commits into
Open
Add Ollama provider and streaming translation output support#461Shungy wants to merge 9 commits into
Shungy wants to merge 9 commits into
Conversation
do not reinvent the wheel
If provider has streaming feature, `window.py::_on_translation` now chooses `stream_translate` over `translate`. Hence it is redundant to have a separate `translate` function in the provider.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds Ollama as a provider and enables streaming translation response.
Ollama.streaming.translation.in.Dialect.mp4
Using Ollama API one can either add local or remote instances for LLM based translations. LLMs provide versatility the traditional translation engines cannot provide. For example, the translation above is using translategemma:12b locally, and it is fairly correct for an archaic piece of text from 17th century. Google Translate completely fails for such texts. Another strength is that it does not require specifying the source language nor it tries to explicitly detect it, because it translates based on understanding and meaning regardless of the classification of the source language. This makes its usage very simple as one does not need to modify the source language constantly. That being said, LLM-based translations can suffer from hallucinations, leading to outright incorrect translations due to prior bias, missing knowledge, or knowledge conflict. All in all, this method has its strengths and weaknesses, and it can be very useful if one is aware of its limitations.
In order to have a proper support for the Ollama provider, the PR also adds
ENGINESandSTREAMINGprovider features. The engines is simply an additional setting to specify which LLM model to use. I have named it "engine" for it to be more universal, as other non-LLM APIs might offer this setting. The streaming feature is for continuous translation responses. Since LLMs do not wait for the entire response to be complete, and instead output results token by token, it made sense to add this feature. Other than it is nice to see the output as it is being translated, it is also a better UX because LLM generation can take a long time depending on the model and the hardware being used. One feature that is missing is the inability to stop the translations, which I will either add to this PR or create a separate PR depending on the feedback.The Ollama provider also has bunch of supported languages listed by default. Since we cannot query an LLM on the languages it supports, I believed the listed languages is a good amount. The more languages listed would offer a bad experience due to required scrolling, and would be more likely to misinform users on which languages a given model supports. The less languages listed would artificially limit the users on which target language they can translate to. I think this is an acceptable compromise without having to rework the UI.