Implementation of LLM-Assisted Translations Backend #3100

ayanaar · 2024-02-15T21:11:59Z

Addresses Issue #3068

This PR introduces the backend infrastructure for the LLM-Assisted Translations in Pontoon project. The implementation is divided into two components:

openai_service.py: This module serves as the interface to the OpenAI API, using the gpt-4-0125-preview model. It defines prompts for three translation refinements—informal, formal, and alternative—tailoring the output to meet specific stylistic or formal requirements.
refine_translation.py: A Django management command that facilitates user interaction with the translation refinement feature directly from the command line. Users can input an English source string along with a machine translation in any given language and specify the desired refinement type. The command returns a refined translation.

This backend setup allows users to use the text generation capabilities of GPT-4 for enhancing translation quality and adaptability.

Usage Instructions:

To utilize the translation refinement feature, execute the following command in the terminal, substituting <characteristic> with informal, formal, or alternative based on the required refinement type:

python manage.py refine_translation <characteristic> "English text" "Translated text" "locale_name"

Examples:

For an Alternative Translation:

python manage.py refine_translation alternative "Email resent. Add { $accountsEmail } to your contacts to ensure smooth delivery." "E-pošta ponovno poslana. Dodajte { $accountsEmail } med svoje stike in si zagotovite nemoteno dostavo." "Slovenian"

For an Informal Translation:

python manage.py refine_translation informal "Your subscription has been confirmed." "Vaša naročnina je bila potrjena." "Slovenian"

For an Formal Translation:

python manage.py refine_translation formal "We appreciate your feedback." "Vaše povratne informacije cenimo." "Slovenian"

Handling Placeholders and Special Characters:

When entering commands, it's essential to handle placeholders ({ placeholder }) and special characters correctly:

Placeholders: Ensure placeholders within the text are enclosed in quotes to prevent the shell from interpreting them as commands or variables.
Special Characters: Use single quotes ' around texts containing special characters or when placeholders are used. If your text includes single quotes, escape them using '\''.

Example with Placeholders:

python manage.py refine_translation alternative "Please update your profile here: { $profileUrl }" 'Prosimo, posodobite svoj profil tukaj: { $profileUrl }' "Slovenian"

Next Steps:

The current backend implementation is the first phase of the LLM-Assisted Translations project. Future developments will focus on integrating this backend with the front-end interface, enabling input and output handling for translations across various locales supported by the feature.

flodolo · 2024-02-20T10:55:56Z

pontoon/base/management/commands/refine_translation.py

+
+
+class Command(BaseCommand):
+    help = "Refines machine translations using OpenAI with specified formality"


It's hard to fit "alternative" into a formality concept. Maybe just "specified characteristics"?

Yes, that makes more sense.

flodolo · 2024-02-20T10:56:16Z

pontoon/base/management/commands/refine_translation.py

+        )
+        parser.add_argument("english_text", type=str, help="The original English text")
+        parser.add_argument(
+            "slovenian_text",


We want to support more than Slovenian, this code should probably start locale agnostic?

flodolo · 2024-02-20T11:00:11Z

pontoon/gpt/openai_connection.py

+        # Define system messages for different translation tasks
+        system_messages = {
+            "informal": """You will be provided with a passage in English, along with its machine-generated translation in Slovenian. Your objective is to revise the Slovenian translation to ensure it utilizes simpler language. Please adhere to the following guidelines to achieve this:
+                    - Simplify Vocabulary: Use common, everyday words that are easily understood by a broad audience, avoiding technical jargon, idiomatic expressions, and complex terms.


This feels too much? Why should it avoid idiomatic expressions?

I was trying to make the text more accessible to a broader audience who may not understand specific idiomatic expressions because of the cultural nuances associated with them. But I see your point, the translation may be oversimplified.

flodolo · 2024-02-20T11:00:47Z

pontoon/gpt/openai_connection.py

+                    - Shorten and Simplify Sentences: Break down long, complex sentences into shorter, more manageable ones. Aim for clarity and conciseness in each sentence.
+                    - Use Basic Grammar Structures: Avoid complex grammatical constructions. Stick to simple tenses and straightforward sentence structures.
+                    - Clarity is Key: Ensure the translation conveys the original message in the clearest possible manner, without ambiguity or unnecessary complexity.
+                    - Engage a Wide Audience: The simplified translation should be accessible and easily understandable to people with varying levels of Slovenian proficiency, including young readers and non-native speakers.


Similarly: do we really want that simple of a language?

flodolo · 2024-02-20T11:05:07Z

pontoon/gpt/openai_connection.py

+                    - Consistent Simplicity: Maintain a consistent level of simplicity throughout the translation. The aim is to make the text as accessible as possible without sacrificing accuracy or meaning.
+                    The goal is to produce a translation that accurately reflects the original English text, but in a way that is more approachable and easier to understand for all Slovenian speakers, regardless of their language proficiency.""",
+            "formal": """You will be presented with text in English, accompanied by its machine-generated translation in Slovenian. Your task is to refine the Slovenian translation to ensure it adheres to a higher level of formality. In doing so, consider the following guidelines:
+                    - Upgrade Language Use: Elevate the language by selecting more sophisticated vocabulary and phrases that are appropriate for formal contexts.


I'd be tempted to drop this. One thing is to ask for formal structure ("Sie" vs "Du" in German), but this is going one step further, asking for super-formal vocabulary.

I agree. Based on the feedback received from the prompt evaluation I think this guideline may have caused the output to be extremely formal.

mathjazz

Nice work with the patch and thorough instructions in the PR!

Please de-hard-code Slovenian from openai_connection.py. For English it is OK to stay hard-coded.

Please also rebase against main in order to resolve conflict in requirements.

mathjazz · 2024-02-22T18:36:41Z

We just landed the python upgrade, so please rebase again if you already did.

Note that typing-extensions is no longer a required black dependency in Python 3.10+, which should solve the problem of installing openai. You can remove the black upgrade in the lint.in file and run make requirements again.

mathjazz

Nice work!

Please make sure you validate the target language.

I also left some notes for stylistic improvements, but nothing major.

mathjazz · 2024-02-26T12:33:02Z

pontoon/base/management/commands/refine_translation.py

+
+
+class Command(BaseCommand):
+    help = "Refines machine translations using OpenAI with specified characteristics"


Suggested change

help = "Refines machine translations using OpenAI with specified characteristics"

help = "Refines machine translations using OpenAI ChatGPT with specified characteristics"

mathjazz · 2024-02-26T12:36:08Z

pontoon/base/management/commands/refine_translation.py

+            "target_text", type=str, help="The machine-generated translation to refine"
+        )
+        parser.add_argument(
+            "locale_name",


We're actually using language, not locale, so I'd replace all locale* variables with language*.

mathjazz · 2024-02-26T12:41:43Z

pontoon/gpt/openai_connection.py

+        self.client = OpenAI()
+
+    def get_translation(
+        self, english_text, translated_text, characteristic, target_locale_name


We should raise a ValueErorr if target_locale_name (should be target_language_name) doesn't exist in the pontoon.base.models.Locale table.

mathjazz · 2024-02-26T12:42:46Z

pontoon/gpt/openai_connection.py

+    def get_translation(
+        self, english_text, translated_text, characteristic, target_locale_name
+    ):
+        system_messages = {


I find this hard to read. Please declare each of the three characteristics as variables and then use them like this:

system_messages = { "informal": informal, "formal": formal, "alternative": alternative, }

mathjazz · 2024-02-26T12:45:03Z

pontoon/gpt/openai_connection.py

+                    - Clarity is Key: Ensure the translation conveys the original message in the clearest possible manner, without ambiguity or unnecessary complexity.
+                    - Consistent Simplicity: Maintain a consistent level of simplicity throughout the translation.
+                    The goal is to produce a translation that accurately reflects the original English text, but in a way that is more approachable and easier to understand for all {target_locale_name} speakers, regardless of their language proficiency.""",
+            "formal": f"""You will be presented with text in English, accompanied by its machine-generated translation in {target_locale_name}. Your task is to refine the {target_locale_name} translation to ensure it adheres to a higher level of formality. In doing so, consider the following guidelines:


Why is this part of the message rephrased from the one used for informal in line 12? Can't we use the same message and only modifiy the vital part - "utilizes simpler language" vs "adheres to a higher level of formality".

Are you suggesting to remove the specific guidelines in the formal prompt? Like the "Adjust the tone" and "Formal addressing"?

No, I'm only referring to the differences between lines 12 and 16:

provided with vs presented with

along with vs accompanied by

Your objective vs Your task

to revise the translation vs to refine the {target_locale_name} translation

Please adhere to the following guidelines to achieve this vs In doing so, consider the following guidelines

mathjazz · 2024-02-26T12:48:49Z

pontoon/gpt/openai_connection.py

+        # Extract the content attribute from the response
+        translation = (
+            response.choices[0].message.content
+            if response.choices[0].message.content
+            else ""
+        )
+        return translation.strip()


Suggested change

# Extract the content attribute from the response

translation = (

response.choices[0].message.content

if response.choices[0].message.content

else ""

)

return translation.strip()

# Extract the content attribute from the response

translation = response.choices[0].message.content or ""

return translation.strip()

Actually, can we just use this?

return response.choices[0].message.content.strip()

In other words, is it possible that response.choices[0].message.content is not a string (e.g. None)?

Yes this works as well

mathjazz

Thanks for the update. Almost there!

mathjazz · 2024-02-26T19:32:39Z

pontoon/machinery/management/commands/refine_translation.py

@@ -0,0 +1,37 @@
+from django.core.management.base import BaseCommand
+from pontoon.machinery.gpt.openai_connection import OpenAIService


I would drop the gpt folder and move the file a level up. I would also rename the file to openai_service.py to match the class name (OpenAIService).

mathjazz · 2024-02-26T19:36:06Z

pontoon/machinery/gpt/openai_connection.py

+
+        formal = (
+            f"You will be provided with text in English, along with its machine-generated translation in {target_language}. "
+            "Your objective is to revise the {target_language} translation to ensure it adheres to a higher level of formality. Adhere to the following guidelines to achieve this:\n"


The verb adhere is now used twice. Maybe use "utilize" in the first case?

mathjazz · 2024-02-26T19:37:13Z

pontoon/machinery/gpt/openai_connection.py

+        )
+
+        alternative = (
+            f"You will provided text in English along with its machine-generated translation in {target_language}. "


"You will be provided with text in English, along with its machine-generated translation in {target_language}. "

mathjazz · 2024-02-26T19:37:34Z

pontoon/machinery/gpt/openai_connection.py

+
+        alternative = (
+            f"You will provided text in English along with its machine-generated translation in {target_language}. "
+            "Your objective is to provide an alternative translation, Adhere to the following guidelines to achieve this:\n"


Replace comma before "Adhere" with a period.

mathjazz

Great work!

ayanaar requested a review from mathjazz February 15, 2024 21:12

flodolo reviewed Feb 20, 2024

View reviewed changes

mathjazz reviewed Feb 21, 2024

View reviewed changes

This was referenced Feb 22, 2024

Update actions in workflows to latest versions #3111

Merged

Update workflows and docker to Python 3.10 #3112

Merged

ayanaar force-pushed the 3068-backend branch from 662cfef to 83dde9f Compare February 23, 2024 06:54

ayanaar added 2 commits February 23, 2024 09:49

First iteration of backend

8c08401

Update prompts and de-hard-code Slovenian

a9a370e

ayanaar force-pushed the 3068-backend branch from 83dde9f to 5f223e6 Compare February 23, 2024 14:52

Generate requirements

ad93cf0

ayanaar force-pushed the 3068-backend branch from 5f223e6 to ad93cf0 Compare February 23, 2024 15:02

mathjazz self-requested a review February 23, 2024 23:42

mathjazz reviewed Feb 26, 2024

View reviewed changes

Stylistic improvements

13c939e

ayanaar requested a review from mathjazz February 26, 2024 19:13

mathjazz reviewed Feb 26, 2024

View reviewed changes

Minor edits

45b3abf

ayanaar requested a review from mathjazz February 26, 2024 19:50

mathjazz approved these changes Feb 26, 2024

View reviewed changes

mathjazz merged commit 3e03b17 into mozilla:main Feb 26, 2024
4 checks passed

mathjazz mentioned this pull request Feb 26, 2024

Integrate GPT-4 APIs in Pontoon, implement back-end logic to support features described in the specification #3068

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of LLM-Assisted Translations Backend #3100

Implementation of LLM-Assisted Translations Backend #3100

ayanaar commented Feb 15, 2024 •

edited

Loading

flodolo Feb 20, 2024

ayanaar Feb 20, 2024

flodolo Feb 20, 2024

flodolo Feb 20, 2024

ayanaar Feb 20, 2024

flodolo Feb 20, 2024

flodolo Feb 20, 2024

ayanaar Feb 20, 2024

mathjazz left a comment

mathjazz commented Feb 22, 2024 •

edited

Loading

mathjazz left a comment

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

ayanaar Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

ayanaar Feb 26, 2024

mathjazz left a comment

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz Feb 26, 2024

mathjazz left a comment



		class Command(BaseCommand):
		help = "Refines machine translations using OpenAI with specified formality"

		@@ -0,0 +1,37 @@
		from django.core.management.base import BaseCommand
		from pontoon.machinery.gpt.openai_connection import OpenAIService

Implementation of LLM-Assisted Translations Backend #3100

Implementation of LLM-Assisted Translations Backend #3100

Conversation

ayanaar commented Feb 15, 2024 • edited Loading

Usage Instructions:

Examples:

For an Alternative Translation:

For an Informal Translation:

For an Formal Translation:

Handling Placeholders and Special Characters:

Next Steps:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathjazz left a comment

Choose a reason for hiding this comment

mathjazz commented Feb 22, 2024 • edited Loading

mathjazz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathjazz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathjazz left a comment

Choose a reason for hiding this comment

ayanaar commented Feb 15, 2024 •

edited

Loading

mathjazz commented Feb 22, 2024 •

edited

Loading