Translation example does not work

[ShankarRIntel]

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

Gaudi2

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

1.1

Description

Translation example does not work after all the servers are up after docker compose

Reproduce steps

sratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ docker compose up -d
[+] Running 6/6
✔ Network gaudi_default Created 0.2s
✔ Container tgi-gaudi-server Started 1.4s
✔ Container llm-tgi-gaudi-server Started 1.9s
✔ Container translation-gaudi-backend-server Started 2.3s
✔ Container translation-gaudi-ui-server Started 2.7s
✔ Container translation-gaudi-nginx-server Started 3.2s
sratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:8008/generate
-X POST
-d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":64, "do_sample": true}}'
-H 'Content-Type: application/json'
curl: (7) Failed to connect to 192.168.1.11 port 8008 after 0 ms: Connection refused
sratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:9000/v1/chat/completions
-X POST
-d '{"query":"Translate this from Chinese to English:\nChinese: 我爱机器翻译。\nEnglish:"}'
-H 'Content-Type: application/json'
Internal Server Errorsratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compcurl http://${host_ip}:8888/v1/translation -H "Content-Type: application/json" -d '{plication/json" -d '{
"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
sratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:${NGINX_PORT}/v1/translation
-H "Content-Type: application/json"
-d '{"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
sratnesh@fm2r81s1gaudi:/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$

Raw log

sratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ docker compose up -d
[+] Running 6/6
 ✔ Network gaudi_default                       Created                                                                                                                                                              0.2s
 ✔ Container tgi-gaudi-server                  Started                                                                                                                                                              1.4s
 ✔ Container llm-tgi-gaudi-server              Started                                                                                                                                                              1.9s
 ✔ Container translation-gaudi-backend-server  Started                                                                                                                                                              2.3s
 ✔ Container translation-gaudi-ui-server       Started                                                                                                                                                              2.7s
 ✔ Container translation-gaudi-nginx-server    Started                                                                                                                                                              3.2s
sratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:8008/generate \
  -X POST \
  -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":64, "do_sample": true}}' \
  -H 'Content-Type: application/json'
curl: (7) Failed to connect to 192.168.1.11 port 8008 after 0 ms: Connection refused
sratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:9000/v1/chat/completions \
  -X POST \
  -d '{"query":"Translate this from Chinese to English:\nChinese: 我爱机器翻译。\nEnglish:"}' \
  -H 'Content-Type: application/json'
Internal Server Errorsratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compcurl http://${host_ip}:8888/v1/translation -H "Content-Type: application/json" -d '{plication/json" -d '{
     "language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
sratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$ curl http://${host_ip}:${NGINX_PORT}/v1/translation \
    -H "Content-Type: application/json" \
    -d '{"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
sratnesh@fm2r81s1gaudi:~/workspace/GenAIExamples/Translation/docker_compose/intel/hpu/gaudi$

[yinghu5]

Hi ShankarRIntel,

Thank you a lot for reporting those issue! We will investigate them.

As we are refactoring the code, maybe bring some unknow issues. From the error message:
ailed to connect to 192.168.1.11 port 8008 after 0 ms: Connection refused and
Internal Server Errors
It seems the docker image or the network are broken

Could you please help to get some check information by command

1. docker ps -a

2. docker logs tgi-gaudi-server -t

To download the llm model may need some times as
https://opea-project.github.io/latest/GenAIExamples/ChatQnA/docker_compose/intel/hpu/gaudi/how_to_validate_service.html mentioned:

and if possible, check OPEA version:
3. git status
environment， are you using Intel IDC or private cluster? please try
4. hl-smi
5. hostname -i

Thanks

[yinghu5]

Hi @ShankarRIntel ,

Any updates about your machine configuration (like networks for LLM downloads, Gaudi driver etc) and the examples?

For your reference, in order to show how the OPEA example works, we release some videos on the AI Software Catalog: https://aiswcatalog.intel.com/explore:
• Code Generation: Intel® AI SW Catalog
• Audio Q&A: Intel® AI SW Catalog
• Visual Q&A: Intel® AI SW Catalog
• Code Translation: Intel® AI SW Catalog
• Content Summarization: Intel® AI SW Catalog
• Chat Q&A: https://aiswcatalog.intel.com/solution/aimodel-18c8e8c2-31d1-4af4-b0a2-ab8613a3c3b9
And FAQ Gen and Multi-Lingual QnA will available soon

thanks

[ShankarRIntel]

LLM downloads seem to be fine, Gaudi driver is v 1.19. I need all the below examples to be working on Gaudi. Thanks Shankar From: Ying Hu ***@***.***> Sent: Wednesday, January 8, 2025 5:10 PM To: opea-project/GenAIExamples ***@***.***> Cc: Ratneshwaran, Shankar ***@***.***>; Mention ***@***.***> Subject: Re: [opea-project/GenAIExamples] Translation example does not work (Issue #1346) Hi @ShankarRIntel<https://github.com/ShankarRIntel> , Any updates about your machine configuration (like networks for LLM downloads, Gaudi driver etc) and the examples? For your reference, in order to show how the OPEA example works, we release some videos on the AI Software Catalog: https://aiswcatalog.intel.com/explore: • Code Generation: Intel® AI SW Catalog<https://aiswcatalog.intel.com/solution/aimodel-ed3df895-69e0-45fc-8c13-caf2c85f8e11> • Audio Q&A: Intel® AI SW Catalog<https://aiswcatalog.intel.com/solution/aimodel-2777c70c-491f-4466-866f-4265d345a985> • Visual Q&A: Intel® AI SW Catalog<https://aiswcatalog.intel.com/solution/aimodel-1453a049-fcf7-40e4-8fa1-7e609d31a541> • Code Translation: Intel® AI SW Catalog<https://aiswcatalog.intel.com/solution/aimodel-ed3df895-69e0-45fc-8c13-caf2c85f8e11> • Content Summarization: Intel® AI SW Catalog<https://aiswcatalog.intel.com/solution/aimodel-3b3f72e8-e27e-4574-b509-515d6b63ed8e> • Chat Q&A: https://aiswcatalog.intel.com/solution/aimodel-18c8e8c2-31d1-4af4-b0a2-ab8613a3c3b9 And FAQ Gen and Multi-Lingual QnA will available soon thanks — Reply to this email directly, view it on GitHub<#1346 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYMDOKYXERKZ22OIURHBNET2JXD6VAVCNFSM6AAAAABUSYA4OOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZYHE4TKMBXGU>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[yinghu5]

Hi ShankarRIntel ,

I will contact you by team and let's try them. thank you!

[xiguiw]

@ShankarRIntel

I built docker image from source code, I run Translation on Gaud2 1.19 successfully.
I cannot reproduce your issue at my side.

Did you pull docker image 1.1 or build docker images from source code?
Would you please provide details instructions to reproduce this issue?
For example, The document link you refer to, your code commit ID (GenAIExampls and GenAIComp), steps to setup environment, and steps to set environment variables, http_proxy used in your network etc.

Please kindly provide the logs. Thanks!

From you logs, the docker start in seconds.

✔ Network gaudi_default Created 0.2s
✔ Container tgi-gaudi-server Started 1.4s
✔ Container llm-tgi-gaudi-server Started 1.9s
✔ Container translation-gaudi-backend-server Started 2.3s
✔ Container translation-gaudi-ui-server Started 2.7s
✔ Container translation-gaudi-nginx-server Started 3.2s

"curl: (7) Failed to connect to 192.168.1.11 port 8008 after 0 ms: Connection refused"

1. The server is not started when you send the first curl command.
2. Next it shows "internal server error"
   The LLM model cannot be ready in seconds as it's a 13B model and torch model conversion to safetensors.

Her are my test environment:

+-----------------------------------------------------------------------------+
| HL-SMI Version:                              hl-1.19.0-fw-57.1.0.0          |
| Driver Version:                                     1.19.0-2427ed8          |
|-------------------------------+----------------------+----------------------+
| AIP  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncor-Events|
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | AIP-Util  Compute M. |
|===============================+======================+======================|
|   0  HL-225              N/A  | 0000:33:00.0     N/A |                 141  |
| N/A   26C   N/A  71W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   1  HL-225              N/A  | 0000:9a:00.0     N/A |                  18  |
| N/A   27C   N/A  90W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
...|-------------------------------+----------------------+----------------------+
|   7  HL-225              N/A  | 0000:b4:00.0     N/A |                   2  |
| N/A   26C   N/A  80W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
| Compute Processes:                                               AIP Memory |
|  AIP       PID   Type   Process name                             Usage      |
|=============================================================================|
|   0        N/A   N/A    N/A                                      N/A        |
|   1        N/A   N/A    N/A                                      N/A        |
...
|   7        N/A   N/A    N/A                                      N/A        |
+=============================================================================+