Repositories list

• HPLT-WP4

  Public
  Information and pipelines on WP4: language models training

  Python

  •

  Creative Commons Zero v1.0 Universal

  •3•2•0•0•Updated Jan 26, 2025Jan 26, 2025

• ia-download

  Public
  Internet archive downloader

  Jupyter Notebook

  •0•2•1•0•Updated Jan 25, 2025Jan 25, 2025

• data-analytics-tool

  Public
  Data Analytics Tool

  Python

  •1•10•1•1•Updated Jan 23, 2025Jan 23, 2025

• warc2text-runner

  Public
  Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

  HTML

  •0•3•5•0•Updated Jan 23, 2025Jan 23, 2025

• release2_inspection

  Public
  Jupyter Notebook

  •7•1•1•0•Updated Jan 22, 2025Jan 22, 2025

• bitextor-slurm

  Public
  Scripts for running bitextor jobs

  Shell

  •1•0•1•0•Updated Jan 20, 2025Jan 20, 2025

• OpusPocus

  Public
  Marian machine translation training pipeline for thousands of models

  Python

  •0•2•19•1•Updated Jan 17, 2025Jan 17, 2025

• OpusCleaner

  Public
  OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

  machine-translationdata-cleaning

  Python

  •14•50•56•1•Updated Jan 16, 2025Jan 16, 2025

• OpusTrainer

  Public
  Curriculum training

  machine-translationtraining-pipeline

  Python

  •

  MIT License

  •6•16•19•0•Updated Jan 13, 2025Jan 13, 2025

• cc-download

  Public
  Shell

  •0•0•0•0•Updated Dec 30, 2024Dec 30, 2024

• bitextor-mt-models

  Public
  Shell

  •0•1•3•0•Updated Dec 19, 2024Dec 19, 2024

• monotextor-slurm

  Public
  Set of scripts to run monotextor-like pipeline under slurm HPCs

  Rust

  •

  GNU General Public License v3.0

  •0•2•0•0•Updated Nov 4, 2024Nov 4, 2024

• monolingual-multilingual-instruction-tuning

  Public
  Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

  large-language-modelsinstruction-tuninginstruction-data

  Python

  •1•9•0•0•Updated Nov 2, 2024Nov 2, 2024

• sacremoses

  Public
  Python port of Moses tokenizer, truecaser and normalizer

  nlptokenizermachine-translation

  Python

  •

  MIT License

  •59•489•27•4•Updated May 26, 2024May 26, 2024

• document-aligner

  Public
  tf/idf-based document aligner from Bitextor

  C++

  •

  Apache License 2.0

  •0•0•0•1•Updated Mar 19, 2024Mar 19, 2024

• OPUS-MT-dashboard

  Public
  PHP

  •

  MIT License

  •1•0•0•0•Updated Mar 9, 2024Mar 9, 2024

• HPLT-MT-Models

  Public
  This contains the configuration and scripts for HPLT MT model releases.

  Python

  •0•4•1•0•Updated Mar 6, 2024Mar 6, 2024

• OpusFilter

  Public
  OpusFilter - Parallel corpus processing toolkit

  Python

  •

  MIT License

  •20•0•0•0•Updated Jan 3, 2024Jan 3, 2024

• clianer

  Public
  A lightweight command-line frontend to OpusCleaner

  Python

  •

  MIT License

  •1•0•0•0•Updated Nov 27, 2023Nov 27, 2023

• paracrawl-dashboard

  Public
  Make-shift interface for managing Paracrawl processing and exploring its outputs

  HTML

  •1•0•0•0•Updated Oct 10, 2023Oct 10, 2023

• MT-winterschool-2023

  Public
  0•1•0•0•Updated Feb 7, 2023Feb 7, 2023