COMET: High-quality Machine Translation Evaluation

What is COMET

COMET is an open-source framework for MT evaluation that can be used for two purposes:

• To evaluate MT systems with our currently available high-performing metrics (check: COMET Metrics).

• To train and develop new metrics.

Contents:

• Installation
• Running COMET
  • Command Line Interface
• Frequently Asked Questions
  • Interpreting Scores:
  • Which COMET model should I use?
  • Where can I find the data used to train COMET models?
• COMET Metrics
  • Model Architectures:
• Available Evaluation Models
• Train your own Metric
  • Config Files
  • Input Data
  • Available Encoders

License

Free software: Apache License 2.0 Model Licenses can be found here

Contributing

We welcome contributions to improve COMET. Please refer to CONTRIBUTING.md for quick instructions or to contributing instructions for more detailed instructions on how to set up your development environment.

Publications

• CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

• COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task

• Searching for Cometinho: The Little Metric That Could

• Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task

• Uncertainty-Aware Machine Translation Evaluation

• COMET - Deploying a New State-of-the-art MT Evaluation Metric in Production

• Unbabel’s Participation in the WMT20 Metrics Shared Task

• COMET: A Neural Framework for MT Evaluation