COMET: High-quality Machine Translation Evaluation

What is COMET

COMET is an open-source framework for MT evaluation that can be used for two purposes:

• To evaluate MT systems with our currently available high-performing metrics (check: COMET Metrics).

• To train and develop new metrics.

Contents:

• Installation
• Running COMET
  • Command Line Interface
• Frequently Asked Questions
  • Is there a theoretical range of values for the COMET regressor?
  • Which COMET model should I use?
  • Where can I find the data used to train COMET models?
• COMET Metrics
  • Model Architectures:
  • Available Metrics:
  • WMT20 COMET Metrics
  • WMT21 COMET Metrics
  • Benchmark
• Train your own Metric
  • Config Files
  • Input Data
  • Available Encoders

License

Free software: Apache License 2.0

Contributing

We welcome contributions to improve COMET. Please refer to CONTRIBUTING.md for quick instructions or to contributing instructions for more detailed instructions on how to set up your development environment.

Publications

• COMET: A Neural Framework for MT Evaluation

• Unbabel’s Participation in the WMT20 Metrics Shared Task

• COMET - Deploying a New State-of-the-art MT Evaluation Metric in Production

• Uncertainty-Aware Machine Translation Evaluation

• Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task