chime logo
The 6th CHiME Speech Separation and Recognition Challenge

Track 2 / Submission

Instructions for submitting your Track 2 system

Prepare your ASR and diarization results for the dev and eval sets

  • Prepare the RTTM and Kaldi transcription files. If you are using Kaldi, they will be obtained by using local/diarize.sh and local/decode_diarized.sh, respectively.
  • For participants who don’t use Kaldi, please also strictly follow the Kaldi format transcriptions for the submission. We only accept the Kaldi format transcriptions as follows: 
    $ head exp/chain_train_worn_simu_u400k_cleaned_rvb/tdnn1b_sp/decode_dev_beamformit_dereverb_diarized_2stage/scoring_kaldi/penalty_0.5/11.txt
S02_U06.ENH-1-008054-008144 um
S02_U06.ENH-1-008916-009050 to the other previous
S02_U06.ENH-1-009064-009326 so it was like
S02_U06.ENH-1-009838-010085 i think i g- is only to go right
S02_U06.ENH-1-010123-010337
S02_U06.ENH-1-011824-012011 and if not she knows for everyone
S02_U06.ENH-1-012580-013280 you can be done sometime previous between and a small bowl
S02_U06.ENH-1-013285-013397 they probably
S02_U06.ENH-1-013576-013717 i don't actually
S02_U06.ENH-1-013957-014063 i dunno
$ head exp/chain_train_worn_simu_u400k_cleaned_rvb/tdnn1b_sp/decode_eval_beamformit_dereverb_diarized_2stage/scoring_kaldi/penalty_0.5/11.txt
S01_U06.ENH-1-002923-003095 do we have
S01_U06.ENH-1-003657-003732 you want
S01_U06.ENH-1-006512-006755 yeah the wok moline here
S01_U06.ENH-1-008854-008969
S01_U06.ENH-1-012895-012970 or just
S01_U06.ENH-1-013195-013495 yeah i put the sauce in a while cuz it's
S01_U06.ENH-1-014232-014307
S01_U06.ENH-1-024428-024503 yeah
S01_U06.ENH-1-025851-025926
S01_U06.ENH-1-026997-027072
  • Please make sure to keep the utterance ID format as <session ID>_<array ID etc.>-<speaker ID>-<start time>-<end-time>. You can change <array ID etc.>, <start time>, and <end-time> with your own IDs or times, but do not change <session ID> and <speaker ID>.
  • We only accept the RTTM files as follows: 
    $ head exp/dev_beamformit_dereverb_stats_seg_diarization/rttm
SPEAKER S02_U06.ENH 1   2.170   1.125 <NA> <NA> 1 <NA> <NA>
SPEAKER S02_U06.ENH 1   3.295   1.285 <NA> <NA> 4 <NA> <NA>
SPEAKER S02_U06.ENH 1   4.600   1.125 <NA> <NA> 4 <NA> <NA>
SPEAKER S02_U06.ENH 1   5.725   0.445 <NA> <NA> 2 <NA> <NA>
SPEAKER S02_U06.ENH 1   6.460   4.420 <NA> <NA> 4 <NA> <NA>
SPEAKER S02_U06.ENH 1  11.560   6.010 <NA> <NA> 4 <NA> <NA>
SPEAKER S02_U06.ENH 1  18.700   3.760 <NA> <NA> 3 <NA> <NA>
SPEAKER S02_U06.ENH 1  23.590   4.600 <NA> <NA> 3 <NA> <NA>
SPEAKER S02_U06.ENH 1  28.360   2.620 <NA> <NA> 3 <NA> <NA>
SPEAKER S02_U06.ENH 1  31.030   3.880 <NA> <NA> 3 <NA> <NA>
$ head exp/eval_beamformit_dereverb_stats_seg_diarization/rttm
SPEAKER S01_U06.ENH 1   0.000   3.375 <NA> <NA> 4 <NA> <NA>
SPEAKER S01_U06.ENH 1   3.375   7.590 <NA> <NA> 2 <NA> <NA>
SPEAKER S01_U06.ENH 1  10.965   2.250 <NA> <NA> 4 <NA> <NA>
SPEAKER S01_U06.ENH 1  13.215   0.750 <NA> <NA> 3 <NA> <NA>
SPEAKER S01_U06.ENH 1  13.965   3.000 <NA> <NA> 2 <NA> <NA>
SPEAKER S01_U06.ENH 1  16.965   0.815 <NA> <NA> 4 <NA> <NA>
SPEAKER S01_U06.ENH 1  17.920   1.480 <NA> <NA> 2 <NA> <NA>
SPEAKER S01_U06.ENH 1  19.480   5.170 <NA> <NA> 2 <NA> <NA>
SPEAKER S01_U06.ENH 1  24.850   2.740 <NA> <NA> 2 <NA> <NA>
SPEAKER S01_U06.ENH 1  27.590   0.900 <NA> <NA> 1 <NA> <NA>
  • Make sure to use the same format. The start time (4th column), duration (5th), and speaker ID (8th) must remain in the same columns. Also, please keep the session ID (e.g., S01, S21) in the 2nd column..
  • Provide the single best result tuned with the development set. For example, see local/score_for_submit.sh about how to tune the language model weight, insertion penalty, and array.
  • Submit the RTTM file used in your ASR evaluation above. Our intention is to have an analysis of the relationship between RTTM and ASR results, and we would not offiicaly rank the result only with DER.

Submit your results via the following Google forms

  • Track 2 Category A

  • Track 2 Category B

    • Please first submit a challenge abstract paper by following https://chimechallenge.github.io/chime2020-workshop/submissions.html and get the CMT paper ID, which is required to finish the form.
    • This form requires a Google account. Please contact us chimechallenge@gmail.com if you have difficulties on access to the form due to the account issue.
    • In the form, we ask you to fill out the team profile, brief technical descriptions and so on for our challenge analysis. It can be updated anytime before the challenge submission deadline (11:59pm (Pacific time), April 10th).
    • If you want to only submit the diarization results, please contact us chimechallenge@gmail.com in advance.
CHiME-6 Challenge
chimechallenge@gmail.com