SpeechColab ASR leaderboard

result result

1. Overview

"If you can’t measure it, you can’t improve it." -- Peter Drucker

SpeechIO leaderboard serves as an ASR benchmarking platform by providing 3 components:

TestSet Zoo: A collection of test sets covering wide range of speech recognition tasks & scenarios
Model Zoo: A collection of models including commercial APIs & open-sourced models
Benchmarking Pipeline: a simple & well-specified pipeline to take care of data preparation / recognition / post processing / error rate evaluation.

People should be able to easily benchmark, reproduce, examine ASR systems from each other

Overview

2. TestSet Zoo: `datasets/*`

<details><summary> Academic Test Sets (EN & ZH) </summary>

已公开 <br> UNLOCKED	编号 <br> DATASET_ID	说明 <br> DESCRIPTION	语言 <br> LANGUAGE
✓	AISHELL1_TEST	test set of AISHELL-1	zh
✓	AISHELL2_IOS_TEST	test set of AISHELL-2 (iOS channel)	zh
✓	AISHELL2_ANDROID_TEST	test set of AISHELL-2 (Android channel)	zh
✓	AISHELL2_MIC_TEST	test set of AISHELL-2 (Microphone channel)	zh
✓	ALIMEETING_EVAL_NEAR_FIELD	AliMeeting	zh
✓	ALIMEETING_TEST_NEAR_FIELD	AliMeeting	zh
✓	ALIMEETING_EVAL_FAR_FIELD	AliMeeting	zh
✓	ALIMEETING_TEST_FAR_FIELD	AliMeeting	zh
✓	LIBRISPEECH_TEST_CLEAN	"test_clean" set of LibriSpeech	en
✓	LIBRISPEECH_TEST_OTHER	"test_other" set of LibriSpeech	en
✓	TEDLIUM_RELEASE3_LEGACY_DEV	tedlium release 3, legacy dir dev set TEDLium3	en
✓	TEDLIUM_RELEASE3_LEGACY_TEST	tedlium release 3, legacy dir test set TEDLium3	en
✓	GIGASPEECH_V1.0.0_DEV	dev set of GigaSpeech	en
✓	GIGASPEECH_V1.0.0_TEST	test set of GigaSpeech	en
✓	VOXPOPULI_V1.0_EN_DEV	dev set of VoxPopuli	en
✓	VOXPOPULI_V1.0_EN_TEST	test set of VoxPopuli	en
✓	VOXPOPULI_V1.0_EN_ACCENTED_TEST	accented test set of VoxPopuli	en
✓	COMMON_VOICE_V11.0_DEV	dev set of Common Voice	en
✓	COMMON_VOICE_V11.0_TEST	test set of Common Voice	en

</details> <details><summary> SpeechIO Test Sets (ZH) </summary>

SpeechIO test sets are carefully curated by SpeechIO authors, crawled from publicly available sources (Youtube, TV programs, Podcast etc), covering various well-known scenarios and topics, transcribed by payed professional annotators.

已公开 <br> UNLOCKED	编号 <br> DATASET_ID	名称 <br> NAME	场景 <br> SCENARIO	内容领域 <br> TOPIC	有效时长 <br> DURATION (HOURS)	难度(1-5) <br> DIFFICULTY
✓	SPEECHIO_ASR_ZH00000	调试集 <br> for debugging	视频会议、论坛演讲 <br> conference & speech	经济、货币、金融 <br> economy, currency, finance	1.0	★★☆
✓	SPEECHIO_ASR_ZH00001	新闻联播	新闻播报 <br> TV News	时政 <br> news & politics	9	★
✓	SPEECHIO_ASR_ZH00002	鲁豫有约	访谈电视节目 <br> TV interview	名人工作/生活 <br> celebrity & film & music & daily	3	★★☆
✓	SPEECHIO_ASR_ZH00003	天下足球	专题电视节目 <br> TV program	足球 <br> Sports & Football & Worldcup	2.7	★★☆
✓	SPEECHIO_ASR_ZH00004	罗振宇跨年演讲	会场演讲 <br> Stadium Public Speech	社会、人文、商业 <br> Society & Culture & Business Trend	2.7	★★
✓	SPEECHIO_ASR_ZH00005	李永乐讲堂	在线教育 <br> Online Education	科普 <br> Popular Science	4.4	★★★
✓	SPEECHIO_ASR_ZH00006	王者荣耀 <br> 张大仙 & 骚白	直播 <br> Live Broadcasting	游戏 <br> Game	1.6	★★★☆
✓	SPEECHIO_ASR_ZH00007	直播带货 <br> 李佳琪 & 薇娅	直播 <br> Live Broadcasting	电商、美妆 <br> Makeup & Online shopping/advertising	0.9	★★★★☆
✓	SPEECHIO_ASR_ZH00008	老罗语录	线下培训 <br> Offline lecture	段子、做人 <br> Life & Purpose & Ethics	1.3	★★★★☆
✓	SPEECHIO_ASR_ZH00009	故事FM	播客 <br> Podcast	人生故事、见闻 <br> Ordinary Life Story Telling	4.5	★★☆
✓	SPEECHIO_ASR_ZH00010	创业内幕	播客 <br> Podcast	创业、产品、投资 <br> Startup & Enterprenuer & Product & Investment	4.2	★★☆
✓	SPEECHIO_ASR_ZH00011	罗翔刑法法考	在线教育 <br> Online Education	法律法考 <br> Law & Lawyer Qualification Exams	3.4	★★☆
✓	SPEECHIO_ASR_ZH00012	张雪峰考研	在线教育 <br> Online Education	考研高校报考 <br> University & Graduate School Entrance Exams	3.4	★★★☆
✓	SPEECHIO_ASR_ZH00013	谷阿莫 <br> 牛叔说电影	短视频 <br> VLog	电影剪辑 <br> Movie Cuts	1.8	★★★
✓	SPEECHIO_ASR_ZH00014	贫穷料理 <br> 琼斯爱生活	短视频 <br> VLog	美食、烹饪 <br> Food & Cooking & Gourmet	1	★★★☆
✓	SPEECHIO_ASR_ZH00015	单田芳白眉大侠	评书 <br> Traditional Podcast	江湖、武侠 <br> Kongfu Fiction	2.2	★★☆
✓	SPEECHIO_ASR_ZH00016	德云社演出	剧场相声 <br> Theater Crosstalk Show	包袱段子 <br> Funny Stories	1	★★★
✓	SPEECHIO_ASR_ZH00017	吐槽大会	脱口秀电视节目 <br> Standup Comedy	明星糗事 <br> Celebrity Jokes	1.8	★★☆
✓	SPEECHIO_ASR_ZH00018	小猪佩奇 <br> 熊出没	少儿动画 <br> Children Cartoon	童话故事、日常 <br> Fairy Tale	0.9	★☆
✓	SPEECHIO_ASR_ZH00019	CCTV5 NBA 转播	体育赛事解说 <br> Sports Game Live	篮球、NBA <br> NBA Game	0.7	★★★
✓	SPEECHIO_ASR_ZH00020	篮球人物	纪录片 <br> Documentary	篮球明星、成长 <br> NBA Super Stars' Life & History	2.2	★★
✓	SPEECHIO_ASR_ZH00021	汽车之家评测	短视频 <br> VLog	汽车测评 <br> Car benchmarks, Road driving test	1.7	★★★☆
✓	SPEECHIO_ASR_ZH00022	小艾大叔豪宅带看	短视频 <br> VLog	房地产、豪宅 <br> Realestate, Mansion tour	1.7	★★★
✓	SPEECHIO_ASR_ZH00023	无聊开箱 <br> Zealer评测	短视频 <br> VLog	产品开箱评测 <br> Unboxing	2	★★★
✓	SPEECHIO_ASR_ZH00024	付老师种植技术	短视频 <br> VLog	农业、种植 <br> Agriculture, Planting	2.7	★★★☆
✓	SPEECHIO_ASR_ZH00025	石国鹏讲历史	线下培训 <br> Offline lecture	历史，古希腊哲学 <br> History, Greek philosophy	1.3	★★☆
✓	SPEECHIO_ASR_ZH00026	张震鬼故事	广播节目 <br> Broadcasting Program	鬼故事 <br> Horror Stories	2.4	★★★
✗	SPEECHIO_ASR_ZH00027	华语辩论世界杯	辩论赛 <br> Debates Contest	兴趣、技能、成长 <br> Hobby, Skill, Growth	1.4	★★★
✗	SPEECHIO_ASR_ZH00028	时政现场同传	同声传译 <br> Simultaneous Translation	时政、社会公共治理 <br> News & Events on Public Governance	2.1	★★★☆
✗	SPEECHIO_ASR_ZH00029	港台明星访谈 <br> 周杰伦,曾志伟 <br> 张家辉,陈小春 <br> 周星驰	口音(港台) <br> HongKong/Taiwan Accents	娱乐、生活、演艺 <br> Entertainment, Acting, Musics	1.5	★★★☆
✗	SPEECHIO_ASR_ZH00030	世界青年说	口音(老外) <br> Foreigner Accents	异国文化比较 <br> Cultural Difference	2	★★★☆
✗	SPEECHIO_ASR_ZH00031	东方甄选	直播 <br> broadcast	带货，英语教学 <br> Online advertising & English Education	2.4	★★★☆
✗	SPEECHIO_ASR_ZH00032	郎朗钢琴课	长视频 <br> long-form video	音乐乐理，钢琴 <br> Music & piano	1.7	★★☆
✗	SPEECHIO_ASR_ZH00033	老石谈芯	短视频 <br> VLog	芯片 <br> chips	2.8	★★★
✗	SPEECHIO_ASR_ZH00034	电丸科技AK	短视频 <br> VLog	网络 IT <br> Internet tech, IT	1.4	★★★☆
✗	SPEECHIO_ASR_ZH00035	新氧医美	短视频 <br> VLog	医疗美容 <br> Medical Cosmetology	1.4	★★
✗	SPEECHIO_ASR_ZH00036	交通广播	交通广播 <br> traffic radio	路况，娱乐 <br> Traffics	1.2	★★★☆
✗	SPEECHIO_ASR_ZH00037	老俞闲聊	在线会议 <br> Online meeting	闲聊 <br> chat	2.4	★★★
✗	SPEECHIO_ASR_ZH00038	电影：疯狂石头+疯狂赛车	电影 <br> Film	重庆话、山东青岛、四川成都话、河北唐山话、粤语、天津话、河南话、陕西话、闽南话，武汉话等 <br> multiple accents	1.3	★★★★☆
✗	SPEECHIO_ASR_ZH00039	电影：1942	电影 <br> Film	河南话 <br> HeNan Accent	0.9	★★★★
✗	SPEECHIO_ASR_ZH00040	电影：白鹿原	电影 <br> Film	陕西话 <br> ShaanXi Accent	1.1	★★★★★
✗	SPEECHIO_ASR_ZH00041	电影：让子弹飞	电影 <br> Film	四川话 <br> SiChuan Accent	1.1	★★★★☆
✗	SPEECHIO_ASR_ZH00042	电影：人生大事	电影 <br> Film	武汉话 <br> WuHan Accent	0.8	★★★★