KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #developer-tools #claude-code #ai-agent #codex #deepseek-harness #ai-agents #cli #react #mcp

Benchmarking_past_present_future

★ 35 Open GitHub ↗

Workshop Home Page for Benchmarking: Past, Present and Future

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<html>
  <body>
<h1>Home Page for ACL-2021 Workshop on Benchmarking: Past, Present and Future (BPPF)</h1>

Where have we been, and where are we going?  It is easier to talk about the past than the future. These days,
benchmarks evolve more bottom up (such as <a href="https://paperswithcode.com/">papers with code</a>). There used to
be more top-down leadership from government (and industry, in the  case of
systems, with benchmarks such as <a href="https://www.spec.org/benchmarks.html">SPEC</a>). Going forward, there may be more
top-down leadership from organizations like <a href="https://mlperf.org/"> MLPerf </a> and/or influencers like
<a href="https://en.wikipedia.org/wiki/David_Ferrucci"> David Ferrucci </a>, who was responsible for IBM’s success with <a href="https://www.youtube.com/watch?v=P18EdAKuC1U">Jeopardy</a>, and
has recently written a paper suggesting how the community should think
about benchmarking for machine comprehension (<a href="https://arxiv.org/pdf/2005.01525.pdf">To Test Machine Comprehension, Start by Defining Comprehension</a>). Tasks such as reading
comprehension become even more interesting as we move beyond English.
Multilinguality introduces many challenges, and even more opportunities.

We have an amazing collection of invited speakers that can share with
us first hand knowledge of how benchmarking became important in
Information Retrieval, and then in speech (starting around 1975), and
then in language (in 1988).  Much of this history is described
in <a href="https://www.simonsfoundation.org/search/liberman/">this
video</a> and two 2016 Interspeech keynotes: Makhoul describes how
benchmarking overcame resistance in speech
in <a href="https://www.superlectures.com/interspeech2016/isca-medalist-for-leadership-and-extensive-contributions-to-speech-and-language-processing">this
keynote</a>, and Jurafsky describes how this approach moved from
speech to language
in <a href="https://www.superlectures.com/interspeech2016/ketchup-interdisciplinarity-and-the-spread-of-innovation-in-speech-and-language-processing">this
keynote</a>.

<h2>Important Dates</h2>

<ul>
  <li> <del>May 5, 2021</del>: Paper submission</li>
  <li>  <del>May 28, 2021</del>: Notification of acceptance </li>
  <li>  <del>June 7, 2021</del>: Camera-ready papers due </li>
  <li>  <del>July 31, 2021 </del>: Pre-recorded talks will be posted on YouTube and bilibili.
    🆕 A talks available now!  Look for 📽️</li>
  <li>  <del>July 31, 2021</del>: Slides for pre-recorded talks will be posted in this github and perhaps elsewhere (look for 📔). 🆕 Most are available now.</li>
  <li>  <del>August 5, 2021</del>: Workshop date; <a href="https://aclanthology.org/volumes/2021.bppf-1/">Papers (hosted in ACL Anthology)</a> <a href="https://underline.io/events/167/sessions?eventSessionId=6282">zoom call (ACL registration required)</a></li>
  <li> Aug 20, 2021: Zoom discussion (search for "Zoom") </li>
  </ul>

<h2>Invited Speakers</h2>

We have an amazing collection of invited talks, many with direct first-hand knowledge of this history, and many insights for the future.

<ol> <li>Past
  <ol>
<li><a href="#Makhoul">John Makhoul</a> </li>
<li><a href="#Liberman">Mark Liberman</a></li>
<li><a href="#Voorhees">Ellen Voorhees</a></li>
<li><a href="#Mashey">John Mashey</a></li>
 </ol> </li>
<li>Present
  <ol>
<li><a href="#Ming">Nan Duan, Qi Zhang and Ming Zhou</a></li>
<li><a href="#Wu">Hua Wu and Jing Liu</a></li>
<li><a href="#Ryant">Neville Ryant</a></li>
<li><a href="#MacWhinney">Brian MacWhinney and Saturnino Luz</a></li>
<li><a href="#Kiela">Douwe Kiela</a> </li>
<li><a href="#Choi">Eunsol Choi</a></li>
<li><a href="#Sogaard">Anders Søgaard</a></li>
   </ol> </li>
<li> Future
  <ol>
<li><a href="#Diamos">Greg Diamos, Peter Mattson and David Kanter</a></li>
<li><a href="#Ferrucci">Dave Ferrucci</a></li>
<li><a href="#Dagan">Ido Dagan</a></li>
    <li><a href="#Bowman">Samuel Bowman</a></li>
  </ol> </li>
</ol>

<h2>Draft Schedule</h2>

All talks (invited and submitted) will be pre-recorded in advance, and posted by the end of July.

In addition, there will be 5 Q&A sessions via Zoom in a single
plenary session.  It is hoped that most speakers (and the
audience) will have seen the pre-recorded talks before the live
Zoom sessions, so the Zoom sessions will have more back and forth
discussion, and will not digress into individual talks.  
<p>
These times are local to New York.

<ol>
  <li> <a href="#S1">Session 1</a> (7AM): Convenient for Asia + Europe (bonus session) </li>
  <li> <a href="#S2">Session 2</a> (9AM): Convenient for America + Europe (Present 1) </li>
  <li> <a href="#S3">Session 3</a> (11AM): Convenient for America + Europe (Past) </li>
  <li> <a href="#S4">Session 4</a> (3PM): Convenient for America + Europe (Future) </li>
  <li> <a href="#S5">Session 5</a> (9PM): Convenient for America + Asia (Present 2) </li>
</ol>

<table border=1>
  <tr><th>Session</th> <th> New York </th> <th> San Francisco</th> <th> Berlin </th> <th> Beijing </th> </tr>
  <tr> <td> <a href="#S1">Session 1</a> </td> <td> 7AM Aug 5 </td>  <td> 4AM Aug 5</td> <td> 1PM Aug 5</td>  <td> 7PM Aug 5 </td>  </tr>
  <tr> <td> <a href="#S2">Session 2</a> </td> <td> 9AM Aug 5 </td>  <td> 6AM Aug 5</td> <td> 3PM Aug 5</td>  <td> 9PM Aug 5</td>  </tr>
  <tr> <td> <a href="#S3">Session 3</a> </td> <td> 11AM Aug 5 </td>  <td> 8AM Aug 5</td> <td> 5PM Aug 5</td>  <td> 11PM Aug 5</td>  </tr>
  <tr> <td> <a href="#S4">Session 4</a> </td> <td> 3PM Aug 5</td>  <td> noon Aug 5</td> <td> 9PM Aug 5</td>  <td> 3AM Aug 6</td>  </tr>
  <tr> <td> <a href="#S5">Session 5</a> </td> <td> 9PM Aug 5</td>  <td> 6PM Aug 5</td> <td> 3AM Aug 6</td>  <td> 9AM Aug 6</td>  </tr>
</table>

Speakers for the following submitted papers can choose to participate in any of the 5 zoom sessions:
<ol>
  <li id="Victor">
Victor Petrén Bach Hansen and Anders Søgaard <br>
<i>Guideline Bias in Wizard-of-Oz Dialogues<br>
<a href="https://www.youtube.com/watch?v=agPS2EibMJ0">📽️ talk (hosted on: YouTube)</a>
<a href="slides/session2/Victor_Guidelines_Bias_ACL_WS.pdf">📔 slides (hosted on: github)</a>
</i> </li>
  <li id="Valerio">Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo
    Poesio and Alexandra Uma <br><i>We Need to Consider Disagreement in Evaluation
      <br><a href="https://www.youtube.com/watch?v=FnQfyileFnw">📽️ talk (hosted on: YouTube)</a>
</i></li>
<li id="Alena">Alëna Aksënova, Daan van Esch, James Flynn and Pavel Golik <br> <i>
    How Might We Create Better Benchmarks for Speech Recognition?</i>
<br><a href="https://www.youtube.com/watch?v=k9uPwqraDLA">📽️ talk (hosted on: YouTube)</a>
<br><a href="https://www.bilibili.com/video/BV16b4y1r7rD">📽️ talk (hosted on: bilibili)</a>
</li>

</ol>
<p>
Invited speakers will be assigned to at least one regular session
(Sessions 2-5).  In addition, there will be a bonus session (Session 1).
<p>
The bonus session is intended to correct for a bias in favor of
America.  The speakers in the bonus session will attend two
sesssions, one that is convenient for timezones in America, plus
one that is convenient for timezones elsewhere.

<ol>
  <li id="S1"> Session 1 (7AM): Convenient for Asia + Europe (bonus session) </li>
  <ol>
    Moderators: Hang Li and Valia Kordoni
<br>
<a href="https://youtu.be/dAlfbwtG9u0">📽️ Zoom Discussion (part 1 of 2) (hosted on: YouTube)</a>
<a href="https://youtu.be/d_YZ8bxqtJQ">📽️ Zoom Discussion (part 2 of 2) (hosted on: YouTube)</a>
<br>

<a href="https://www.bilibili.com/video/BV1dg411c7vw">📽️ Zoom Discussion (part 1 of 2) (hosted on: bilibili)</a>
<a href="https://www.bilibili.com/video/BV1yf4y1E7x8">📽️ Zoom Discussion (part 2 of 2) (hosted on: bilibili)</a>

<br>
    <li><a href="#Ming">Nan Duan, Qi Zhang and Ming Zhou</a></li>
    <li><a href="#Wu">Hua Wu and Jing Liu</a></li>
    <li><a href="#MacWhinney">Brian MacWhinney and Saturnino Luz</a></li>
    <li><a href="#Sogaard">Anders Søgaard</a></li>
    <li><a href="#Valerio">Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio and Alexandra Uma</a></li>
    <li><a href="#Alena">Alëna Aksënova, Daan van Esch, James Flynn and Pavel Golik</a></li>
  </ol>

<li id="S2"> Session 2 (9AM): Convenient for America + Europe (Present 1)</li>
  <ol>
    Moderator: Mark Liberman <a href="slides/session2/LibermanBenchmarkingSession2.pptx">📔 slides (hosted on: github)</a>
    <br>
    <a href="https://youtu.be/d_YZ8bxqtJQ">📽️ Zoom Discussion (hosted on: YouTube)</a>
    <br>
<a href="https://www.bilibili.com/video/BV1zf4y1w7Wb">📽️ Zoom Discussion (hosted on: bilibili)</a>

<li><a href="#MacWhinney">Brian MacWhinney and Saturnino Luz</a></li>
<li><a href="#Kiela">Douwe Kiela</a> </li>
<li><a href="#Sogaard">Anders Søgaard</a></li>
<li><a href="#Victor">Victor Petrén Bach Hansen and Anders Søgaard</a></li>
  </ol>

<li id="S3"> Session 3 (11AM): Convenient for America + Europe (Past) </li>
  <ol>
    Moderator: Henry Kautz
    <br>
    <a href="https://youtu.be/egcP_qEEb1E">📽️ Zoom Discussion (hosted on: YouTube)</a>
    <br>
<a href="https://www.bilibili.com/video/BV1fQ4y1z7hY">📽️ Zoom Discussion (hosted on: bilibili)</a>
    <li><a href="#Makhoul">John Makhoul</a></li>
    <li><a href="#Liberman">Mark Liberman</a></li>
    <li><a href="#Voorhees">Ellen Voorhees</a></li>
    <li><a href="#Mashey">John Mashey</a></li>
  </ol>

<li id="S4"> Session 4 (3PM): Convenient for America + Europe (Future) </li>
<ol>
  Moderator: Ken Church
<br>
<a href="https://youtu.be/7wwcqKyzEtE">📽️ talk (hosted on: YouTube)</a>
<a href="https://www.bilibili.com/video/BV1uU4y1J7XZ">📽️ talk (hosted on: bilibili)</a>
<a href="slides/session4/moderator4.pptx">📔 slides (hosted on: github)</a>
    <br>
    <a href="https://youtu.be/Fb-KGI8uHfk">📽️ Zoom Discussion (part 1 of 2) (hosted on: YouTube)</a>
<br>
    <a href="https://youtu.be/oTjPry9v5KE">📽️ Zoom Discussion (part 2 of 2) (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1wb4y117oC">📽️ Zoom Discussion (part 1 of 2)(hosted on: bilibili)</a>
<a href="https://www.bilibili.com/video/BV1ff4y1w75Z">📽️ Zoom Discussion (part 2 of 2)(hosted on: bilibili)</a>
  <li><a href="#Diamos">Greg Diamos, Peter Mattson and David Kanter</a></li>
  <li><a href="#Ferrucci">Dave Ferrucci</a></li>
  <li><a href="#Dagan">Ido Dagan</a></li>
  <li><a href="#Bowman">Samuel Bowman</a></li>
  <li><a href="#Choi">Eunsol Choi</a></li>
  <li><a href="#Alena">Alëna Aksënova, Daan van Esch, James Flynn and Pavel Golik</a></li>
</ol>

<li id="S5"> Session 5 (9PM): Convenient for America + Asia (Present 2) </li>
  <ol>
    Moderator: Ellen Voorhees <a href="https://youtu.be/0pE7QTaJ1TU">📽️ talk (hosted on: YouTube)</a>

<a href="https://www.bilibili.com/video/BV16b4y1r7rD">📽️ talk (hosted on: bilibili)</a>
<a href="slides/session5/Session5.pptx">📔 slides (hosted on: github)</a>
    <br>
    <a href="https://youtu.be/VJXT-qAKmkg">📽️ Zoom Discussion (hosted on: YouTube)</a>
    <br>
    <a href="https://www.bilibili.com/video/BV1jU4y1A7UV">📽️ Zoom Discussion (hosted on: bilibili)</a>
<li><a href="#Ming">Nan Duan, Qi Zhang and Ming Zhou</a></li>
<li><a href="#Wu">Hua Wu and Jing Liu</a></li>
<li><a href="#Ryant">Neville Ryant</a></li>

</ol>

<h2>Submissions</h2>

We invite original research papers from a wide range of topics, including
but not limited to:

<ol>
<li>What important technologies and underlying sciences need to be fostered, now and in the future?</li>
<li>In each case, are there existing tasks/benchmarks that move the field in the right direction?</li>
<li>Where are there gaps?</li>
<li>For the gaps, are there initial steps that are accessible, attractive, and cost effective?</li>
<li>How large should a benchmark be?  
  <ol>
<li>How much data do we need to measure significant differences?  </li>
<li>How much data do machines need to obtain good performance?  </li>
<li>How much data do babies need to learn language? </li>
  </ol> </li>
</ol>

</ul>

Submissions to https://www.softconf.com/acl2021/w01_Benchmarking-2021/ should
follow the ACL2021 style, and the <a href="https://www.aclweb.org/adminwiki/index.php?title=ACL_Policies_for_Submission,_Review_and_Citation">ACL submission policy</a>. 
Long papers may consist of up to eight (8) pages of content, plus
unlimited references; short papers may consist of up to four (4) pages of
content.  Final versions will be given one additional page of content so
that reviewers' comments can be taken into account.

<p>
The workshop is scheduled to last for one day either August 5th or 6th. If
you have any questions, contact us at <a href = "mailto: pc-benchmarking-ws-acl2021@googlegroups.com">pc-benchmarking-ws-acl2021@googlegroups.com</a>.

<h2>Workshop Organizers</h2>

<ul>
  <li> Kenneth Church (Baidu USA)</li>
<li> Mark Liberman (University of Pennsylvania)</li>
<li> Valia Kordoni (Humboldt-Universität zu Berlin)</li>
  </ul>
  
<h2>Program Committee</h2>

<ul>
<li> Eduardo Blanco (University of North Texas)</li>
<li> Nicoletta Calzolari (Italy) </li>
<li> Kenneth Church (Baidu, USA) </li>
  <li> Christian Federmann (Microsoft Research, USA) </li>
<li> Valia Kordoni (Humboldt, Germany) </li>
<li> Julia Hirshberg (Columbia, USA) </li>
<li> Lori Lamel (LIMSI, France) </li>
<li> Mark Liberman (Penn, USA) </li>
  <li> Phillip Koehn (JHU, USA) </li>
<li>Barbara Plank (IT University of Copenhagen, Denmark)</li>

<li>Preslav Nakov (Qatar Computing Research Institute (QCRI), HBKU)</li>

<li>Anette Frank (University of Heidelberg, Germany)</li>

<li>Roy Bar-Haim (IBM Research - Haifa, Israel)</li>
  </ul>

<h2>Abstracts for Invited Talks</h2>

<h3 id="Bowman">Speaker: Sam Bowman</h3>

<br>
New York University
<br>
<a href="https://cims.nyu.edu/~sbowman/">https://cims.nyu.edu/~sbowman/</a>
<br>
<a href="https://twitter.com/sleepinyourhat">https://twitter.com/sleepinyourhat</a>
<p>
  <a href="https://www.youtube.com/watch?v=OYaDkOHFhAI">📽️ talk (hosted on: YouTube)</a>
  <br>
<a href="slides/session4/Bowman.pdf">📔 slides (hosted on: github)</a>
  <br>
<a href="https://underline.io/events/167/sessions?eventSessionId=6282">📔 slides (hosted on: Google Drive)</a>

<h4>Title: What Will it Take to Fix Benchmarking in Natural Language Understanding?</h4>

Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.

<p>

Bio: Sam Bowman has been on the faculty at NYU since 2016, when he
completed PhD with Chris Manning and Chris Potts at Stanford. At NYU,
he is a member of the Center for Data Science, the Department of
Linguistics, and Courant Institute's Department of Computer
Science. His research focuses on data, evaluation techniques, and
modeling techniques for sentence and paragraph understanding in
natural language processing, and on applications of machine learning
to scientific questions in linguistic syntax and semantics. He is the
senior organizer behind the GLUE and SuperGLUE benchmark competitions
and he received a 2015 EMNLP Best Resource Paper Award, a 2019 *SEM
Best Paper Award, a 2017 Google Faculty Research Award, and a 2021 NSF
CAREER award.

<h3 id="Choi">Speaker: Eunsol Choi</h3>
<h4>Title: Context for Interpreting Benchmark Performances</h4>

<a href="https://youtu.be/3Lo-iqo1PjQ">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1Jv411J76z">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session4/benchmark_talk_acl2021_workshop_eunsol.pdf">📔 slides (hosted on: github)</a>

Interpreting benchmark results requires a more nuanced study than
simply comparing a single number (e.g., accuracy). For example, higher
performance on benchmark focusing on multi-hop reasoning does not
translate to model architecture focusing on multi-hop reasoning but
often a bigger pretrained model. In the first half of the talk, I will
discuss the nuances of interpreting benchmark results, and our
previous efforts in integrating highly relevant axis, computational
resources, into evaluation. In the second half of the talk, I will
talk about the issues with the static benchmarks in the evolving
world. Unlike traditional benchmarks which mostly targeted linguistic
knowledge, modern benchmark embraces common sense, social context, and
encyclopedic world knowledge into the task definition. All these
components change over time, urging NLP benchmarks to be refreshed.
<p>

Bio: Eunsol Choi is an assistant professor in the computer science
department at the University of Texas at Austin. Her research focuses
on natural language processing, various ways to recover semantics from
unstructured text. Prior to UT, she was a visiting faculty researcher
at Google AI. She received a Ph.D. from the University of Washington
(with Luke Zettlemoyer and Yejin Choi) and an undergraduate degree in
mathematics and computer science from Cornell University. She is a
recipient Facebook Research Fellowship, Google Research Award and has
co-organized many workshops related to question answering at NLP and
ML venues.

<h3 id="Dagan">Speaker: Ido Dagan</h3>

<br>
Bar-Ilan University 
<br>
<a href="https://u.cs.biu.ac.il/~dagan/">https://u.cs.biu.ac.il/~dagan</a>
<br>
<a href="https://youtu.be/Jb6G25ZJ-VU">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1cb4y1z75a">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session4/Dagan.pptx">📔 slides (hosted on: github)</a>
<p>
<h4>Title: Moving out of the comfort zones: desired shifts in NLP benchmarking</h4>
<p>
As the deep-learning era has transformed the NLP field, benchmarking
practices haven’t changed that much, often addressing earlier language
analysis tasks and applications. While performance on many benchmarks
rocketed, mostly in deep learning comfort zones, profound language
technology is still a long way ahead. In this talk, I will argue for
three desired interrelated shifts in NLP benchmarking, which motivate
and support each other, that should direct further research.
<p>
First, much more emphasis should be given to typical realistic
settings, in which large training data for the target task is not
available, like few-shot and transfer learning. Moreover, benchmarks
design should fit realistic data compositions, rather than synthetic
ones within the comfort zone, as I will illustrate by a recent
few-shot relation classification dataset. Second, recognizing the
limits of foreseeable fully-automated methods in addressing the hard
NLP challenges, I suggest developing principled evaluation
methodologies for various interactive NLP settings. Interaction may
lead to better results, with the help of a human in the loop, and
moreover allow personalized and explorative behavior, as I will
demonstrate with a recent framework for evaluating interactive
summarization. Lastly, while many current models operate in an
end-to-end manner over implicit language structures, I argue that it
is pertinent to pursue also explicit representations for textual
information structure, to facilitate refined and better-controlled
modeling. Unlike traditional semantic formalisms, I propose pursuing
semi-structured representations, consisting of natural language
expressions over which current powerful text-embeddings can be
applied. I will illustrate this direction by an approach for
decomposing the information in single and multiple texts into sets of
question-answer pairs, and draw some analogies from our successful
experience in designing the Recognizing Textual Entailment (RTE, later
aka NLI) task.
<p>
Bio: 
Ido Dagan is a Professor at the Department of Computer Science at
Bar-Ilan University, Israel, the founder of the Natural Language
Processing (NLP) Lab at Bar-Ilan, the founder and head of the
nationally-funded Bar-Ilan University Data Science Institute and a
Fellow of the Association for Computational Linguistics (ACL). His
interests are in applied semantic processing, focusing on textual
inference, natural open semantic representations, consolidation and
summarization of multi-text information, and interactive text
summarization. Dagan and colleagues initiated textual entailment
recognition (RTE, later aka NLI) as a generic empirical task. He was
the President of the ACL in 2010 and served on its Executive Committee
during 2008-2011. In that capacity, he led the establishment of the
journal Transactions of the Association for Computational Linguistics,
which became one of two premiere journals in NLP. Dagan received his
B.A. summa cum laude and his Ph.D. (1992) in Computer Science from the
Technion. He was a research fellow at the IBM Haifa Scientific Center
(1991) and a Member of Technical Staff at AT&T Bell Laboratories
(1992-1994). During 1998-2003 he was co-founder and CTO of FocusEngine
and VP of Technology of LingoMotors, and has been regularly consulting
in the industry. His academic research has involved extensive
industrial collaboration, including funds from IBM, Google,
Thomson-Reuters, Bloomberg, Intel and Facebook, as well as
collaboration with local companies under funded projects of the Israel
Innovation Authority.

<h3 id="Diamos">Speakers: Greg Diamos, Peter Mattson and David Kanter</h3>
<br>
<a href="https://www.anandtech.com/show/14754/hot-chips-31-live-blogs-mlperf-benchmark">https://www.anandtech.com/show/14754/hot-chips-31-live-blogs-mlperf-benchmark</a>
<br>
<a href="https://youtu.be/unyQPxeGDw4">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1244y1178M">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session4/Diamos.pptx">📔 slides (hosted on: github)</a>

<h4>Title: MLPerf</h4>

Two topics: (1) What is MLPerf?  (2) Advice for groups wanting to create new sets of benchmarks.

<p>

Bio: Greg is helping build Landing AI, a new company focused on
bringing AI to every major industry starting with our first
manufacturing visual inspection product, LandingLens.  Greg co-founded
MLPerf and MLCommons.  Greg helped found Baidu’s Silicon Valley AI
Lab, where he contributed to the DeepSpeech, DeepVoice, and Mixed
Precision training systems. Greg contributed the independent thread
scheduling system to the NVIDIA Volta GPU.
<p>
He holds a Ph.D. in electrical engineering from the Georgia Institute of Technology.

<h3 id="Ming">Speakers: Nan Duan, Qi Zhang and Ming Zhou</h3>

<br>
Nan Duan (Microsoft Research Asia)
<br>
Qi Zhang (Fudan University)
<br>
Ming Zhou (Sinovation Ventures) 
<br>
<a href="https://www.youtube.com/watch?v=7kAR3eUkH_0">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.dropbox.com/sh/w6h0w63l27aakxt/AAAYBZYYV0-QkK7dW7JQTNjFa?dl=0">📽️ talk and 📔 slides (hosted on: Dropbox)</a>
<br>
<a href="https://www.bilibili.com/video/BV1Y64y1W7At">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session5/Zhang_Duan_Zhou_Benchmark.ACL2021.final.pptx">📔 slides (hosted on: github)</a>

<h4> Title: Really Reaching Human Parity? –Addressing Benchmark Issues on Robustness, Bias and Metrics</h4>

We use Machine Reading Comprehension as an example to recap the
current status of NLP benchmarks and highlight four key issues with
the existing benchmarks including (1) lack of robustness testing on
the new independent (but similar) dataset or adversarial inputs, (2)
strong constraints on experimental conditions, (3) bias brought by
data sampling or human annotation, and (4) lack of suitable evaluation
metrics. Then we present our thoughts and experiments on the possible
solutions to these challenges from various aspects.

<p>
Bios:
<p>
Nan DUAN is currently a principal researcher & research manager at
Microsoft Research Asia. He is also an adjunct professor at Tianjin
University. His research interests include question answering,
semantic parsing, large-scale pre-trained models, code intelligence
and machine reasoning. He served as evaluation chair of NLPCC and SAC
or AC of ACL/EMNLP/NAACL. He leads the benchmark dataset efforts such
as XGLUE and CodeXGLUE. He published 100+ research papers and holds
10+ patents. His research has been applied in many Microsoft products.
<p>
 
Qi Zhang is a professor in the school of computer science at the Fudan
University.  His Dr. degree in Computer Science was received from
Fudan Univerisity, in 2009. From January 2005 to January 2006, he
served as a Research Intern at the Bosch Research and Technology
Center, Palo Alto, USA. Since April 2009, he has been with the School
of Computer Science, Fudan University. His research focuses on natural
language processing and information retrieval. He is a recipient IBM
faculty Award and ACM Shanghai Rising Star Award.
<p>

Ming Zhou is the Chief Scientist of Sinovation Ventures AI
Institute. Before joining Sinovation Ventures in the end of 2020,
Dr. Zhou was the Assistant Managing Director at Microsoft Research
Asia (MSRA). He was the president of ACL in 2019. He is vice president
of China Computer Federation (CCF) and steering committee member of
Chinese Information Processing Society of China (CIPS). His research
interests are advancing NLP technologies and applications with the
support of big data, knowledge graph and domain modellings. He was the
inventor of China’s first Chinese-English machine translation (CEMT-1,
1989) and the most famous Chinese-Japanese machine translation
products(J-Beijing,1998). He has published over 220 papers at top NLP
and AI conferences including over 80 papers at ACL over the years.

<h3 id="Ferrucci">Speaker: Dave Ferrucci</h3>
Founder & CEO, <a href="https://ec.ai/">Elemental Cognition</a>
<br>
<a href="mailto:davef@ec.ai">davef@ec.ai</a>
<br>
<a href="https://ec.ai/">ec.ai</a>
<br>
<a href="https://vimeo.com/580829316/53ca42f3c4">📽️ talk (hosted on: Vimeo)</a>
<br>
<a href="https://www.youtube.com/watch?v=2YR9oUor3n0">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1Ah411z77c/">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session4/ACL_Talk_Ferrucci.pdf">📔 slides (hosted on: github)</a>

<h4>Title: Machine Understanding in Context</h4>

The ability for machines to read, understand and reason about natural
language would dramatically transform the knowledge economy across all
industries. Today’s latest Deep Learning marvels do not understand
what they read to the extent required for rational problem solving and
transparent decision making. And yet we need machines to read,
understand and engage with us at a rational level for us to take
responsibility for their predictions.  A potential problem slowing the
advancement of natural language understanding may be that we are not
ambitiously or rigorously defining what it means to comprehend
language in the first place. Current metrics and tests may be
insufficient to drive the right results. In this talk, I will present
a definition of comprehension and early experimental results that
strongly suggest existing systems are not up to the task. I will also
demonstrate a system architecture and behavior that reflects the sort
of language understanding capabilities we envision would do better to
advance the field of NLU.

<p>  
Bio: Dave Ferrucci is an award-winning Artificial Intelligence
researcher who started and led the IBM Watson team from its inception
through its landmark Jeopardy success in 2011. Dr. Ferrucci’s more
than 25 years in AI and his passion to see computers fluently think,
learn, and communicate inspired him to found Elemental Cognition in
2015. Elemental Cognition is an AI company focused on deep natural
language understanding. It explores methods of learning that result in
explicable models of intelligence and cross-industry applications.  
<p>
Dr. Ferrucci graduated from Rensselaer Polytechnic Institute with a
Ph.D. in Computer Science. He has over 100 patents and
publications. He is an IBM Fellow and has worked at IBM Research and
Bridgewater Associates directing their AI research. He has keynoted at
highly distinguished venues around the world. Dr. Ferrucci serves as a
member of the Connecticut Academy of Science and Engineering and an
Adjunct Professor of Entrepreneurship and Innovation at the Kellogg
School of Management at Northwestern University.

<h3 id="Kiela">Speaker: Douwe Kiela</h3>
<br>
Facebook AI Research
<br>
<a href="https://douwekiela.github.io/">https://douwekiela.github.io/</a>
<br>
@douwekiela on Twitter

<h4>Title: Dynabench: Rethinking Benchmarking in AI</h4>

<a href="https://www.youtube.com/watch?v=OK-AqphGhbw">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1H64y1B7vN/">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session2/Dynabench_ACL_Benchmarking_Workshop.pdf">📔 slides (hosted on: github)</a>

The current benchmarking paradigm in AI has many issues: benchmarks
saturate quickly, are susceptible to overfitting, contain exploitable
annotator artifacts, have unclear or imperfect evaluation metrics, and
do not necessarily measure what we really care about. I will talk
about our work in trying to rethink the way we do benchmarking in AI,
specifically in natural language processing, focusing mostly on the
Dynabench platform.

<p>

Bio: Douwe Kiela is a Research Scientist at Facebook AI Research, working on natural language processing and multimodal reasoning and understanding. His work has mainly been focused on representation learning, grounded language learning and multi-agent communication. Recently, he has become interested in improving the way we evaluate AI systems.

<h3 id="Makhoul">Speaker: John Makhoul</h3>
<a href="https://www.youtube.com/watch?v=OMBZ7NV6Hck">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1i54y1E7Xx">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="https://drive.google.com/file/d/1n4pNYfPfXoTM1dvXWMFy4mOctRIjJ6Ph/view?usp=sharing">📔 slides</a>
<br>
<a href="slides/session3/Makhoul Benchmarking 2021 refs.pptx">📔 slides (hosted on: github)</a>

<h4>Title: The Dawn of Benchmarking</h4>

Benchmarking, or common evaluations, can be traced back to a speech recognition workshop in 1987 that pitted a knowledge- or rule-based method against an automatically trainable method on an evaluation task with a defined corpus.  The workshop was part of the DARPA Strategic Computing Program.  Deciding on an evaluation metric was a contentious issue that was settled soon after into the currently used word error rate.  Program managers at DARPA continued to champion the idea of metrics-based common evaluations with defined training and test corpora and, by inviting international research groups to participate in these annual common evaluations, this benchmarking paradigm took hold and spread to other DARPA programs and internationally.  DARPA also provided seed funding for the establishment of the Linguistic Data Consortium, which was instrumental in making common corpora available to the world at large.

<p>
Bio: John Makhoul is a Chief Scientist at Raytheon BBN Technologies, Cambridge, MA, where he has been working on various aspects of speech and language processing, including speech analysis and synthesis, speech coding, speech recognition, speech enhancement, artificial neural networks, human-machine interaction using voice, optical character recognition, machine translation, and cross-lingual information retrieval.  He is a Fellow of the IEEE, the International Speech Communication Association (ISCA), and the Acoustical Society of America.  Makhoul is the recipient of the ISCA medal and several IEEE awards, including the Flanagan medal in speech and audio processing.

<h3 id="Liberman">Speaker: Mark Liberman</h3>

<h4>Title: Benchmarking as a Method for Long-Term Research Management: The Common Task Method</h4>
<br>
<a href="https://youtu.be/a-ukPup8gKw">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1Nf4y1V742">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session3/LibermanBenchmarking.pptx.gz">📔 slides (hosted on: github)</a>

Over the course of half a century, DARPA's Human Language Technology program created capabilities such as speech recognition, machine translation, and text understanding, turning them from science fiction fantasies to everyday practical fact. This sustained success was based on the development of the Common Task Method, which allowed decades of incremental progress in advance of commercial viability. I'll describe the origin and (sometimes counter-intuitive) progress of this method, distinguish it from other uses of benchmarking, and speculate about its future.
<p>
Bio: Mark Liberman is the Christopher H. Browne Professor of Linguistics at the University of Pennsylvania, with positions in the department of computer science and in the psychology graduate group. He is also founder and director of the Linguistic Data Consortium. Before coming to the University of Pennsylvania, he was head of the linguistics research department at AT&T Bell Laboratories.

<h3 id="MacWhinney">Speakers: Brian MacWhinney and Saturnino Luz</h3>
<br>
Language Technologies and Modern Languages, CMU
<br>
University of Edinburgh
<br>
<a href="https://www.research.ed.ac.uk/en/persons/saturnino-luz-filho">https://www.research.ed.ac.uk/en/persons/saturnino-luz-filho</a>
<br>

<a href="https://youtu.be/K_Et3r2GWf0">📽️ Brian's talk (hosted on: YouTube)</a> and 
<a href="https://youtu.be/pxdcn7fkvsI">📽️ Saturnino's talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1mo4y1S7hH">📽️ Brian's talk (hosted on: bilibili)</a>
<br>

<a href="slides/session1/Brian.pptx">📔 Brian's slides (hosted on: github)</a>
<br>
<a href="slides/session1/luz-macwhinney-talk.pdf">📔 Luz's slides (hosted on: github)</a>

<h4>Title: Detection of Dementia from Speech Samples</h4>
<p>
Diagnosis or early detection of the onset of dementia is important
for interventions and planning for life-style changes.  Ideally, we
would like to achieve accurate diagnosis based on samples of
naturalistic language production, as well as samples ellicited using
some standard formats, such as narrative, script reading, or picture
description. Currently, research in this area relies primarily on the
Pitt Corpus in DementiaBank which includes cookie theft narratives
from 104 controls, 208 persons with dementia, and 85 persons with
unknown diagnosis. These data were used in the ADReSS challenge for
INTERSPEECH2020 and will be used in a new challenge for 2021.  The
previous challenge used hand-created transcripts.  The new challenge
focuses on a pipeline that can be applied automatically, using ASR and
NLP methods. The four major gaps in the current data set are: 1) we
need fuller ancillariy data on cognitive and medical status, 2) we
need longitudinal data on progression, 3) we need more data across
language task and interaction types, and 4) ideally, we would like to
have data recorded in the home with voice assistant technology.
Currently, challenge participants are committed to open sharing of
algorithms, but we need more sharing of primary language data,
including data outside of English.

<p>
Bios: Brian MacWhinney is Teresa Heinz Professor of Psychology,
Computational Linguistics, and Modern Languages at Carnegie Mellon
University. His Unified Competition Model analyzes first and second
language learning as aspects of a single basic system. He has
developed a series of 13 TalkBank open access online databases for the
study of language learning, multilingualism, and language
disorders. The databases for language disorders include AphasiaBank,
ASDBank, DementiaBank, FluencyBank, RHDBank, and TBIBank. These
databases provide transcriptions of spoken language linked to audio
and video media, along with programs for analysis and linguistic
profiling. His other research topics include methods for online
learning of second language vocabulary and grammar, neural network
modeling of lexical development, fMRI studies of children with focal
brain lesions, ERP studies of between-language competition, and the
role of embodied perspectival imagery in sentence processing.
<p>
Dr. Luz is a reader in medical informatics at the Usher Institute,
Edinburgh medical School. His is interested in the use of computational
methods in the study of behavioural changes caused by neurodegenerative
diseases, with focus on vocalisation and linguistic behaviour.  He has
also studied interaction in multidisciplinary medical team meetings,
doctor-patient consultations, telemedicine and patient safety.

<h3 id="Mashey">Speaker: John Mashey</h3>
<br>
<a href="https://en.wikipedia.org/wiki/John_Mashey">https://en.wikipedia.org/wiki/John_Mashey</a>
<br>
<a href="https://www.spec.org/benchmarks.html">https://www.spec.org/benchmarks.html</a>
<br>
<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.325.7027&rep=rep1&type=pdf"> War of the Benchmark Means: Time for a Truce</a>, ACM SIGARCH, 2004.
<br>
<a href="https://ieeexplore.ieee.org/document/1525995">Summarizing Performance is No Mean Feat</a>, IEEE 2005 (<a href="https://techviser.com/docs/Mashey.nomeanfeat.2008.pdf">slides</a>)
<br>
<a href="https://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story">https://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story</a>
<br>
Twitter:@johnmashey
<br>
<a href="https://youtu.be/KmwFw0GHET4">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV17o4y1S7Er">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session3/Mashey2.pptx">📔 slides (hosted on: github)</a>

<h4>Title: Lessons from SPEC</h4>

In the 1980s, amidst fierce competition among new microprocessor architectures, CPU benchmarking was in poor condition.
Many commonly-used benchmarks were  small synthetic benchmarks like Whetstone and Dhrystone that poorly-matched realistic programs.
Companies sometimes outright cheated by special-casing compilers to recognize major benchmarks.
Some vendors honestly reported results from realistic benchmarks, but even when running the same programs, often used different inputs,
so that potential customers could not easily make direct comparisons. Many customers did not trust performance claims.

<p> 
The talk reviews the odd way SPEC got started in 1988, initially by MIPS, Apollo, Hewlett-Packard and Sun, later joined by many others,
then covers the ground rules that evolved to let fierce competitors work together successfully to produce benchmarks that became industry standards and
exemplars of good methodologies for selecting benchmarks, validating results, reporting them carefully and deciding when they had to be retired as obsolete for one reason or another.

<p>
SPEC of course is still active, 30+ years later.
The talk reviews lessons learned about high-stakes benchmarking, evolution of benchmark suites over time, competitor social issues, credibility issues when people think the foxes are guarding the henhouse,
as we were asked by a member of the press.
From the beginning, SPEC reported performance on a set of benchmarks as a set of ratios versus a base system, so that people could find benchmarks they thought relevant to their own and ignore the others.
Many arguments had occurred over summary means, but as had been done in some performance reports, SPEC correctly used the Geometric Mean, but without really delving into the underlying statistics,
which only happened in 2004.

<p>
A set of benchmark ratios can be viewed as a sample (representative if selected by experts) from a large population of programs.
In practice, many sets of benchmark ratios are well-fit by the lognormal distribution, whose mean is the Geometric Mean, but also allows computation of a (Multiplicative) Standard Deviation, Confidence Intervals, etc.
The talk briefly reviews the relevant, simple statistics and the rationale for them.

<p>
Bio: John Mashey is a semi-retired computer scientist/corporate executive at Bell Labs, Convergent Technologies, MIPS Computer Systems and Silicon Graphics,
where he is was originator of the phrase “Big Data” (according to NY Times).
He later consulted for venture capitalists, advised startups and occasionally consulted for companies like Nvidia. He is a 20-year Trustee at the Computer History Museum.
He was one of the 4 cofounders of the SPEC benchmarking group in 1988 and was asked in 2018 to advise the MLperf benchmarking group on relevant statistics.

<h3 id="Ryant">Speaker: Neville Ryant</h3>
Linguistic Data Consortium, University of Pennsylvania
      <br>
      <a href="https://dihardchallenge.github.io/dihard3/">DIHARD</a>
<p>
  <a href="https://youtu.be/3TpB0n3HliY">📽️ talk (hosted on: YouTube)</a>
<br><a href="https://www.bilibili.com/video/BV1eQ4y1f7rj">📽️ talk (hosted on: bilibili)</a>
  <br>
<a href="slides/session5/bppf_2021_ryant_slides.pptx">📔 slides (hosted on: github)</a>
<p>
<h4>Title: Benchmarking for diarization. Lessons from the DIHARD evaluation series</h4>

Recently, there has been renewed interest in speaker diarization --
that is, the task of determining “who spoke when” in a recording. With
this renewed interest has come major improvements in system
performance with error rates for the DIHARD challenge falling by 33%
in the span of 4 years. However, despite these successes, the goal of
truly robust diarization which is resilient to the full range of
natural variation in recordings (e.g., conversational domain,
recording equipment, reverberation, ambient noise) remains elusive. In
this talk we will review the evolution of the state-of-the-art on
multiple domains from the DIHARD dataset as well as some challenges we
have encountered in attempting to construct a representative
diarization benchmark.
<p>

Bio: Neville Ryant is a researcher at the Linguistic Data Consortium (LDC) at the University of Pennsylvania, where he has worked on many topics in speech recognition including: forced alignment, speech activity detection, large scale corpus linguistics, computational paralinguistics, and automated analysis of tone. Since 2017, he has been the principal organizer of the DIHARD challenge, the most recent iteration of which (DIHARD III) completed in December 2020.

<h3 id="Sogaard">Speaker: Anders Søgaard</h3>
University of Copenhagen
<br><a href="https://anderssoegaard.github.io/">https://anderssoegaard.github.io/</a>
<br>
<a href="https://www.youtube.com/watch?v=jA9dUFMmcrM">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1gf4y1L7rk">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session2/Sogaard.pdf">📔 slides (hosted on: github)</a>

<h4>Title: 5 Ways to Make Your Data More Relevant</h4>

This talk briefly summarizes works I’ve been involved in that propose
improvements to how we evaluate our models, e.g., presenting sampling
strategies that better simulate real-life scenarios. The talk will be
a sort of self help talk with simple, practical advice for how to add
value to your existing data.

<h3 id="Voorhees">Speaker: Ellen Voorhees</h3>
<br><a href="https://trec.nist.gov">National Institute of Standards and Technology</a>
<br><a href="https://www.nist.gov/people/ellen-m-voorhees">https://www.nist.gov/people/ellen-m-voorhees</a>
<br><a href="slides/session3/Ellen_benchmarking.pdf">📔 slides (hosted on: github)</a>
<br>
<a href="https://www.nist.gov/video/coopetition-ir-research-presented-ellen-voorhees">📽️ talk (hosted on: NIST)</a>

<p>
Coopetitions are activities in which competitors cooperate for a common good. Community evaluations such as the Text REtrieval Conference (TREC) are prototypical examples of coopetitions in information retrieval (IR) and have now been a part of the field for thirty years. This longevity and the proliferation of shared evaluation tasks suggest that, indeed, the net impact of community evaluations is positive.  But what are these benefits, and what are the attendant costs?

This talk will use TREC tracks as case studies to explore the benefits and disadvantages of different evaluation task designs. Coopetitions can improve state-of-the-art effectiveness for a retrieval task by establishing a research cohort and constructing the infrastructure---including problem definition, test collections, scoring metrics, and research methodology---necessary to make progress on the task. They can also facilitate technology transfer and amortize the infrastructure costs. The primary danger of coopetitions is for an entire research community to overfit to some peculiarity of the evaluation task. This risk can be minimized by building multiple test sets and regularly updating the evaluation task.]

<p> Bio: Ellen Voorhees is a Senior Research Scientist at the US National Institute of Standards and Technology (NIST).  Her primary responsibility at NIST is to manage the Text REtrieval Conference (TREC) project, a project that develops the infrastructure required for large-scale evaluation of search engines and other information access technology.  Voorhees' research focuses on developing and validating appropriate evaluation schemes to measure system effectiveness for diverse user tasks.
<p>
Voorhees is a fellow of the ACM and an inaugural member of the ACM SIGIR Academy.   She has published numerous articles on information retrieval techniques and evaluation methodologies and serves on the review boards of several journals and conferences.

<h3 id="Wu">Speakers: Hua Wu and Jing Liu</h3>
<br>
Baidu
<br>
<a href="https://wuhuanlp.github.io/">Hua Wu</a>
<br>
<a href="https://www.machinereading.ai/">Jing Liu</a>
<br>
<a href="https://youtu.be/lKPeaFAgHrk">📽️ talk (hosted on: YouTube)</a>
<br>
<a href="https://www.bilibili.com/video/BV1Ug41177dQ">📽️ talk (hosted on: bilibili)</a>
<br>
<a href="slides/session5/wu_hua.pdf">📔 slides (hosted on: github)</a>

<h4>Title: Benchmarks: An Industry Perspective</h4>

In recent years, the researchers from academia created large-scale
datasets mainly in a crowdsourcing way, that accelerate the
development of NLP technology. However, these datasets might present
different distributions and different challenges from the ones in
real-world applications. In this talk, we will introduce our efforts
on building NLP benchmarks from an industry perspective. Specifically,
we will describe our released datasets on the tasks including question
answering, dialogue and simultaneous translation that were created to
tackle with the problems in industrial applications. We will present
the challenges of these datasets and show how these datasets drive the
advancements of NLP technologies. Additionally, we will talk about
LUGE, which is an Open-Source Project of Chinese NLP benchmarks. LUGE
aims to evaluate NLP models in terms of robustness and adaptability
across multiple tasks and multiple domains, which are very crucial for
their success in industrial applications.
 
<p>
Bios:

Hua Wu is the chair of Baidu tech committee and tech leader of Baidu NLP. Before that, she worked for Toshiba (China) R&D center and Microsoft Research Asia. She obtained her Ph.D. degree from Institute of Automation, Chinese Academy of Science in 2001. Her research interests span a wide range of topics including machine translation, dialogue systems, knowledge graph, etc. She was the Program Co-Chair of ACL 2014 and AACL 2020 (Asia-Pacific Chapter of ACL).
<p>
Jing Liu is a principal architect and a tech leader of deep question answering team at Baidu NLP since 2017. Before that, he was a researcher at Microsoft Research Asia (MSRA). He obtained Ph.D. degree in computer science from Harbin Institute of Technology (HIT) in 2014. He is interested broadly in natural language processing and information retrieval, with a particular focus on building robust end-to-end question answering system. He published over 30 research papers in prestigious conferences including ACL, EMNLP, NAACL, SIGIR, WSDM, CIKM, etc. He served as an Area Chair in ACL 2021.

</body>
</html>

Benchmarking_past_present_future

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README