Academic Affiliate
The National Institute of Standards and Technology's (NIST) Information Technology Laboratory is
seeking a qualified candidate to support the Text Retrieval Conference (TREC, trec.nist.gov). TREC
evaluates AI technologies in information retrieval (IR), search, natural language processing, and
multimedia search, creating human-labeled data to measure effectiveness. TREC hosts annual
evaluations & workshops, releasing datasets and research papers upon completion. The candidate for
this position will work alongside world-class researchers at NIST
Evaluation of search, natural language processing, multimedia, and generative information systems.
- U.S. Citizen Preferred
- A PhD degree in Computer Science.
- 5+ years of relevant experience.
- Experience building systems with large language models as components.
- Familiarity with Docker and Git.
- Experience with Python, JavaScript, and web frameworks such as React and Django.
- Ability to develop prototypes of tools needed to analyze data.
Strong oral and written communication skills.
Other Desirable Qualifications:
- Experience in conducting large-scale IR & multimedia retrieval evaluations.
- Experience with ElasticSearch, Pyserini, or Terrier (open-source IR research platforms).
Key responsibilities will include but are not limited to:
- Develop software for topic creation, relevance assessment, and generative output annotation.
- Develop scoring software for evaluation outputs.
- Develop and maintain systems used to register, submit outputs, and see evaluation results.
- Conduct research on evaluating information access systems, including but are not limited to:
Automating the evaluation of AI-generated outputs.
Leveraging human insight and expertise with AI support for data annotation and evaluation.
Designing metrics for AI-generated outputs in information-seeking contexts.
Interfaces for annotating data that support consistency and identify errors.
Quality control processes for data annotation.
Leaderboard designs that support cooperative research instead of competition.