Project Proposal

DUE WEDNESDAY, SEPTEMBER 30 AT 11:59PM.

Table of contents

Overview

You can choose one of the following project options:

Team Formation

You will work in teams of 3-4 people. Smaller or larger teams require instructor approval.

  • Use Piazza to find teammates and form groups early in the semester
  • Once you’ve finalized your team, register on Canvas (People → Project Groups) so other students can see who is still looking for teammates.
  • Team size will be considered when setting project scope and grading expectations.

Timeline and Deliverables

WeekDeliverableWeightDescription
W6Project Proposal5%Detailed proposal outlining approach, implementation and evaluation plan
W11Milestone Report5%Progress update with implementation status and preliminary results
W16Final Report15%Complete project including final report and code

PACE Access

Our class has requested access to PACE-ICE for both CPU and GPU compute. Please checkout this tutorial for instructions on how to access and run jobs on PACE.

Option 1: RAG System

A Retrieval-Augmented Generation (RAG) system answers questions over a document corpus in two stages: retrieval, which locates the passages most likely to contain the answer, and generation, which reads those passages and produces a natural-language response.

In this project, you will design and implement a RAG system with a particular focus on the retrieval stage: how documents are represented, indexed, searched, and ranked so that the right evidence can be found reliably and efficiently. Your approach may also use LLMs within the retrieval loop—for example, for query rewriting, reranking, multi-hop retrieval, or deciding when to retrieve additional information (agentic RAG).

You will evaluate retrieval and generation separately. Because language model quality is outside the scope of this course, retrieval quality is the primary evaluation target: does your system retrieve the passages that actually contain the evidence needed to answer the question? A system that consistently retrieves the correct evidence but occasionally generates an imperfect answer is therefore a stronger result than one that produces fluent answers without retrieving the right supporting evidence.

Requirements

The project will be evaluated primarily based on implementation effort, technical depth, and completeness, rather than algorithmic novelty. You are encouraged to build on ideas from existing papers, blog posts, and open-source systems, but you should cite the sources that inform your design.

  • Benchmarks: All teams must evaluate on two benchmarks:
    • Primary: QASPER — Question answering over NLP research papers, with both ground-truth answers and gold evidence paragraphs. The evidence annotations allow retrieval quality to be evaluated directly.
    • Secondary: A second QA or retrieval benchmark of your choice (see References for examples). You are not required to report the same metrics on both benchmarks; choose metrics appropriate for the annotations available in your selected benchmark.
  • Retrieval Baselines: Your proposed retrieval approach must be compared with these two baselines using the same document chunking and corpus setup. You are expected to implement both:
    • Sparse Retrieval: Keyword-based retrieval without semantic understanding (e.g., BM25).
    • Dense Retrieval: Embedding-similarity retrieval using a standard pretrained embedding model (e.g., sentence-transformers/all-MiniLM-L6-v2), without additional reranking, query rewriting, or other retrieval enhancements.

  • Provenance: Every generated answer must include the retrieved text chunk(s) used to support it. At minimum, report the source document and the specific retrieved text segment.

  • Original Implementation: Because many end-to-end RAG frameworks exist (LlamaIndex, LangChain, etc.), it is easy to assemble a working system with minimal engineering effort. This project requires non-trivial original implementation. You should implement the core logic of your retrieval approach rather than simply configure an existing RAG framework.
    • Acceptable: Using an off-the-shelf embedding model, vector index, LLM, or other component within a retrieval pipeline that you design and implement.
    • Not acceptable: Configuring a pre-built RAG framework to provide the complete retrieval and generation pipeline with little or no custom implementation.
  • Evaluation: Clearly separate retrieval quality from end-to-end QA accuracy:
    • Retrieval quality (primary): On QASPER, compare retrieved passages against the gold evidence paragraphs. Report appropriate retrieval metrics for your approach and both required baselines. If your approach uses an LLM for query rewriting, reranking, iterative retrieval, etc., evaluate the resulting retrieved passages in the same way.
    • QA accuracy (secondary): Pass the retrieved passages to a language model and compare its generated answers against ground-truth answers. Report answer accuracy (e.g., exact match, F1, or token overlap). This result is influenced by both retrieval quality and language model capability, so it is treated as supplementary.

Resources and References

  1. LLM access: LLMs are used for generation and, in agentic approaches, within the retrieval loop itself (e.g., query rewriting, iterative retrieval). You may use any LLM API (e.g., OpenAI, Anthropic, Google). If you prefer not to incur API costs, you can serve an open-weight model locally:
    • Ollama: A convenient option for laptops that provides a simple CLI and API for running local models. Supports CPU, Apple Silicon, and CUDA.
    • vLLM: A high-throughput serving framework suitable for teams with access to GPU machines.
    • Course LLM serving guide: step-by-step setup for a chat model and an embedding model on your laptop or on a free PACE ICE GPU.
  2. Popular open-weight model families:
  3. Additional Benchmarks:
    • BEIR: Heterogeneous retrieval benchmark across 18 datasets and domains
    • BRIGHT: A reasoning-intensive retrieval benchmark
    • MuSiQue: Multi-hop question answering
  4. Papers:

Proposal Format

Your proposal should cover the following:

  1. Proposed Approach
    • Describe your end-to-end system design: e.g., document parsing, chunking strategy, embedding and indexing, retrieval, and answer generation.
    • What is the non-trivial original contribution your team will implement (e.g., reranking, query expansion, hybrid sparse+dense retrieval, or agentic retrieval)? Explain its motivation and why you expect it to improve over the required baselines.
    • Cite all sources that inform your proposed solution.
  2. Implementation Plan
    • What programming language and libraries do you plan to use? Clearly distinguish between what you will reuse (e.g., a pretrained embedding model) versus implement from scratch (e.g., chunking logic, reranker, evaluation harness).
    • How do you plan to access LLMs — via API or a locally served model? Which serving framework and model family are you planning to use?
  3. Evaluation Plan
    • Metric of Success: Define the main metrics you will report, including retrieval quality (primary), QA accuracy/generation quality (secondary), and system performance (e.g., latency, cost).
    • Benchmarks: Which second benchmark will you use alongside QASPER? What annotations does it provide, and which metrics will you report on it?
    • Evaluation Set: Both benchmarks can be large. Specify how many samples you plan to evaluate from each and how you will select them (e.g., random sampling or stratification by question type). Justify why your evaluation set is large enough to support meaningful conclusions.
    • How will you ensure a fair comparison between the required retrieval baselines and your proposed approach?
  4. Timeline and Division of Work
    • What do you plan to complete by the milestone report deadline?
    • At a high level, how do you plan to divide the work between the team members?

Option 2: Replicating Research

You will replicate a prior result from a database research paper, following the ACM’s definitions of repeatability, reproducibility, and replicability:

  • Repeatability (Same team, same setup): The original team can consistently reproduce their own results using the same methods, tools, and data.
  • Reproducibility (Different team, same setup): An independent team can verify the results using the original authors’ artifacts (code, data, and methods).
  • Replicability (Different team, different setup): A new team independently re-implements the experiment (with different tools/data) and confirms the conclusions.

For this project, you will focus on replicability–developing your own implementation to validate the paper’s claims.

Where to find papers?

Your selected paper must be published in one of these venues:

  • PACMMOD (Proceedings of the ACM on Management of Data)
  • SIGMOD (ACM Special Interest Group on Management of Data)
  • PVLDB (Proceedings of the VLDB Endowment)
  • ICDE (International Conference on Data Engineering)

A good first step is to look over the programs from recent SIGMOD, VLDB, and ICDE conferences. Identify sessions/topics that interest your team. Skim paper titles/abstracts before selecting one.

Here are some example papers for your inspiration:

Proposal Format

Your proposal should cover the following:

  1. Paper Selection & Citation
    • Please give a full citation that includes the names of all authors, the title, the venue that published it, the year of publication, and a URL.
  2. Intellectual Contribution
    • What is the relevant intellectual contribution of this paper that you will focus on? Explain this in language that is accessible to somebody who has not read the paper.
    • Example: “The paper contributes a new method/technique/approach for solving the problem of A, which arises in the context of B. The new method/technique/approach is C.” Or: “This paper describes an interesting measurement/discovery/finding in the area of A, which is important in the context of B because of C. The discovery/finding is reported to be D.”
    • Before the paper arrived, what was the “status quo? i.e. what was the state of knowledge before the paper and its intellectual contributions arrived?
  3. Result/Claim to Replicate
    • What is the result or claim that you will replicate, that is related to the intellectual contribution you identified above? Please identify a specific figure/table/number and cite where it is mentioned in the paper. Please screenshot the figure or number and include an excerpt in your report.
    • In your own words, what is the result saying? E.g., “The experiments found that the new method/technique/approach produces a 13% improvement in query time compared with the best prior work”.
  4. Replication Plan
    • Briefly outline the steps your team will do to replicate this result. What infrastructure or software will you reuse from prior work? What will you build by following the paper’s instructions? What part(s)/lines/numbers from the figure or text that you screenshotted will you attempt to replicate?
    • Draw a “sketch” of what the resulting replication might look like. E.g. if you’re planning to redraw two of the lines from a graph, draw those lines freehand just to illustrate what the result of your replication might be. (Okay to do this by hand.)
  5. Timeline and Divison of Work
    • What do you plan to complete by the milestone report deadline? At a high level, how do you plan to divide the work between the team members?

Submission and Grading

Use the LaTeX template for the proposal here.

Submit a PDF with your project proposal on canvas. Only one submission is required per group.

This assignment is worth 5% of your grade. We will evaluate on the basis of clarity of this report, completeness of the design and overall description of the approach taken.