Project Milestone

DUE WEDNESDAY, NOVEMBER 4 AT 11:59PM.

Table of contents

Overview

The milestone report serves as a mid-project progress check. The goal is to document your progress, demonstrate initial implementation and evaluation results, justify any changes to your original plan, and identify remaining risks before the final report.

Option 1: RAG System

Your report should cover the following:

1. Progress Check in

  • Implementation Progress: Break down your project into major components or steps (e.g., dataset processing, baseline implementations, and your proposed approach), and indicate which are completed, in progress, or not yet started. For completed components, briefly describe what you implemented and whether they are working as expected.
  • Division of Work: Note which group member was responsible for each major task or component.
  • Progress Assessment and Remaining Work: Are you on track with your original timeline? Identify the major implementation steps and experiments still needed to complete your project, and provide a concrete plan for finishing them before the final deadline. If behind schedule, explain why and how you plan to catch up, including any adjustments to your timeline or project scope.

2. Preliminary Results

By the milestone, your team must have implemented and evaluated both required retrieval baselines (sparse and dense retrieval) on at least one of your benchmark datasets (QASPER or your selected second benchmark).

  • Experimental Setup: Describe the dataset, evaluation subset, retrieval baseline, and evaluation metrics. Explain how you selected the evaluation subset. Use the same document chunking and corpus setup for both baselines to ensure a fair comparison.
  • Results: Report quantitative retrieval-quality results (e.g., Recall@k, Precision@k, or MRR) for both baselines using the same evaluation metrics. Present your results in a table or figure.
  • Discussion: Compare the performance of the two baselines and discuss what you learned from the results. Did you observe any unexpected behavior or encounter implementation challenges?

You are encouraged to include additional results from your proposed approach or end-to-end QA evaluation, but these are optional at this stage.

3. Evolving Methodology

  • Updated Plan: Provide an updated version of your “Proposed Approach” section from the proposal. Highlight all changes from the original proposal. In your LaTeX document, you are encouraged to use the changes package for this (\added{…}, \deleted{…}). If your plan has not changed, you may reuse the original section and state that explicitly.
  • Justification of Changes: Briefly justify each significant change. Explain how your implementation experience, preliminary findings, or challenges led to these adjustments.
  • Response to Proposal Feedback: For comments received on your project proposal, provide a brief (1–2 sentence) response explaining how you have addressed it or plan to address it. If you decide not to incorporate a suggestion, briefly explain why.

4. Potential Risks and Mitigation

  • Risks Already Addressed (if any): Describe any important technical uncertainties or challenges that you have already resolved or reduced, and explain how you addressed them.
  • Remaining Risks (if any): Identify up to two potential risks to completing your project successfully (e.g., “our proposed approach may not improve retrieval quality over the baselines” or “integrating different components may take longer than expected”).
  • Mitigation Plan: For each remaining risk, propose a concrete mitigation strategy. What is your backup plan if the risk materializes?

If you have no risks to report in either category, briefly explain why you do not anticipate significant obstacles.

Option 2: Replicating Research

Your report should cover the following:

1. Progress Check in

  • Implementation Progress: Break down your replication plan into major components or steps, and indicate which are completed, in progress, or not yet started. For completed components, briefly describe what you implemented and whether they are working as expected.
  • Division of Work: Note which group member was responsible for each major task or component.
  • Progress Assessment and Remaining Work: Are you on track with your original timeline? Identify the major implementation steps and experiments still needed to replicate your target result, and provide a concrete plan for completing them before the final deadline. If behind schedule, explain why and how you plan to catch up, including any adjustments to your timeline or project scope.

2. Preliminary Validation and Results

Provide concrete experimental evidence of technical progress toward replicating your selected result. We do not expect you to have completed the full replication experiment by the milestone. Depending on your implementation stage, your evidence could include:

  • Component validation: Tests or small-scale experiments showing that a key component of your implementation works as intended (e.g., correctness tests for a query rewriting algorithm, transaction processing mechanism).
  • Baseline Measurements: Initial performance measurements of the original/baseline system under your experimental setup.
  • Partial Replication: Preliminary results for a subset of the target experiments (e.g., fewer workloads, smaller datasets, or a subset of the configurations in the original paper).

Present your evidence using tables, figures, or test outputs, as appropriate. Explain what the results demonstrate, how you validated them, and how they contribute to your replication goal.

Preliminary results that do not match the original paper are acceptable at this stage. If your implementation is not yet ready for performance evaluation, focus on demonstrating the correctness or functionality of the components you have built.

3. Evolving Methodology

  • Updated Plan: Provide an updated version of your “Replication Plan” section from the proposal. Highlight all changes from the original proposal. In your LaTeX document, you are encouraged to use the changes package for this (\added{…}, \deleted{…}). If your plan has not changed, you may reuse the original section and state that explicitly.
  • Justification of Changes: Briefly justify each significant change. Explain how unforeseen complications (e.g., ambiguous descriptions in the paper or differences in experimental setup) or new findings led you to deviate from your original plan.
  • Response to Proposal Feedback: For comments received on your project proposal, provide a brief (1–2 sentence) response explaining how you have addressed it or plan to address it. If you decide not to incorporate a suggestion, briefly explain why.

4. Potential Risks and Mitigation

  • Risks Already Addressed (if any): Describe any important uncertainties or challenges that you have already resolved or reduced, and explain how you addressed them.
  • Remaining Risks (if any): Identify up to two potential risks to completing your replication successfully (e.g., missing implementation details or hardware limitations).
  • Mitigation Plan: For each remaining risk, propose a concrete mitigation strategy. For instance, “If we cannot replicate Figure 5, we will focus on replicating Table 2 and documenting the reasons for the discrepancy in Figure 5.”

If you have no risks to report in either category, briefly explain why you do not anticipate significant obstacles.

Submission and Grading

Please reuse the LaTeX template from your project proposal.

Submit a PDF of your milestone report on Canvas. Only one submission is required per group.

This assignment is worth 5% of your grade. We will evaluate on the basis of clarity of this report, demonstrated progress toward project goals, and quality of planning and risk assessment.

Note that:

  • Failed experiments or negative results are acceptable and even valuable if properly documented and analyzed
  • Being behind schedule is not penalized if the team provides an honest assessment and concrete recovery plan