Skip to content

Latest commit

Β 

History

1,001 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Capstone Project - HCMUT CS252

Educational Content Processing & Retrieval-Augmented Generation System A comprehensive research platform for multimodal lecture processing, intelligent retrieval, and RAG pipeline development.


Repository Layout

The repo is organized around tracked code/docs and local-only data:

  • docs/ - project documentation, references, architecture notes, and report-ready supporting material.
  • Phase_1/code/ - Phase 1 experiments and prototypes.
  • Phase_1/data/ - Phase 1 raw inputs and generated outputs; ignored by Git and stored in Drive.
  • Phase_1/Report/ - Phase 1 report PDF and report source; tracked.
  • Phase_2/code/ - Phase 2 backend, frontend, deployment, Terraform, SageMaker, and scripts.
  • Phase_2/data/ - Phase 2 raw inputs, pipeline outputs, evaluation data, indexes, model artifacts, and third-party local checkouts; ignored by Git and stored in Drive.
  • Phase_2/Report/ and Phase_2/Manuscript/ - final report/manuscript PDFs and sources; tracked.

See docs/04_PROJECT_REFERENCE/data_manifest.md for Drive restore locations.


README Map

Use this root README as the project entry point. Other README files are scoped to their folder:

README Use it for
README.md Project overview, repo layout, high-level architecture, and where to go next
Phase_1/code/README.md Phase 1 experiment/code map after the week-folder cleanup
Phase_2/README.md Phase 2 app runbook: backend, frontend, SageMaker, Terraform, scripts
Phase_2/code/*/README.md Component-specific instructions for backend, frontend-related scripts, SageMaker, Terraform

If two README files disagree, treat the more specific folder README as authoritative for commands inside that folder, and treat this root README as authoritative for repository structure.


🎯 Project Overview

This capstone builds an educational content processing and Retrieval-Augmented Generation (RAG) system: ingest multimodal lecture materials, align and structure them, index them for text and visual retrieval, and support question answering with citations, lecture-aware summaries, and personalized learning features behind a modern web UI and production-style deployment options.

The authoritative requirements baseline is docs/requirements.md (Software Requirements Specification): 37 requirements in total 22 functional (FR-001–FR-022 and extended FRs in that doc), 8 non-functional (NFR-001–NFR-008), and 7 technical (TR-001–TR-007). Highlights from the SRS scope:

  • Content processing: ASR and timed exports (FR-001); documents, OCR, dual outputs (FR-002); spreadsheet merged cells and Markdown (FR-003, FR-004); images / VLM (FR-005); deduplication (FR-006); audio–slide alignment and temporal navigation (FR-007, FR-008).
  • Retrieval & QA: BM25, dense, hybrid (FR-009); vision–language retrieval (FR-010); query handling (FR-011); grounded answers (FR-012, FR-013); chat decomposition, strategy, and multi-search aggregation (FR-014).
  • Product features: file management and search UI (FR-021, FR-022); automated summaries and summary navigation (FR-023, FR-024); learning paths, assessment, and analytics (FR-025–FR-027).
  • Non-functional: latency and scale targets (NFR-001–NFR-002); availability, integrity, UX, accessibility, security, and privacy (NFR-003–NFR-008).
  • Technical: FastAPI + async APIs, React 18 + Vite + Tailwind, vector and metadata stores, external LLM/embedding services, Docker, and cloud-ready infrastructure (TR-001–TR-007).

Phase 1 research milestones now live under Phase_1/code/; Phase_2 is the maintained integrated app (Firebase UI, Qdrant/S3, optional SageMaker, Terraform for ECS/ALB/ECR).


πŸ—οΈ System Architecture

BK-MInD follows a six-tier Clean Architecture pattern that separates concerns across distinct layers, enabling maintainability, testability, and independent scaling. The system is designed to achieve: (1) multimodal data ingestion from diverse educational materials, (2) asynchronous processing to support concurrent operations without bottlenecks, and (3) production-grade security and scalability on AWS infrastructure.

High-Level System Architecture

The following diagram shows the complete system topology aligning with the SRS: multimodal ingest β†’ process β†’ index β†’ retrieve β†’ generate, organized across six architectural tiers plus cross-cutting concerns for auth, security, and persistence.

BK-MInD High-Level Architecture - Six-Tier Clean Architecture

Layer summary

Layer Role SRS touchpoints
Client Uploads, search, summaries, dashboards, auth FR-021–FR-022, FR-023–FR-027, NFR-005–NFR-008
API Orchestration, RBAC hooks, integration TR-001, NFR-003–NFR-004
Processing ASR, OCR/VLM, spreadsheets, sync, corpus FR-001–FR-008
Storage Vectors, sparse index, blobs, metadata TR-003, TR-004, NFR-004
Retrieval & generation Hybrid + visual search, RAG, chat, LLM FR-009–FR-014, TR-004
Deployment Containers, cloud LB TLS, optional managed GPU TR-006–TR-007, NFR-002–NFR-003

For HTTPS and custom domains on AWS, see docs/technical/DOCS_deployment-alb-acm-custom-domain.md.


AWS Deployment Architecture

The latest deployment architecture (v4) shows production-grade cloud infrastructure on AWS with ECS Fargate, ALB, ElastiCache, vector databases, and auto-scaling:

AWS Deployment Architecture Diagram

For capstone presentations / documentation review: open docs/report/ for Phase 2 reports and presentation material.

For development setup: follow docs/requirements.md for the baseline environment. The main pieces are Python 3.9+ for the backend, React 18 + Vite + Tailwind for the frontend, and FFmpeg / Tesseract / Poppler for media and OCR. GPU use is optional locally if you offload heavy inference to APIs or SageMaker (Phase_2/code/sagemaker/README.md).

Additional Diagrams:

  • docs/diagram/ Complete diagram collection including document processing flows and system documentation

Complete Environment Setup Guide

Last Updated: July 4, 2026 For: Phase 2 maintained application stack Duration: 20-40 minutes for a local backend/frontend setup

This guide matches the current codebase structure documented in ../README.md, ../Phase_2/README.md, ../Phase_2/code/backend/README.md, and ../Phase_2/code/terraform/README.md.

What You Need

  • Python 3.10+ for the backend
  • Node.js 18+ for the frontend
  • Git
  • Docker and Docker Compose for local Qdrant and Redis
  • Optional system binaries: FFmpeg, Tesseract, and Poppler if you run the document-processing paths locally

1. Clone the Repository

git clone https://github.com/pdz1804/capstone-project.git
cd capstone-project

The maintained app lives in Phase_2/.

2. Start Local Infrastructure

From the backend folder, start the local service dependencies:

cd Phase_2/code/backend
docker compose up -d

This starts Redis and Qdrant from docker-compose.yml; it does not start the FastAPI backend itself.

Useful checks:

curl http://localhost:6333/health
docker compose ps

3. Configure and Run the Backend

Create the backend environment file and fill in the values you need:

cd Phase_2/code/backend
cp .env.example .env

Start the backend with the maintained entrypoint:

python -m venv .venv
python -m pip install -r requirements.txt
python run_api.py

Activate the virtual environment with the equivalent command for your shell before installing packages if you prefer not to use the system interpreter.

The API runs on port 5001 by default. The runner accepts --host, --port, --workers, --reload, and --no-reload if you need to override the defaults.

Common backend env values to review:

  • OPENAI_API_KEY
  • GEMINI_API_KEY
  • QDRANT_MODE, QDRANT_HOST, QDRANT_PORT, or QDRANT_URL
  • REDIS_URL
  • FILE_STORAGE_BACKEND
  • USE_AWS_SAGEMAKER_INFERENCE
  • DYNAMODB_USERS_TABLE, DYNAMODB_APP_USAGE_TABLE, DYNAMODB_JOBS_TABLE

4. Configure and Run the Frontend

In a separate terminal:

cd Phase_2/code/frontend
cp .env.example .env
npm install
npm run dev

The current frontend dev script is tsx server.ts, and the default local URL is http://localhost:5173. The template in ../Phase_2/code/frontend/.env.example uses VITE_API_BASE_URL=/api and API_PROXY_TARGET=http://localhost:5001.

5. Verify the Stack

Backend health:

curl http://localhost:5001/health
curl http://localhost:5001/api/health

Frontend should be reachable at http://localhost:5173.

If you want to smoke-test search after indexing data, use the current API contract from ../Phase_2/code/backend/README.md and send requests with the X-User-Id header.

6. Optional Local Validation

Backend tests:

cd Phase_2/code/backend
python -m pytest tests -v

Terraform validation only:

cd Phase_2/code/terraform
terraform init -backend=false
terraform fmt -recursive
terraform validate

7. Common Failure Points

  • If Qdrant is unreachable, confirm docker compose up -d completed and that port 6333 is free.
  • If Redis is unreachable, confirm the same compose stack is running and that the backend REDIS_URL points at redis://localhost:6379/0 or your configured instance.
  • If the frontend cannot reach the API, confirm API_PROXY_TARGET=http://localhost:5001 and that the backend is running on port 5001.
  • If pip install fails on large wheels, clear the pip cache or use a drive with more free space before retrying.

8. Where To Read Next


Phase 2 Integrated Application: FE + AI + AWS (Phase_2/)

Single tree that combines the production-style FastAPI backend (Qdrant, S3, optional SageMaker inference), the React + Firebase frontend from the FE track, SageMaker hosting packs (unified Docling + Whisper + ColQwen container and optional split endpoints), and Terraform for AWS: ECR, ECS Fargate, Application Load Balancer with optional HTTPS (ACM), auto scaling, and an optional SageMaker endpoint aligned with sagemaker/unified.

Area Path Documentation
Folder overview Phase_2/ Phase_2/README.md
Terraform (ALB, ECS, ECR, SageMaker) Phase_2/code/terraform/ Phase_2/code/terraform/README.md
SageMaker build / deploy Phase_2/code/sagemaker/ Phase_2/code/sagemaker/README.md
HTTPS + custom domain runbook docs/technical/DOCS_deployment-alb-acm-custom-domain.md ACM validation, DNS, ALB listeners

Use Phase_2 as the maintained application tree for local development, technical review, deployment, and testing.


πŸš€ Quick Start

πŸ“š For capstone presentations / documentation review: Start with docs/report/ folder for Phase 2 reports and presentation guides.

πŸ‘¨β€πŸ’» For development setup: Prerequisites follow docs/requirements.md (TR-001–TR-005, NFR-005–NFR-006): Python 3.9+, FastAPI backend; React 18+, Vite, Tailwind frontend; FFmpeg, Tesseract, Poppler for media; GPU optional locally if you offload heavy inference to APIs or SageMaker (Phase_2/code/sagemaker/README.md). Docker and Terraform are for packaging and cloud layout (TR-006–TR-007).

Clone and base setup

git clone https://github.com/pdz1804/capstone-project.git
cd capstone-project
python -m venv venv

Activate the virtual environment with the equivalent command for your shell.

Recommended: merged app (Phase_2/)

Full UI (Firebase), Qdrant/S3-aware API, tests, Terraform and SageMaker docs see Phase_2/README.md.

Backend:

cd Phase_2/code/backend
python -m pip install -r requirements.txt
cp .env.example .env
# Edit .env with keys, Qdrant, S3, and SageMaker settings.
python run_api.py

Frontend:

cd Phase_2/code/frontend
npm install
cp .env.example .env
npm run dev

URLs (typical): UI http://localhost:5173 (or your Vite default), API http://localhost:5001, docs http://localhost:5001/docs.

Terraform (local validation only, no apply):

cd Phase_2/code/terraform
terraform init -backend=false
terraform fmt -recursive
terraform validate

Research and pipeline folders (optional)

cd Phase_1/code/asr_ocr/multi_model_benchmark/src
python main.py asr --output-dir results/asr <list-of-video-files>

cd Phase_1/code/retrieval/baseline_bm25_dense_hybrid
jupyter notebook manual_bm25_dense_hybrid.ipynb

cd Phase_1/code/processing_rag_pipeline/rag_framework_comparison
python setup_and_run.py

cd Phase_1/code/processing_rag_pipeline/unified_processor
python src/pipeline.py input/ output/
# Optional: add --fast-mode where that script supports it

Use cd <repoRoot> first if you are not already at the repository root.


πŸŽ“ Academic Context

Course: CS252 - Capstone Project Institution: Ho Chi Minh City University of Technology (HCMUT) Focus: Applied AI for Educational Content Processing Domain: Information Retrieval, NLP, Multimodal Learning, RAG Systems

Research Contributions:

  1. Vietnamese-optimized ASR/OCR pipeline for lecture processing
  2. Comprehensive retrieval method comparison on MS MARCO
  3. RAG framework selection guide for educational Q&A
  4. Production-grade retrieval system implementations
  5. Multimodal document understanding with Docling
  6. Dual-mode RAG processing pipeline (text + image retrieval)
  7. Intelligent document deduplication and caching system
  8. Performance-quality tradeoff framework (Fast vs Full modes)

πŸ“š Documentation

πŸ“– Start Here: docs/requirements.md ⭐ Software Requirements Specification: functional, non-functional, technical constraints (37 requirements total).

Authoritative Technical Documents

Testing and Performance Evidence

Architecture and Deployment

Security and WAF Configuration

Cost Estimation

Merged Production Application (Phase_2/)

Research Milestones and Utilities

  • READMEs inside Phase_1/code/asr_ocr/, Phase_1/code/retrieval/, Phase_1/code/processing_rag_pipeline/, and downloads/ directories (datasets and paper references).
  • Phase_1/code/processing_rag_pipeline/rag_framework_comparison/DETAILED_PIPELINE_FLOWS.md β€” Detailed RAG pipeline flow diagrams and explanations.

πŸ”¬ Research Papers & References

The downloads/ directory contains a curated collection of research papers covering:

  • Retrieval-Augmented Generation (RAG) architectures
  • Dense retrieval methods (DPR, ColBERT, ANCE)
  • Multimodal learning (CLIP, LayoutLM, Docling)
  • Speech recognition (Whisper, Wav2Vec 2.0)
  • OCR and document understanding

🀝 Contributing

This is an academic capstone project. For collaboration or questions:

  • Repository: github.com/pdz1804/capstone-project
  • Issues: Use GitHub Issues for bug reports or feature requests
  • Contact: See individual weekly READMEs for team member information

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright (c) 2025 Quang Phu, Ngoc Khoi, and Minh Khoi


πŸ™ Acknowledgments

Open-source models, APIs, and platforms that this codebase builds on (see also TR-004–TR-005 and integration notes in docs/requirements.md):

  • OpenAI Whisper and LLM APIs used in ASR and generation experiments.
  • Google Gemini (multimodal/API), Firebase (authentication in the merged frontend stack), and embedding-related tooling referenced in weekly work.
  • Hugging Face transformers, model hubs, and pretrained checkpoints (e.g. ColQwen, sentence encoders).
  • IBM Docling and related document-understanding components.
  • Qdrant Vector Database used in the Phase 2 AI service and merge backend.
  • Amazon Web Services S3, SageMaker real-time inference, and (via Terraform) ECS, ECR, ALB, ACM for optional cloud deployment.
  • HashiCorp Terraform for infrastructure as code in Phase_2/code/terraform/.
  • Pyserini / Anserini & Milvus retrieval stacks explored in research-week milestones.
  • LangChain & LlamaIndex RAG framework comparisons (early-phase notebooks and prototypes).
  • FFmpeg, Tesseract, Poppler media, OCR, and PDF tooling (TR-005).
  • React, Vite, Tailwind CSS frontend stack (TR-002).

Version: 1.0 Last Updated: May 10, 2026

Team: MKhoi, NKhoi, QPhu.

Releases

Packages

Used by

Contributors

Languages