Educational Content Processing & Retrieval-Augmented Generation System A comprehensive research platform for multimodal lecture processing, intelligent retrieval, and RAG pipeline development.
The repo is organized around tracked code/docs and local-only data:
docs/- project documentation, references, architecture notes, and report-ready supporting material.Phase_1/code/- Phase 1 experiments and prototypes.Phase_1/data/- Phase 1 raw inputs and generated outputs; ignored by Git and stored in Drive.Phase_1/Report/- Phase 1 report PDF and report source; tracked.Phase_2/code/- Phase 2 backend, frontend, deployment, Terraform, SageMaker, and scripts.Phase_2/data/- Phase 2 raw inputs, pipeline outputs, evaluation data, indexes, model artifacts, and third-party local checkouts; ignored by Git and stored in Drive.Phase_2/Report/andPhase_2/Manuscript/- final report/manuscript PDFs and sources; tracked.
See docs/04_PROJECT_REFERENCE/data_manifest.md for Drive restore locations.
Use this root README as the project entry point. Other README files are scoped to their folder:
| README | Use it for |
|---|---|
README.md |
Project overview, repo layout, high-level architecture, and where to go next |
Phase_1/code/README.md |
Phase 1 experiment/code map after the week-folder cleanup |
Phase_2/README.md |
Phase 2 app runbook: backend, frontend, SageMaker, Terraform, scripts |
Phase_2/code/*/README.md |
Component-specific instructions for backend, frontend-related scripts, SageMaker, Terraform |
If two README files disagree, treat the more specific folder README as authoritative for commands inside that folder, and treat this root README as authoritative for repository structure.
This capstone builds an educational content processing and Retrieval-Augmented Generation (RAG) system: ingest multimodal lecture materials, align and structure them, index them for text and visual retrieval, and support question answering with citations, lecture-aware summaries, and personalized learning features behind a modern web UI and production-style deployment options.
The authoritative requirements baseline is docs/requirements.md (Software Requirements Specification): 37 requirements in total 22 functional (FR-001βFR-022 and extended FRs in that doc), 8 non-functional (NFR-001βNFR-008), and 7 technical (TR-001βTR-007). Highlights from the SRS scope:
- Content processing: ASR and timed exports (FR-001); documents, OCR, dual outputs (FR-002); spreadsheet merged cells and Markdown (FR-003, FR-004); images / VLM (FR-005); deduplication (FR-006); audioβslide alignment and temporal navigation (FR-007, FR-008).
- Retrieval & QA: BM25, dense, hybrid (FR-009); visionβlanguage retrieval (FR-010); query handling (FR-011); grounded answers (FR-012, FR-013); chat decomposition, strategy, and multi-search aggregation (FR-014).
- Product features: file management and search UI (FR-021, FR-022); automated summaries and summary navigation (FR-023, FR-024); learning paths, assessment, and analytics (FR-025βFR-027).
- Non-functional: latency and scale targets (NFR-001βNFR-002); availability, integrity, UX, accessibility, security, and privacy (NFR-003βNFR-008).
- Technical: FastAPI + async APIs, React 18 + Vite + Tailwind, vector and metadata stores, external LLM/embedding services, Docker, and cloud-ready infrastructure (TR-001βTR-007).
Phase 1 research milestones now live under Phase_1/code/; Phase_2 is the maintained integrated app (Firebase UI, Qdrant/S3, optional SageMaker, Terraform for ECS/ALB/ECR).
BK-MInD follows a six-tier Clean Architecture pattern that separates concerns across distinct layers, enabling maintainability, testability, and independent scaling. The system is designed to achieve: (1) multimodal data ingestion from diverse educational materials, (2) asynchronous processing to support concurrent operations without bottlenecks, and (3) production-grade security and scalability on AWS infrastructure.
The following diagram shows the complete system topology aligning with the SRS: multimodal ingest β process β index β retrieve β generate, organized across six architectural tiers plus cross-cutting concerns for auth, security, and persistence.
Layer summary
| Layer | Role | SRS touchpoints |
|---|---|---|
| Client | Uploads, search, summaries, dashboards, auth | FR-021βFR-022, FR-023βFR-027, NFR-005βNFR-008 |
| API | Orchestration, RBAC hooks, integration | TR-001, NFR-003βNFR-004 |
| Processing | ASR, OCR/VLM, spreadsheets, sync, corpus | FR-001βFR-008 |
| Storage | Vectors, sparse index, blobs, metadata | TR-003, TR-004, NFR-004 |
| Retrieval & generation | Hybrid + visual search, RAG, chat, LLM | FR-009βFR-014, TR-004 |
| Deployment | Containers, cloud LB TLS, optional managed GPU | TR-006βTR-007, NFR-002βNFR-003 |
For HTTPS and custom domains on AWS, see docs/technical/DOCS_deployment-alb-acm-custom-domain.md.
The latest deployment architecture (v4) shows production-grade cloud infrastructure on AWS with ECS Fargate, ALB, ElastiCache, vector databases, and auto-scaling:
For capstone presentations / documentation review: open docs/report/ for Phase 2 reports and presentation material.
For development setup: follow docs/requirements.md for the baseline environment. The main pieces are Python 3.9+ for the backend, React 18 + Vite + Tailwind for the frontend, and FFmpeg / Tesseract / Poppler for media and OCR. GPU use is optional locally if you offload heavy inference to APIs or SageMaker (Phase_2/code/sagemaker/README.md).
Additional Diagrams:
docs/diagram/Complete diagram collection including document processing flows and system documentation
Last Updated: July 4, 2026 For: Phase 2 maintained application stack Duration: 20-40 minutes for a local backend/frontend setup
This guide matches the current codebase structure documented in ../README.md, ../Phase_2/README.md, ../Phase_2/code/backend/README.md, and ../Phase_2/code/terraform/README.md.
- Python 3.10+ for the backend
- Node.js 18+ for the frontend
- Git
- Docker and Docker Compose for local Qdrant and Redis
- Optional system binaries: FFmpeg, Tesseract, and Poppler if you run the document-processing paths locally
git clone https://github.com/pdz1804/capstone-project.git
cd capstone-projectThe maintained app lives in Phase_2/.
From the backend folder, start the local service dependencies:
cd Phase_2/code/backend
docker compose up -dThis starts Redis and Qdrant from docker-compose.yml; it does not start the FastAPI backend itself.
Useful checks:
curl http://localhost:6333/health
docker compose psCreate the backend environment file and fill in the values you need:
cd Phase_2/code/backend
cp .env.example .envStart the backend with the maintained entrypoint:
python -m venv .venv
python -m pip install -r requirements.txt
python run_api.pyActivate the virtual environment with the equivalent command for your shell before installing packages if you prefer not to use the system interpreter.
The API runs on port 5001 by default. The runner accepts --host, --port, --workers, --reload, and --no-reload if you need to override the defaults.
Common backend env values to review:
OPENAI_API_KEYGEMINI_API_KEYQDRANT_MODE,QDRANT_HOST,QDRANT_PORT, orQDRANT_URLREDIS_URLFILE_STORAGE_BACKENDUSE_AWS_SAGEMAKER_INFERENCEDYNAMODB_USERS_TABLE,DYNAMODB_APP_USAGE_TABLE,DYNAMODB_JOBS_TABLE
In a separate terminal:
cd Phase_2/code/frontend
cp .env.example .env
npm install
npm run devThe current frontend dev script is tsx server.ts, and the default local URL is http://localhost:5173. The template in ../Phase_2/code/frontend/.env.example uses VITE_API_BASE_URL=/api and API_PROXY_TARGET=http://localhost:5001.
Backend health:
curl http://localhost:5001/health
curl http://localhost:5001/api/healthFrontend should be reachable at http://localhost:5173.
If you want to smoke-test search after indexing data, use the current API contract from ../Phase_2/code/backend/README.md and send requests with the X-User-Id header.
Backend tests:
cd Phase_2/code/backend
python -m pytest tests -vTerraform validation only:
cd Phase_2/code/terraform
terraform init -backend=false
terraform fmt -recursive
terraform validate- If Qdrant is unreachable, confirm
docker compose up -dcompleted and that port 6333 is free. - If Redis is unreachable, confirm the same compose stack is running and that the backend
REDIS_URLpoints atredis://localhost:6379/0or your configured instance. - If the frontend cannot reach the API, confirm
API_PROXY_TARGET=http://localhost:5001and that the backend is running on port 5001. - If
pip installfails on large wheels, clear the pip cache or use a drive with more free space before retrying.
Phase_2/README.mdfor the maintained Phase 2 setup summaryPhase_2/code/backend/README.mdfor backend runtime, API routes, and indexing workflowPhase_2/code/terraform/README.mdfor infrastructure validation and deployment notes
Single tree that combines the production-style FastAPI backend (Qdrant, S3, optional SageMaker inference), the React + Firebase frontend from the FE track, SageMaker hosting packs (unified Docling + Whisper + ColQwen container and optional split endpoints), and Terraform for AWS: ECR, ECS Fargate, Application Load Balancer with optional HTTPS (ACM), auto scaling, and an optional SageMaker endpoint aligned with sagemaker/unified.
| Area | Path | Documentation |
|---|---|---|
| Folder overview | Phase_2/ |
Phase_2/README.md |
| Terraform (ALB, ECS, ECR, SageMaker) | Phase_2/code/terraform/ |
Phase_2/code/terraform/README.md |
| SageMaker build / deploy | Phase_2/code/sagemaker/ |
Phase_2/code/sagemaker/README.md |
| HTTPS + custom domain runbook | docs/technical/DOCS_deployment-alb-acm-custom-domain.md |
ACM validation, DNS, ALB listeners |
Use Phase_2 as the maintained application tree for local development, technical review, deployment, and testing.
π For capstone presentations / documentation review:
Start with docs/report/ folder for Phase 2 reports and presentation guides.
π¨βπ» For development setup:
Prerequisites follow docs/requirements.md (TR-001βTR-005, NFR-005βNFR-006): Python 3.9+, FastAPI backend; React 18+, Vite, Tailwind frontend; FFmpeg, Tesseract, Poppler for media; GPU optional locally if you offload heavy inference to APIs or SageMaker (Phase_2/code/sagemaker/README.md). Docker and Terraform are for packaging and cloud layout (TR-006βTR-007).
git clone https://github.com/pdz1804/capstone-project.git
cd capstone-project
python -m venv venvActivate the virtual environment with the equivalent command for your shell.
Full UI (Firebase), Qdrant/S3-aware API, tests, Terraform and SageMaker docs see Phase_2/README.md.
Backend:
cd Phase_2/code/backend
python -m pip install -r requirements.txt
cp .env.example .env
# Edit .env with keys, Qdrant, S3, and SageMaker settings.
python run_api.pyFrontend:
cd Phase_2/code/frontend
npm install
cp .env.example .env
npm run devURLs (typical): UI http://localhost:5173 (or your Vite default), API http://localhost:5001, docs http://localhost:5001/docs.
Terraform (local validation only, no apply):
cd Phase_2/code/terraform
terraform init -backend=false
terraform fmt -recursive
terraform validatecd Phase_1/code/asr_ocr/multi_model_benchmark/src
python main.py asr --output-dir results/asr <list-of-video-files>
cd Phase_1/code/retrieval/baseline_bm25_dense_hybrid
jupyter notebook manual_bm25_dense_hybrid.ipynb
cd Phase_1/code/processing_rag_pipeline/rag_framework_comparison
python setup_and_run.py
cd Phase_1/code/processing_rag_pipeline/unified_processor
python src/pipeline.py input/ output/
# Optional: add --fast-mode where that script supports itUse cd <repoRoot> first if you are not already at the repository root.
Course: CS252 - Capstone Project Institution: Ho Chi Minh City University of Technology (HCMUT) Focus: Applied AI for Educational Content Processing Domain: Information Retrieval, NLP, Multimodal Learning, RAG Systems
Research Contributions:
- Vietnamese-optimized ASR/OCR pipeline for lecture processing
- Comprehensive retrieval method comparison on MS MARCO
- RAG framework selection guide for educational Q&A
- Production-grade retrieval system implementations
- Multimodal document understanding with Docling
- Dual-mode RAG processing pipeline (text + image retrieval)
- Intelligent document deduplication and caching system
- Performance-quality tradeoff framework (Fast vs Full modes)
π Start Here: docs/requirements.md β Software Requirements Specification: functional, non-functional, technical constraints (37 requirements total).
Authoritative Technical Documents
docs/technical/APPLICATION_OVERVIEW.mdβ Product scope, user workflows, architecture summary, features, quality attributes, and engineering assessment.docs/technical/API_REFERENCE.mdMaintainer-level API reference covering authentication, files, processing, indexing, search, chat, insights, feedback, and operational guidance.docs/technical/DOCS_TECHNICAL_GUARDRAIL_CONFIGURATION.mdAWS Bedrock guardrails configuration, content safety filters, PII protection, implementation details.
Testing and Performance Evidence
docs/report/FRESH_EVALUATION_REPORT_2026_05_07.mdFinal evaluation report with component testing, performance benchmarks, and production readiness assessment.docs/jmeter-capacity-tests/runs/README_MAIN_APIS.mdJMeter runbook and result exports for Process, Index, and Search APIs.docs/jmeter-capacity-tests/runs/README_NON_MAIN_APIS.mdJMeter runbook and result exports for Auth, User, Stats, Upload, Chat, and Insights APIs.
Architecture and Deployment
docs/technical/DOCS_deployment-alb-acm-custom-domain.mdACM certificates, DNS validation, ALB HTTPβHTTPS, and custom domain setup.docs/technical/DOCS_search-cache-redis-setup.mdRedis/ElastiCache search cache setup and operational notes.docs/technical/DOCS_REDIS_ASYNC_JOB_SYSTEM_GUIDE.mdAsync job tracking system (Redis-based), job lifecycle, and monitoring.
Security and WAF Configuration
docs/technical/DOCS_TECHNICAL_WAF_CONFIGURATION.mdAWS WAF rules, IP whitelisting, DDoS protection, and security group configuration.docs/technical/SECURITY_SECTION_CAPSTONE_REPORT.mdSecurity architecture, threat modeling, and compliance considerations.
Cost Estimation
docs/others/AWS_Cost_Estimation_50_Users_Professional.xlsxDetailed cost analysis and scalability projections for 50 concurrent users.
Merged Production Application (Phase_2/)
Phase_2/README.mdTop-level map: frontend, backend, SageMaker pack, Terraform; local quick paths.Phase_2/code/backend/README.mdFastAPI layout, Qdrant/BM25/hybrid/image retrieval, S3 vs local storage.Phase_2/code/terraform/README.mdAWS resources (ECR, ECS, ALB, optional HTTPS, optional SageMaker) and Terraform checks.Phase_2/code/sagemaker/README.mdUnified container, ECR push, deploy/delete scripts, and backend environment variables.
Research Milestones and Utilities
- READMEs inside
Phase_1/code/asr_ocr/,Phase_1/code/retrieval/,Phase_1/code/processing_rag_pipeline/, anddownloads/directories (datasets and paper references). Phase_1/code/processing_rag_pipeline/rag_framework_comparison/DETAILED_PIPELINE_FLOWS.mdβ Detailed RAG pipeline flow diagrams and explanations.
The downloads/ directory contains a curated collection of research papers covering:
- Retrieval-Augmented Generation (RAG) architectures
- Dense retrieval methods (DPR, ColBERT, ANCE)
- Multimodal learning (CLIP, LayoutLM, Docling)
- Speech recognition (Whisper, Wav2Vec 2.0)
- OCR and document understanding
This is an academic capstone project. For collaboration or questions:
- Repository: github.com/pdz1804/capstone-project
- Issues: Use GitHub Issues for bug reports or feature requests
- Contact: See individual weekly READMEs for team member information
This project is licensed under the MIT License - see the LICENSE file for details.
Copyright (c) 2025 Quang Phu, Ngoc Khoi, and Minh Khoi
Open-source models, APIs, and platforms that this codebase builds on (see also TR-004βTR-005 and integration notes in docs/requirements.md):
- OpenAI Whisper and LLM APIs used in ASR and generation experiments.
- Google Gemini (multimodal/API), Firebase (authentication in the merged frontend stack), and embedding-related tooling referenced in weekly work.
- Hugging Face
transformers, model hubs, and pretrained checkpoints (e.g. ColQwen, sentence encoders). - IBM Docling and related document-understanding components.
- Qdrant Vector Database used in the Phase 2 AI service and merge backend.
- Amazon Web Services S3, SageMaker real-time inference, and (via Terraform) ECS, ECR, ALB, ACM for optional cloud deployment.
- HashiCorp Terraform for infrastructure as code in
Phase_2/code/terraform/. - Pyserini / Anserini & Milvus retrieval stacks explored in research-week milestones.
- LangChain & LlamaIndex RAG framework comparisons (early-phase notebooks and prototypes).
- FFmpeg, Tesseract, Poppler media, OCR, and PDF tooling (TR-005).
- React, Vite, Tailwind CSS frontend stack (TR-002).
Version: 1.0 Last Updated: May 10, 2026
Team: MKhoi, NKhoi, QPhu.

