This repository comprises three main areas:
data_miner/: Scripts to fetch external Debian data sources (snapshot dumps, buildinfo files, tracker pages, source repositories).generate_db/: Scripts to ingest the downloaded artifacts into local SQLite databases.analysis/: Analysis scripts and reports built on top of the ingested databases.
-
Download data: Browse
data_miner/and run whichever data mining scripts you need. Each one stores raw artifacts in its own subdirectory. -
Generate databases: Run the corresponding scripts under
generate_db/to ingest the downloaded data into SQLite databases at the repository root (publish.db,buildinfo.db,source_repos.db). -
Analyze traceability: Run
generate_paths.pyto compute end-to-end package paths from publish → buildinfo → source repositories. Outputs statistics on what is fully traceable and what gaps exist. -
Further analysis: See
analysis/for additional research on the ingested data and traceability patterns.
- Run all commands from the repository root.
- Shared path configuration is in
project_paths.py.