Python wrapper for UDLF, the Unsupervised Distance Learning Framework.
Authors: Gustavo Rosseto Leticio, Lucas Pascotti Valem and Daniel Carlos Guimarães Pedronette
Dept. of Statistic, Applied Math. and Computing, Universidade Estadual Paulista (UNESP), Rio Claro, Brazil
- Overview
- Installation
- Quickstart
- Examples
- Usage
- Running your own config file
- Legacy API
- Command line
- File formats
- Development
- Applications
- Cite
- Contact
- Acknowledgments
- License
UDLF is a C++ framework of unsupervised distance learning methods for image and multimedia retrieval: given the ranked lists (or a distance / similarity matrix) of a dataset, a method re-ranks them using the contextual information they contain, with no labels. pyUDLF runs these methods from Python: you pass numpy arrays (or files) and get back the re-ranked lists and the evaluation measures (MAP, Precision@k, Recall@k) as numpy arrays and numbers. The configuration files, input files and the UDLF binary are handled for you. The same API also runs Sparse UDLF, which implements CPRR, LHRR, RDPAC, RFE and BFSTREE with sparse matrices for large datasets.
Methods (UDLF 1.70), with the names and papers listed by UDLF:
| Name in pyUDLF | Method (paper) |
|---|---|
CPRR |
CPRR |
LHRR |
LHRR |
RDPAC |
RDPAC |
RFE |
RFE |
BFSTREE |
BFSTree |
CONTEXTRR |
ContextRR |
CORGRAPH |
Correlation Graph |
RKGRAPH |
Rk Graph Dist. |
RECKNNGRAPH |
ReckNNGraph |
RLRECOM |
RL-Recom |
RLSIM |
RL-Sim* |
NONE |
no re-ranking (evaluates the input) |
Requires Python 3.9+.
pip install git+https://github.com/UDLF/pyUDLF.git # core
pip install "pyUDLF[viz] @ git+https://github.com/UDLF/pyUDLF.git" # + ranked list images (Pillow)Then install the UDLF binary (official release, checked with its SHA-256):
pyudlf downloadIt goes to ~/.pyudlf/bin (or $PYUDLF_HOME/bin); pyudlf where shows the binary in use.
There are official builds for Linux and Windows. On other systems (e.g. macOS), build UDLF from
source (make in the UDLF repository, or scripts/build_udlf.sh
here) and use it with UDLF(bin_path=...) or the UDLF_BIN environment variable.
import numpy as np
from pyUDLF import UDLF
# A small dataset: 200 elements in 10 classes, with noisy 32-D features
rng = np.random.default_rng(7)
labels = np.repeat(np.arange(10), 20)
features = rng.normal(size=(10, 32))[labels] + rng.normal(scale=1.4, size=(200, 32))
# Ranked lists: each row is a query, with all elements sorted by distance
distances = np.linalg.norm(features[:, None] - features[None], axis=2)
ranked_lists = np.argsort(distances, axis=1, kind="stable")
# Re-rank them with CPRR (needs the UDLF binary: run `pyudlf download` once)
udlf = UDLF()
res = udlf.run("CPRR", ranked_lists=ranked_lists, params={"l": 100, "k": 10},
classes=labels)
print(res) # Result(method='CPRR', ranked_lists=(200, 100), MAP 0.5471 -> 0.7602)
print(res.metrics["before"]["P@10"], "->", res.metrics["after"]["P@10"]) # 0.631 -> 0.8315
before, after = res.per_query() # average precision of each query
print((after > before).sum(), "of 200 queries improved") # 186 of 200 queries improvedThis script is also in examples/quickstart.py.
Runnable scripts in examples/, on the synthetic dataset of the repository (python examples/<script>.py). The tests run all of them.
| Script | What it shows |
|---|---|
| compare_methods.py | all 12 methods on the same input, ranked by MAP |
| parameter_search.py | grid search over k and l |
| input_formats.py | ranked lists (numpy, file, names) and distance / similarity matrices |
| fusion.py | fusion of two descriptors |
| per_query_analysis.py | queries that improved or got worse |
| your_own_data.py | command line for your own files |
| sparse_methods.py | UDLF vs Sparse UDLF, method by method |
| legacy_api.py | the API of previous versions |
The examples below continue the quickstart.
# Ranked lists: one row per query with element indices (numpy array, list of lists or file)
res = udlf.run("LHRR", ranked_lists=ranked_lists, params={"l": 100}, classes=labels)
res = udlf.run("LHRR", ranked_lists="ranked_lists.txt", params={"l": 100}, classes=labels)
# Distance or similarity matrix (N x N)
res = udlf.run("RDPAC", matrix=distances, matrix_type="dist", params={"l": 50}, classes=labels)
# Files in the UDLF formats: ranked lists, element names ("lists") and "name:class" classes
res = udlf.run("CPRR", ranked_lists="ranked_lists.txt", names="lists.txt",
classes="classes.txt", params={"l": 100})Input files are used in place; arrays are written to a temporary directory, removed after the run.
Without classes, the evaluation is skipped (and a warning is logged).
params takes the parameters of the method by name; the others keep the default values of the
UDLF reference configuration. List them with UDLF.describe:
print(UDLF.describe("CPRR"))
# CPRR parameters (params={name: value}):
# l = 400 (int): Size of ranked lists (must be lesser than SIZE_DATASET)
# k = 20 (int): Number of nearest neighbors
# t = 2 (int): Number of iterationsUDLF.methods() lists the methods. Note that the default l of several methods (e.g. 400 or
1400) assumes larger datasets: it must not be larger than the number of elements, and the input
ranked lists must have at least l elements.
res.ranked_lists # re-ranked lists: numpy array (queries x output size)
res.metrics # {"before": {...}, "after": {...}, "gain": {...}}: MAP, P@k, Recall@k
res.map_before, res.map_after
res.time # execution time of the method, in seconds (measured by UDLF)
res.per_query("MAP") # value of each query, before and after (also "precision", "recall")
res.config, res.stdout # configuration used and UDLF output, for reproducibilityThe measures are computed by UDLF up to the size of the output ranked lists (MAP@L); gain is
the relative gain in percent. Other options:
# Output as a distance (or similarity) matrix instead of ranked lists
res = udlf.run("CPRR", ranked_lists=ranked_lists, params={"l": 100}, output="matrix")
res.matrix.shape # (200, 200)
# Choose the P@k / Recall@k values and keep the files of the run
res = udlf.run("CPRR", ranked_lists=ranked_lists, params={"l": 100}, classes=labels,
precisions=[5, 10], recalls=[10], keep_files=True)
res.files # paths of the config, log, inputs and output of the run# Fuse ranked lists of the same dataset obtained with different features / distances
unit = features / np.linalg.norm(features, axis=1, keepdims=True)
ranked_lists_cosine = np.argsort(-(unit @ unit.T), axis=1, kind="stable")
res = udlf.fuse("CPRR", [ranked_lists, ranked_lists_cosine], params={"l": 100}, classes=labels)
res.map_after # FUSION results only have metrics after fusionSparse UDLF implements CPRR, LHRR, RDPAC, RFE and
BFSTREE with sparse matrices, for datasets where the dense N x N matrices of UDLF do not fit in
memory. On the same input it returns the same ranked lists as UDLF. It has no official build:
scripts/build_sparse.sh compiles its two binaries (udlf and udlf_strict; pyUDLF picks the
right one for each method). Pass their directory with bin_path= or the UDLF_SPARSE_BIN
environment variable.
sparse = UDLF(backend="sparse") # binaries from UDLF_SPARSE_BIN
res = sparse.run("CPRR", ranked_lists=ranked_lists, params={"l": 100, "k": 10},
classes=labels)
print(res) # Result(method='CPRR_SPARSE', ranked_lists=(200, 100), MAP 0.5471 -> 0.7602)
res.raw_metrics # measures written by Sparse UDLF itselfThe result is the same Result as with UDLF, and metrics are computed by pyUDLF with the UDLF
definitions. Methods can be named as in UDLF ("BFSTREE") or as in Sparse UDLF
("BFSTree", "CPRR_SPARSE"). Sparse UDLF only takes ranked lists of element indices (no
matrix, names or fusion). It adds three parameters: threads for LHRR (default: the
threads of UDLF()), and b and memory_budget_mb for BFSTREE. In the pinned version
(commit de3c14c), RFE does not run (its flags are dropped by the config parser), RFE with
perform_ccs=True is not supported, and the matrix output is all zeros, so output="matrix" is
refused.
Problems are reported as exceptions, before running UDLF whenever possible:
from pyUDLF import UDLFParamError
try:
udlf.run("CPRR", ranked_lists=ranked_lists, params={"l": 500})
except UDLFParamError as e:
print(e) # PARAM_CPRR_L = 500 is larger than the dataset (N = 200). Pass a smaller value in params.UDLFParamError (invalid method, parameter or input), UDLFRunError (UDLF failed: has
.problems, .output and .config) and UDLFNotFoundError (no binary) all derive from
UDLFError. Warnings and errors are logged with the standard logging module; call
pyUDLF.enable_logging("INFO") to also see informative messages.
A UDLF configuration file runs as it is. Relative paths are relative to the current directory:
res = udlf.run_config("config.ini")
res.ranked_lists, res.metrics, res.filesThe API of previous versions (InputType and run_calls) is still available and gives the same
results. The wiki documents it.
from pyUDLF import run_calls
from pyUDLF.utils.inputType import InputType
inp = InputType() # starts from the UDLF reference configuration
inp.set_method_name("CPRR")
inp.set_input_file("ranked_lists.txt")
inp.set_lists_file("lists.txt")
inp.set_classes_file("classes.txt")
inp.set_dataset_size(200)
inp.set_method_parameters("CPRR", l=100, k=10)
inp.set_output_file_format("RK")
inp.set_output_rk_format("NUM")
inp.set_output_file_path("output") # writes output.txt
inp.set_output_log_file_path("log.txt")
output = run_calls.run(inp, get_output=True)
print(output.log_dict["MAP"]) # {'Before': 0.5471, 'After': 0.7602, 'Gain': 38.9512}See the CHANGELOG for the behavior changes of this version.
pyudlf download # install the official UDLF binary (--version, --dest, --force)
pyudlf where # binary used by pyUDLF
pyudlf methods # available methods
pyudlf describe CPRR # parameters of a methodThese are the formats UDLF reads (see the UDLF wiki):
- Ranked lists: one line per query, elements separated by spaces, either as indices
(
0 12 25 ...) or as names (img_0001.jpg img_0013.jpg ..., then passnames=). Each line must have at leastlelements; the query is usually the first one. - Matrix: N lines with N values (distances or similarities) separated by spaces.
- Lists (
names=): one element name per line, in the order of the dataset. - Classes (
classes=): onename:classper line.
UDLF cannot read paths with spaces or # in its configuration files, so pyUDLF rejects them.
git clone https://github.com/UDLF/pyUDLF.git && cd pyUDLF
python3 -m venv .venv && .venv/bin/pip install -e ".[viz,test]"
scripts/build_udlf.sh # builds UDLF 1.70 from source into .udlf/ (used by the tests)
.venv/bin/python -m pytestThe tests compare every pyUDLF run with UDLF executed directly on the same configuration, using
the synthetic dataset in tests/data/synthetic. GitHub Actions runs them on every push and pull
request (.github/workflows/tests.yml): Linux with Python 3.9 to 3.13, the built package, and
Windows / macOS.
The methods available in the pyUDLF framework have been utilized in a variety of multimedia and machine-learning scenarios. Among them:
- Image and Video Retrieval
- Person Re-ID
- Weakly Supervised Classification
- Clustering
If you use this software, please cite
@inproceedings{pyUDLF,
author = {Gustavo Leticio and Lucas Pascotti Valem and Leonardo Tadeu Lopes and Daniel Carlos Guimar\~{a}es Pedronette},
title = {PyUDLF: A Python Framework for Unsupervised Distance Learning Tasks},
year = {2023},
isbn = {9798400701085},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3581783.3613466},
doi = {10.1145/3581783.3613466},
booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},
pages = {9680–9684},
numpages = {5},
series = {MM '23}
}Gustavo Rosseto Leticio: gustavo.leticio@gmail.com or gustavo.leticio@unesp.br
Lucas Pascotti Valem: lucaspascottivalem@gmail.com or lucas.valem@unesp.br
Daniel Carlos Guimarães Pedronette: daniel.pedronette@unesp.br
The authors are grateful to São Paulo Research Foundation – FAPESP (grants 2013/08645-0, 2014/04220-8, 2018/15597-6) and Brazilian National Council for Scientific and Technological Development – CNPq (grants 309439/2020-5 and 422667/2021-8).
This project is licensed under GPLv2. See details.