🐛 Publish nested documentation guides

README.md

@@ -21,7 +21,8 @@ SPDX-License-Identifier: Apache-2.0
 `continuous-delivery-scripts` provides Git-based CI/CD commands for release
 automation, semantic versioning, changelog generation, SPDX SBOMs,
 third-party IP (TPIP) and licence reports, source licence headers and secret
-checks.
+checks. It is designed as a CI-agnostic set of composable delivery tools rather
+than a Git-history-driven release system.
 
 ## Overview
 
@@ -34,7 +35,9 @@ a release. The shared project delivery definition lives in one place:
 including paths, versioning rules and the selected `PROGRAMMING_LANGUAGE`,
 regardless of the project's language or CI system. The selected plugin then
 uses appropriate ecosystem tools: for example, GoReleaser for Go releases or
-wheel and Twine for Python packages. Consult the
+wheel and Twine for Python packages. CDS also builds on tools such as
+Towncrier, pdoc, detect-secrets and SPDX data sources where they are the best
+fit for a specific delivery task. Consult the
 [plugin guides](./guides/plugin-guides.md) for language-specific
 requirements, including the [Go plugin guides](./guides/plugin-golang.md);
 native build manifests and credentials may still be needed. See
@@ -46,6 +49,30 @@ the project's Git repository and install the required tools. This avoids
 infrastructure lock-in: the team can decide which CI system suits its project,
 including after the delivery workflow has been defined.
 
+## Why Continuous Delivery Scripts?
+
+CDS takes a different approach to release automation by deliberately separating
+development history from consumer-facing release information.
+
+Unlike commit-driven tools such as
+[release-please](https://github.com/googleapis/release-please) or
+[semantic-release](https://semantic-release.gitbook.io/), CDS follows
+Towncrier's philosophy: Git commits document technical changes for maintainers,
+while explicit news fragments describe their significance to consumers.
+
+Beyond changelog generation, CDS provides composable, CI-agnostic tools for
+versioning, coordinated releases, licensing, SBOM generation, documentation and
+other delivery responsibilities. CDS often uses specialised tools rather than
+replacing them outright: for example, Towncrier for release notes, GoReleaser
+for Go packaging, wheel and Twine for Python publication, pdoc for Python API
+documentation, and detect-secrets for secret scanning.
+
+This makes it particularly suitable for projects seeking consistent release
+practices across languages and CI/CD platforms without imposing a particular
+commit convention or release workflow.
+
+Learn more in [Why choose Continuous Delivery Scripts?](./guides/why-continuous-delivery-scripts.md).
+
 ## Usage and documentation
 
 Start with the [common use cases](#common-use-cases) below, then follow the
@@ -184,11 +211,14 @@ same commands whichever CI system the team chooses.
 | [`cd-determine-version`](./guides/previewing-versions.md) | Calculates and prints the **project's prospective release version** without generating a changelog. | Required `--release-type` (`-t`) | Developer preview or CI |
 | [`cd-generate-news`](./guides/managing-changelogs.md) | Updates the project version and, for beta or production releases, builds the changelog from news fragments; prints the resulting version. | Required `--release-type` (`-t`) | CI release flow |
 | [`cd-get-config`](./guides/reading-project-configuration.md) | Prints a project configuration value. | Either `--key` (`-k`) or `--config-variable` (`-c`) | Developer or CI |
+| [`cd-check-project-config`](./guides/checking-project-configuration.md) | Validates CDS-specific `ProjectConfig` values before other commands run. | Optional `--config-file` (`-f`), `--verbose` (`-v`) | Developer setup or CI gate |
 | [`cd-tag-and-release`](./guides/automating-releases.md) | Runs the release flow, including documentation, licensing summaries, Git tagging, packaging and publication through the selected [language plugin](./continuous_delivery_scripts/plugins). | Required `--release-type` (`-t`); optional `--current-branch` (`-b`) | CI release flow |
 | [`cd-generate-docs`](./guides/generating-code-documentation.md) | Generates code documentation in the configured output directory (or one specified on the command line). | `--output_directory` | Developer preview or CI |
 | [`cd-generate-spdx`](./guides/generating-an-spdx-sbom.md) | Generates SPDX documents and third-party licence summaries, then checks licence compliance when project metadata is available. | Required `--output-dir` (`-o`); optional `--lookup-scancode`, `--skip-dependency-download` | CI audit; developer review also possible |
 | [`cd-check-licence-compliance`](./guides/checking-licence-compliance.md) | Checks project and dependency licences without creating SPDX documents; optionally writes third-party IP summaries. | Optional `--output-dir` (`-o`) for reports, `--lookup-scancode`, `--skip-dependency-download` | CI gate or developer review |
 | [`cd-license-files`](./guides/licence-header-management.md) | Adds or updates source-file licence and copyright headers when the language plugin supports them. | `--verbose` (`-v`) | Developer or CI |
+| [`cd-sync-project-licence`](./guides/syncing-project-licence.md) | Creates or refreshes the top-level project licence file from the configured SPDX identifier. | `--verbose` (`-v`) | Developer setup or CI compliance prep |
+| [`cd-describe-licence-obligations`](./guides/describing-project-licence-obligations.md) | Prints a human-readable summary of obligations for the configured project licence. | `--verbose` (`-v`) | Developer review or CI reporting |
 | [`cd-record-secrets`](./guides/recording-secrets.md) | Records accepted findings in the project's [detect-secrets](https://github.com/Yelp/detect-secrets) registry. | `--registry-file` (`-r`); defaults to the configured registry | Developer (registry maintenance) |
 | [`cd-detect-secrets`](./guides/checking-for-secrets.md) | Checks Git-tracked files against that registry and fails if new secrets are found. | `--registry-file` (`-r`); defaults to the configured registry | CI; developer checks also possible |
 

continuous_delivery_scripts/generate_docs.py

@@ -12,6 +12,7 @@
 import shutil
 import sys
 from pathlib import Path
+from urllib.parse import SplitResult, urlsplit
 
 import markdown
 
@@ -28,9 +29,50 @@
 logger = logging.getLogger(__name__)
 
 
-def _html_guide_link(match: re.Match[str]) -> str:
-    """Link to the rendered HTML page instead of its Markdown source."""
-    return f'href="{match.group(1)}.html{match.group(2) or ""}"'
+def _guide_html_path(source: Path, guide_sources: Path) -> Path:
+    """Map a guide source file to its published HTML path beneath the guide output."""
+    return source.relative_to(guide_sources).with_suffix(".html")
+
+
+def _guide_sort_key(source: Path, guide_sources: Path) -> tuple[bool, str]:
+    """Keep the top-level index first and sort the rest by relative path."""
+    relative = source.relative_to(guide_sources)
+    return relative != Path("index.md"), relative.as_posix()
+
+
+def _rewrite_guide_link_target(parts: SplitResult, source: Path, guide_sources: Path) -> str | None:
+    """Rewrite only relative Markdown links that point to published guides."""
+    if (
+        parts.scheme
+        or parts.netloc
+        or not parts.path
+        or parts.path.startswith("/")
+        or not parts.path.lower().endswith(".md")
+    ):
+        return None
+    target = (source.parent / Path(parts.path)).resolve()
+    root = guide_sources.resolve()
+    if not target.is_file() or not target.is_relative_to(root):
+        return None
+    return parts._replace(path=f"{parts.path[:-3]}.html").geturl()
+
+
+def _rewrite_rendered_guide_links(body: str, source: Path, guide_sources: Path) -> str:
+    """Retarget rendered guide links to the published HTML pages."""
+
+    def replace(match: re.Match[str]) -> str:
+        href = match.group(1)
+        rewritten = _rewrite_guide_link_target(urlsplit(href), source, guide_sources)
+        if rewritten is None:
+            return match.group(0)
+        return f'href="{escape(rewritten, quote=True)}"'
+
+    return re.sub(r'href="([^"]+)"', replace, body)
+
+
+def _guide_navigation_prefix(guide_folder: Path, relative_html_path: Path) -> str:
+    """Build the route from a guide page back to the documentation root."""
+    return "/".join(".." for _ in (guide_folder / relative_html_path.parent).parts)
 
 
 def _html_page(
@@ -127,15 +169,32 @@ def _guide_settings() -> tuple[Path, Path, Path, str] | None:
 
 
 def _render_guide(
-    source: Path, destination: Path, project_name: str, nav_prefix: str, has_api_index: bool, page_url: str | None
+    source: Path,
+    destination: Path,
+    guide_sources: Path,
+    guide_folder: Path,
+    project_name: str,
+    has_api_index: bool,
+    page_url: str | None,
 ) -> str:
     """Render one Markdown guide to an HTML page and return its title."""
     text = source.read_text(encoding="utf8")
     title = next((line[2:].strip() for line in text.splitlines() if line.startswith("# ")), source.stem)
     body: str = markdown.markdown(text, extensions=["fenced_code", "tables"])
-    body = re.sub(r'href="([a-z0-9-]+)\.md(#[^"]*)?"', _html_guide_link, body)
+    body = _rewrite_rendered_guide_links(body, source, guide_sources)
     description = f"{title} — {project_name}."
-    destination.write_text(_html_page(title, description, body, nav_prefix, has_api_index, page_url), encoding="utf8")
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    destination.write_text(
+        _html_page(
+            title,
+            description,
+            body,
+            _guide_navigation_prefix(guide_folder, _guide_html_path(source, guide_sources)),
+            has_api_index,
+            page_url,
+        ),
+        encoding="utf8",
+    )
     return title
 
 
@@ -144,16 +203,22 @@ def _render_guides(
 ) -> list[tuple[str, str]]:
     """Render all guides and return links for the documentation landing page."""
     guide_output.mkdir(parents=True, exist_ok=True)
-    navigation_prefix = "/".join(".." for _ in guide_folder.parts)
     site_url = documentation_site_url()
     links = []
-    for source in sorted(guide_sources.glob("*.md")):
-        page_url = f"{site_url}{guide_folder.as_posix()}/{source.stem}.html" if site_url else None
+    for source in sorted(guide_sources.rglob("*.md"), key=lambda path: _guide_sort_key(path, guide_sources)):
+        relative_html_path = _guide_html_path(source, guide_sources)
+        page_url = f"{site_url}{(guide_folder / relative_html_path).as_posix()}" if site_url else None
         title = _render_guide(
-            source, guide_output / f"{source.stem}.html", project_name, navigation_prefix, has_api_index, page_url
+            source,
+            guide_output / relative_html_path,
+            guide_sources,
+            guide_folder,
+            project_name,
+            has_api_index,
+            page_url,
         )
-        if source.stem != "index":
-            links.append((title, source.stem))
+        if source.relative_to(guide_sources) != Path("index.md"):
+            links.append((title, relative_html_path.as_posix()))
     return links
 
 

continuous_delivery_scripts/templates/documentation_landing.html.jinja2

@@ -3,5 +3,5 @@
 {% if licence %}<p>Project licence: {{ licence }}</p>{% endif %}
 <p><a href="{{ guide_folder }}/index.html">All task guides</a>{% if has_api_index %} · <a href="api.html">API reference</a>{% endif %}</p>
 <ul>
-{% for title, filename in links %}<li><a href="{{ guide_folder }}/{{ filename }}.html">{{ title }}</a></li>
+{% for title, href in links %}<li><a href="{{ guide_folder }}/{{ href }}">{{ title }}</a></li>
 {% endfor %}</ul>

continuous_delivery_scripts/templates/llms.txt.jinja2

@@ -5,8 +5,8 @@
 ## Documentation
 
 - [Overview]({{ site_url }})
-{% for source, title in guides if folder -%}
-- [{{ title }}]({{ site_url }}{{ folder.as_posix() }}/{{ source.stem }}.html)
+{% for source, title, href in guides if folder -%}
+- [{{ title }}]({{ site_url }}{{ folder.as_posix() }}/{{ href }})
 {% endfor -%}
 {% if has_api -%}
 - [API reference]({{ site_url }}api.html)

continuous_delivery_scripts/utils/documentation_discovery.py

@@ -48,14 +48,16 @@ def structured_data_script(title: str, description: str, url: str, schema_type:
     return f'<script type="application/ld+json">{encoded}</script>'
 
 
-def _annotate_existing_pages(output: Path, site_url: str, guides: list[tuple[Path, str]], folder: Path | None) -> None:
+def _annotate_existing_pages(
+    output: Path, site_url: str, guides: list[tuple[Path, str, str]], folder: Path | None
+) -> None:
     """Annotate already-rendered landing and guide pages without rebuilding the API."""
     if not guides or folder is None:
         return
     pages = [(output / "index.html", site_url, "WebSite")]
     pages.extend(
-        (output / folder / f"{source.stem}.html", f"{site_url}{folder.as_posix()}/{source.stem}.html", "TechArticle")
-        for source, _ in guides
+        (output / folder / relative_html_path, f"{site_url}{folder.as_posix()}/{relative_html_path}", "TechArticle")
+        for _source, _title, relative_html_path in guides
     )
     for page, url, schema_type in pages:
         text = page.read_text(encoding="utf8")
@@ -75,13 +77,26 @@ def _guide_title(source: Path) -> str:
     )
 
 
-def _published_guides(output: Path, sources: Path | None, folder: Path | None) -> list[tuple[Path, str]]:
+def _guide_html_path(source: Path, sources: Path) -> Path:
+    """Map a guide source file to its published HTML path beneath the guide folder."""
+    return source.relative_to(sources).with_suffix(".html")
+
+
+def _guide_sort_key(source: Path, sources: Path) -> tuple[bool, str]:
+    """Keep the guide overview first and sort all nested guides by path."""
+    relative = source.relative_to(sources)
+    return relative != Path("index.md"), relative.as_posix()
+
+
+def _published_guides(output: Path, sources: Path | None, folder: Path | None) -> list[tuple[Path, str, str]]:
     if sources is None or folder is None:
         return []
+    guide_sources = sources
     return [
-        (source, _guide_title(source))
-        for source in sorted(sources.glob("*.md"), key=lambda path: (path.name != "index.md", path.name))
-        if (output / folder / f"{source.stem}.html").is_file()
+        (source, _guide_title(source), relative_html_path.as_posix())
+        for source in sorted(guide_sources.rglob("*.md"), key=lambda path: _guide_sort_key(path, guide_sources))
+        for relative_html_path in [_guide_html_path(source, guide_sources)]
+        if (output / folder / relative_html_path).is_file()
     ]
 
 
@@ -100,7 +115,7 @@ def _render_sitemap(output: Path, site_url: str) -> str:
 
 
 def _render_llms_index(
-    output: Path, site_url: str, name: str, guides: list[tuple[Path, str]], folder: Path | None
+    output: Path, site_url: str, name: str, guides: list[tuple[Path, str, str]], folder: Path | None
 ) -> str:
     """Render the concise index from pages that were actually published."""
     rendered = str(
@@ -111,19 +126,24 @@ def _render_llms_index(
     return rendered.rstrip() + "\n"
 
 
-def _render_llms_full(output: Path, site_url: str, name: str, guides: list[tuple[Path, str]], folder: Path) -> str:
+def _render_llms_full(output: Path, site_url: str, name: str, guides: list[tuple[Path, str, str]], folder: Path) -> str:
     """Consolidate only the Markdown guides which have corresponding published pages."""
-    published = {source.stem for source, _ in guides}
-
-    def published_link(match: re.Match[str]) -> str:
-        stem, fragment = match.group(1), match.group(2) or ""
-        return f"]({site_url}{folder.as_posix()}/{stem}.html{fragment})" if stem in published else match.group(0)
+    published = {source.resolve(): relative_html_path for source, _title, relative_html_path in guides}
 
     sections: list[tuple[str, str]] = []
-    for source, title in guides:
+    for source, title, _relative_html_path in guides:
         text = source.read_text(encoding="utf8")
         text = re.sub(r"^# .*(?:\n|$)", "", text, count=1, flags=re.MULTILINE)
-        text = re.sub(r"\]\(([a-z0-9-]+)\.md(#[^)]*)?\)", published_link, text)
+
+        def published_link(match: re.Match[str]) -> str:
+            destination = match.group(1)
+            fragment = match.group(2) or ""
+            target = (source.parent / Path(destination)).resolve()
+            if target not in published:
+                return match.group(0)
+            return f"]({site_url}{folder.as_posix()}/{published[target]}{fragment})"
+
+        text = re.sub(r"\]\(([^)#]+\.md)(#[^)]*)?\)", published_link, text)
         in_code = False
         body = []
         for line in text.strip().splitlines():

guides/index.md

@@ -10,6 +10,10 @@ the implementation, while these pages focus on inputs, outputs and examples.
 For language-specific packaging, credentials and metadata support, start with
 the [plugin guide index](plugin-guides.md).
 
+## Choosing CDS
+
+- [Why choose Continuous Delivery Scripts?](why-continuous-delivery-scripts.md): Understand CDS design goals, the specialised tools it uses, and when another approach may fit better.
+
 ## Dependency licences and OpenChain workflows
 
 - [Assess dependency licence risks](assessing-dependency-licences.md): Review directional, explainable licence screening and optional project policy overrides.