Resource Backend

A high-performance Go-based resource distribution backend service for managing versioned software releases with support for both full and incremental updates, multi-region CDN distribution, and flexible version comparison strategies.

Features

Versioned Resource Management - Manage software resources with semantic versioning or datetime-based versioning
Full & Incremental Updates - Support both complete package distribution and delta updates
Multi-Region CDN - Weighted region-based distribution with configurable CDN/direct download ratios
Release Channels - Separate stable/beta/alpha channels for gradual rollouts
Platform Support - Multi-platform (OS/Arch) package management
Caching Layer - Ristretto in-memory cache with singleflight pattern for high performance
Async Task Queue - Asynq-based background job processing
Health Monitoring - Prometheus metrics and health check endpoints
Service Discovery - Optional Consul integration for cluster mode

Tech Stack

Framework: Fiber v2 (high-performance HTTP framework)
Database: MySQL with Ent ORM (Facebook's entity framework)
Cache: Redis + Ristretto (in-memory)
Task Queue: Asynq (Redis-backed)
DI: Google Wire (compile-time dependency injection)
Logging: Uber Zap
JSON: Sonic (ByteDance's high-performance JSON library)
Version Parsing: SemVer, DateTime formats (pluggable parser system)

Prerequisites

Go 1.25.3+
MySQL 8.0+
Redis 6.0+
(Optional) Consul for service discovery

Quick Start

1. Clone the Repository

git clone https://github.com/MirrorChyan/resource-backend.git
cd resource-backend

2. Configure the Application

Edit config/config.yaml:

instance:
  port: 8000
  only_local: true  # Set to false for cluster mode

database:
  host: "localhost"
  port: 3306
  username: "root"
  password: "your_password"
  name: "resource_db"

redis:
  addr: "localhost:6379"
  db: 0
  asynq_db: 1

For cluster mode, set environment variables:

export INSTANCE_IP="your_instance_ip"
export SERVICE_ID="unique_service_id"
export REGION_ID="region_name"  # Optional, defaults to "default"

3. Generate Code

# Generate Ent ORM code
make entgen

# Generate Wire dependency injection code
make wiregen

4. Build and Run

# Build binary
make build

# Or run directly
go run .

The service will start on http://localhost:8000

API Endpoints

Public Endpoints

Get Latest Version

GET /resources/:rid/latest?channel=stable&system=windows&arch=amd64&current=1.0.0

Query Parameters:

channel - Release channel: stable, beta, alpha
system - Operating system: windows, linux, darwin, etc.
arch - Architecture: amd64, arm64, 386, etc.
current - Current version (optional, for incremental updates)
cdk - CDK token (optional, for authentication)

Response:

{
  "code": 0,
  "data": {
    "resource_id": "my-app",
    "version_name": "1.2.0",
    "version_number": 120,
    "channel": "stable",
    "update_type": "incremental",
    "download_url": "https://cdn.example.com/...",
    "release_note": "What's new in 1.2.0...",
    "file_size": 1048576,
    "file_hash": "sha256:abc123..."
  }
}

Download Resource

GET /resources/download/:key

Redirects to the actual download URL (CDN or direct).

Head Download Info

HEAD /resources/download/:key

Returns file metadata headers:

Content-Length
Content-Type

Admin Endpoints (Require Authentication)

Create Resource

POST /admin/resources

{
  "id": "my-app",
  "name": "My Application",
  "description": "Description here",
  "update_type": "incremental"
}

Platform Update Types

GET    /admin/resources/:rid/platforms
PUT    /admin/resources/:rid/platforms   {"os": "android", "arch": "arm64", "update_type": "full"}
DELETE /admin/resources/:rid/platforms?os=android&arch=arm64

A platform with its own update type uses it; any other platform uses the resource's update_type. For a platform with neither a prior upload nor its own setting, the first upload automatically selects incremental updates for .zip and .tar.gz (case-insensitive), and full updates for all other filenames (e.g. .exe, .dmg, .apk). Only ZIP and TAR.GZ currently support incremental processing; .7z, .tar, and .tgz are delivered as full packages. The upload callback saves this selection as a platform setting after file-type validation; requesting an upload URL alone does not save it. The setting is saved before the processing task is queued and stays if queuing fails: a retried callback of the same package goes through, a package that needs the other mode is rejected. Later uploads keep the selected mode, and admins can override it as usual. Platforms with prior uploads keep their existing behavior, including uploads in another channel or packages that have been purged. Queued uploads retain the processing mode accepted by their callback. Changing a platform to incremental updates does not retroactively unpack full-mode uploads; version pairs without file hashes fall back to full downloads. GET lists the platforms packages were uploaded for and the ones with their own update type, each with the type in effect and its source (inherit / override). PUT and DELETE return the platform's update type before and after the change; DELETE returns the platform to the resource's. os / arch take the same aliases as the version endpoints (e.g. x64 → amd64).

Create Version

POST /resources/:rid/versions
Authorization: Bearer <token>
Content-Type: multipart/form-data

version_name=1.0.0
channel=stable
system=windows
arch=amd64
file=@package.zip
old_version=0.9.0  # For incremental updates

Update Release Note

PUT /resources/:rid/versions/release-note
Authorization: Bearer <token>

{
  "version_name": "1.0.0",
  "channel": "stable",
  "content": "Release notes..."
}

Update Custom Data

PUT /resources/:rid/versions/custom-data
Authorization: Bearer <token>

{
  "version_name": "1.0.0",
  "channel": "stable",
  "content": "{\"custom\": \"json data\"}"
}

Health Check

GET /health

Metrics

GET /metrics

Architecture

Layer Structure

main.go
  ↓
internal/application/app.go (Adapter pattern)
  ↓
internal/interfaces/rest (HTTP handlers)
  ↓
internal/logic (Business logic)
  ↓
internal/repo (Data access)
  ↓
internal/ent (ORM entities)

Database Schema

Resource (Top-level entity)

id - Resource identifier
name - Display name
description - Description
update_type - Default update strategy (full/incremental)

ResourcePlatformConfig (Per-platform settings of a resource)

resource_id, os, arch - The platform, unique per resource; "" is the platform-independent package, not a wildcard
update_type - Update strategy of the platform; a platform without a row uses the resource's

Version (Release versions)

channel - Release channel (stable/beta/alpha)
name - Version string (e.g., "1.0.0")
number - Numeric version for comparison
release_note - Changelog
custom_data - Arbitrary JSON data

Storage (Platform-specific packages)

update_type - Full or incremental
os - Operating system
arch - Architecture
package_path - File location
package_hash_sha256 - Package checksum
file_type - Archive format (zip, tgz, etc.)
file_size - Size in bytes
file_hashes - Hash map of files (for full updates)
old_version - Source version (for incremental updates)

Configuration Modes

Standalone Mode (`only_local: true`)

Single instance deployment
Configuration from local YAML files
No service discovery

Cluster Mode (`only_local: false`)

Multi-instance deployment
Consul service discovery
Remote configuration management
Health check registration
Automatic service deregistration on shutdown

Development

Running Tests

# Run all tests
go test ./...

# Run specific test
go test ./internal/pkg/vercomp/vercomp_test.go

# Run with race detector
go test -race ./...

Code Generation

After modifying schemas or dependencies:

# Regenerate Ent ORM code
make entgen

# Regenerate Wire DI code
make wiregen

# Build
make build

Project Structure

resource-backend/
├── cmd/                      # (Optional) Command-line tools
├── config/                   # Configuration files
│   ├── config.yaml          # Default config
│   ├── config.local.yaml    # Local overrides
│   └── config.cluster.yaml  # Cluster config
├── internal/
│   ├── application/         # App lifecycle management
│   ├── cache/               # Cache implementations
│   ├── config/              # Configuration loading
│   ├── db/                  # Database connections
│   ├── ent/                 # Ent ORM (generated)
│   │   └── schema/          # Database schemas
│   ├── interfaces/          # External interfaces
│   │   └── rest/            # HTTP handlers & routing
│   ├── logic/               # Business logic
│   ├── middleware/          # HTTP middleware
│   ├── model/               # Data models & DTOs
│   ├── oss/                 # Object storage integration
│   ├── pkg/                 # Utility packages
│   │   ├── archiver/        # Archive operations
│   │   ├── filehash/        # File hashing
│   │   ├── fileops/         # File operations
│   │   ├── patcher/         # Incremental patching
│   │   ├── validator/       # Request validation
│   │   └── vercomp/         # Version comparison
│   ├── repo/                # Data repositories
│   ├── tasks/               # Async task queue
│   └── wire/                # Dependency injection
├── bin/                     # Build output
├── main.go                  # Application entry point
├── Makefile                 # Build commands
└── CLAUDE.md                # AI development guide

Version Comparison

The service supports pluggable version parsers:

SemVer Parser

1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
v1.0.0 (automatically strips 'v' prefix)

DateTime Parser

2024-01-15 < 2024-01-16
2024-01-15T10:30:00Z
20240115103000

Custom Parsers

Implement the Parser interface in internal/pkg/vercomp/:

type Parser interface {
    Name() string
    CanParse(version string) bool
    Parse(version string) (interface{}, error)
    Compare(a, b interface{}) int
}

CDN Distribution

The service supports weighted multi-region CDN distribution:

Configuration Example

extra:
  cdn_prefix: "https://cdn.example.com"
  distribute_cdn_ratio: 70  # 70% CDN, 30% direct
  distribute_cdn_region: ["default", "hk", "kr"]
  download_prefix_info:
    default:
      - url: "https://download1.example.com"
        weight: 1
      - url: "https://download2.example.com"
        weight: 2
    hk:
      - url: "https://hk-download.example.com"
        weight: 1

How It Works

Client requests /resources/:rid/latest
Server generates temporary download key (10-minute TTL)
Server selects distribution method:
- CDN: Generates authenticated URL with MD5 token
- Direct: Selects region based on weighted random
Client receives download URL
Client uses HEAD request to check file size before download

Monitoring

Prometheus Metrics

Available at /metrics:

HTTP request counts and durations
Go runtime metrics (goroutines, memory, GC)
Custom business metrics

Health Check

Available at /health:

{
  "status": "ok"
}

Logging

Structured logging with Zap:

Request/response logging (via fiberzap middleware)
Error tracking with context
Configurable log levels (debug/info/warn/error)

Security

Authentication: Bearer token validation via middleware
Rate Limiting: Configurable download limits
CDN Token: MD5-based authentication for CDN URLs
Path Validation: Prevents directory traversal attacks
CORS: Configurable cross-origin policies

Performance Optimizations

Connection Pooling: MySQL (100 max, 50 idle, 25min lifetime)
In-Memory Cache: Ristretto with singleflight pattern
Redis Cache: Multi-level caching for version metadata
Distributed Locks: Redsync for concurrent operations
Async Processing: Background tasks via Asynq
JSON Performance: Sonic encoder/decoder
HTTP Performance: Fiber framework with fasthttp

Deployment

Docker Deployment

FROM golang:1.25-alpine AS builder
WORKDIR /app
COPY . .
RUN go mod download
RUN make entgen && make wiregen && make build

FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/bin/app .
COPY --from=builder /app/config ./config
CMD ["./app"]

Environment Variables (Cluster Mode)

INSTANCE_IP=10.0.1.5
SERVICE_ID=resource-backend-1
REGION_ID=us-west

Database Migration

Schema auto-migration on startup:

if err := mysql.Schema.Create(context.Background()); err != nil {
    // Handle error
}

For production, consider using migration tools like atlas or migrate.

Troubleshooting

Common Issues

Q: Build fails with "package X is not in std" A: Ensure Go version 1.25.3+ is installed and GOROOT is set correctly.

Q: Wire generation fails A: Run go install github.com/google/wire/cmd/wire@latest first.

Q: Database connection fails A: Check MySQL is running and credentials in config/config.yaml are correct.

Q: Redis connection fails A: Verify Redis is running on the configured address (redis.addr).

Q: Consul registration fails (cluster mode) A: Ensure INSTANCE_IP and SERVICE_ID environment variables are set.

Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Run code generation if needed (make entgen && make wiregen)
Run tests (go test ./...)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0) - see the LICENSE.md file for details.

AGPL-3.0 Summary

✅ Commercial use
✅ Modification
✅ Distribution
✅ Private use
⚠️ Network use is distribution - If you run a modified version on a server, you must make the source code available to users
⚠️ Disclose source
⚠️ License and copyright notice
⚠️ Same license (copyleft)
⚠️ State changes

For commercial licensing or questions, please contact the project maintainers.

Contact & Support

GitHub: https://github.com/MirrorChyan/resource-backend
Issues: GitHub Issues

Acknowledgments

Fiber - Express-inspired web framework
Ent - Entity framework for Go
Wire - Compile-time dependency injection
Asynq - Redis-based task queue
Sonic - High-performance JSON library
Zap - Blazing fast, structured logging

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
config		config
internal		internal
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE.md		LICENSE.md
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Latest commit

History

Folders and files

Repository files navigation

Resource Backend

Features

Tech Stack

Prerequisites

Quick Start

1. Clone the Repository

2. Configure the Application

3. Generate Code

4. Build and Run

API Endpoints

Public Endpoints

Get Latest Version

Download Resource

Head Download Info

Admin Endpoints (Require Authentication)

Create Resource

Platform Update Types

Create Version

Update Release Note

Update Custom Data

Health Check

Metrics

Architecture

Layer Structure

Database Schema

Configuration Modes

Standalone Mode (only_local: true)

Cluster Mode (only_local: false)

Development

Running Tests

Code Generation

Project Structure

Version Comparison

SemVer Parser

DateTime Parser

Custom Parsers

CDN Distribution

Configuration Example

How It Works

Monitoring

Prometheus Metrics

Health Check

Logging

Security

Performance Optimizations

Deployment

Docker Deployment

Environment Variables (Cluster Mode)

Database Migration

Troubleshooting

Common Issues

Contributing

License

AGPL-3.0 Summary

Contact & Support

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Standalone Mode (`only_local: true`)

Cluster Mode (`only_local: false`)