# Replacing Jenkins with a Single-Binary Deploy Panel

URL: https://mepritam.dev/references/minimal-internal-deploy-tool/

Architectural evaluation of zero-CI, local-execution internal deployment tools like deploy-panel, detailing pros, cons, security model, and decision matrix.

---

**Author:** Staff Software Engineer & Senior Systems Architect
**Topic:** Architectural Evaluation of Zero-CI Internal Deployment Tools
**Case Study:** `deploy-panel` (Go, Systemd, Traefik, Bash)

---

## Executive Summary

Modern CI/CD pipelines have ballooned into resource-heavy ecosystems. Setting up Jenkins, GitLab CI, or GitHub Actions self-hosted runners introduces operational overhead. Managing JVMs, maintaining SSH key distribution across build nodes, and debugging complex YAML DAGs cost developer time. Running heavy build daemons to execute simple `git pull` and `docker compose up -d` commands wastes server memory.

For single-server applications, staging environments, agency setups, or bootstrapped SaaS platforms, traditional CI tools represent unnecessary abstraction.

This paper explores the **Minimal Internal Tool System** model. We analyze `deploy-panel`, a zero-dependency Go admin dashboard. It triggers host-native deployment scripts using Go's `os/exec` package, teeing stdout/stderr directly to disk logs and live WebSockets.

---

## Visual Architecture and Comparison

### Heavy CI/CD vs. Minimal Single-Binary Deploy Panel

![CI/CD System Comparison](/images/references/ci_vs_deploy_panel.jpg)

| Metric / Aspect | Traditional Heavy CI (Jenkins, GitLab CI) | Minimal Internal Deploy Panel (`deploy-panel`) |
| :--- | :--- | :--- |
| **Idle Memory Footprint** | 1.0 GB – 2.5 GB RAM (JVM / Runner Daemons) | **10 MB – 20 MB RAM** (Compiled Go Binary) |
| **Execution Model** | Remote SSH / Agent Polling / Webhook Queues | **Host-Local `os/exec`** |
| **Dependencies** | Java JVM, Node.js, Plugins, External DB | **Zero External Runtimes** (Embedded Web UI) |
| **Trigger Latency** | 10s – 60s (Runner initialization, artifact sync) | **< 10ms (Instant Execution)** |
| **Attack Surface** | Distributed SSH keys in cloud secrets managers | **Local Unix socket / Docker Gateway IP** |
| **Config Surface** | Complex pipeline YAMLs, Groovy scripts, plugins | **Single `config.yaml` + Native Bash Scripts** |

---

## How deploy-panel Executes

### System Execution Blueprint

![Deploy Panel Architecture Blueprint](/images/references/deploy_panel_architecture.jpg)

### 1. Single Binary with Embedded UI
`deploy-panel` leverages Go’s `go:embed` package to bundle HTML, CSS, and client-side JavaScript into a single ELF binary (`~12MB`). The deployment needs no `node_modules`, no Webpack build step, and no separate static file server.

```go
// web/embed.go - Zero external static asset deployment
package web

import "embed"

//go:embed templates/*.html
var templateFS embed.FS

//go:embed static
var staticRaw embed.FS
```

### 2. Host-Local Execution and Live Stream Teeing
Instead of establishing remote SSH tunnels, `deploy-panel` runs directly on the target host. When an operator triggers a deployment:
- The server spawns `bash /path/to/script.sh` through `os/exec.CommandContext`.
- The process pipes `stdout` and `stderr` streams into an `io.MultiWriter`.
- One writer appends to a disk log at `logs/<project>/<run-id>.log` paired with a `.json` status sidecar.
- A parallel writer broadcasts live ANSI terminal output across WebSockets to the web dashboard.

```mermaid
sequenceDiagram
    autonumber
    actor Admin as Operator / UI
    participant Web as Go Web Server
    participant Lock as Mutex Runner
    participant Bash as Local OS (Bash Script)
    participant Disk as File System (logs/)
    participant WS as WebSocket Client

    Admin->>Web: POST /api/projects/{id}/deploy (CSRF Token)
    Web->>Lock: Acquire Project Lock
    alt Project Already Deploying
        Lock-->>Web: 409 Conflict
        Web-->>Admin: Error: Deploy in progress
    else Lock Acquired
        Lock->>Bash: exec bash /opt/app/deploy.sh
        loop Live Streaming Output
            Bash->>Disk: Append to logs/{id}/{run-id}.log
            Bash->>WS: Stream ANSI text over WebSocket
        end
        Bash-->>Lock: Exit Code 0 / 1
        Lock->>Disk: Write {run-id}.json metadata
        Lock-->>Web: Deploy Complete
        Web-->>Admin: Success Notification
    end
```

### 3. Concurrency Control
To avoid overlapping deployments, `deploy-panel` maintains an in-memory execution map guarded by `sync.Mutex`. If a second deploy trigger arrives while a project runs, the server rejects the request with HTTP `409 Conflict` rather than queuing up stacked runs.

### 4. Defense-in-Depth Security Model
Running arbitrary shell scripts on a production box requires strict security guardrails:

* **Bcrypt Password Hash & Session Hygiene:** Passwords live as bcrypt hashes in gitignored `.env` files. Authentication checks use constant-time operations to prevent timing side-channel attacks. Sessions use server-side in-memory tokens with a 12-hour TTL.
* **Fail2Ban IP Rate Limiting:** Five consecutive failed login attempts lock out the originating IP for 15 minutes. Behind reverse proxies like Traefik, the panel parses client IPs accurately from trusted `X-Forwarded-For` headers.
* **Double-Submit CSRF Tokens:** State-changing requests enforce per-session CSRF token headers alongside session cookies.
* **Systemd Sandbox Isolation:** The systemd unit enforces `ProtectSystem=strict` and `ProtectHome=true`, confining file system write access to configured paths (`ReadWritePaths=/opt/deploy-panel/logs /root/apps`).
* **Network Binding:** The server binds to Traefik over Docker's internal gateway IP (`172.17.0.1:8099`) or Unix sockets (`unix:/run/deploy-panel.sock`), hiding the HTTP port from the public internet.

---

## Pros: Why Senior Engineers Choose Zero-CI Minimalism

1. **Extreme Resource Efficiency:**
   Running Jenkins on a 2GB RAM VPS takes 70% of available memory before building a single line of code. `deploy-panel` operates at ~15MB RAM and near-zero CPU idle, freeing host resources for actual application workloads.

2. **Elimination of SSH Key Proliferation:**
   Remote CI systems require storing private SSH keys or API access tokens inside cloud SaaS vaults. If attackers breach the CI platform, they compromise your production servers. Local execution removes SSH key distribution entirely.

3. **Operational Simplicity & "Boring Technology":**
   The entire deployment logic lives in human-readable Bash scripts or Makefiles (`git pull --ff-only && docker compose up -d`). There are no proprietary plugin ecosystems, Jenkinsfiles, or obscure pipeline DSLs to maintain.

4. **Zero Cold-Start Latency:**
   Deploys start instantly upon clicking the button. There is no waiting for build node provisioning, container image pulls for runners, or artifact uploads.

5. **Immutable Audit Trails:**
   Every build log persists as plain text (`.log`) alongside a structured JSON sidecar (`.json`). History survives service restarts without requiring a database engine (SQLite/PostgreSQL).

---

## Cons & Limitations: Where the Pattern Fails

1. **Single-Host Scope:**
   `deploy-panel` targets single-node topologies. It cannot orchestrate zero-downtime rolling deploys across a multi-region Kubernetes cluster or auto-scaling AWS EC2 target group out of the box.

2. **Host Build Contention:**
   Compiling code (`go build`, `npm run build`, `docker build`) directly on the target host uses the production server's CPU and RAM. On high-traffic nodes, resource spikes during builds can degrade application performance.

3. **Lack of Advanced CI Pipeline Matrices:**
   It does not support complex parallel testing matrix pipelines (e.g., testing against 10 versions of Node.js across operating system flavors before merging code).

4. **Single-Admin RBAC Constraints:**
   Built with a lean single-admin user model, it lacks multi-tenant Enterprise SSO (Okta/SAML) and fine-grained per-user permissions (e.g., User A can deploy Staging, but only User B can deploy Production).

---

## Production Infrastructure Blueprint

### Systemd Unit File Security Hardening (`deploy-panel.service`)

```ini
[Unit]
Description=deploy-panel
After=network.target

[Service]
Type=simple
User=deploy
WorkingDirectory=/opt/deploy-panel
ExecStart=/opt/deploy-panel/deploy-panel -config /opt/deploy-panel/config.yaml
Restart=on-failure
RestartSec=2

; Security Hardening Directives
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/deploy-panel/logs /root/apps

; Optional Unix Socket Runtime Directory (for Nginx/Caddy)
RuntimeDirectory=deploy-panel
RuntimeDirectoryMode=0750

[Install]
WantedBy=multi-user.target
```

### Traefik Dynamic Reverse Proxy Setup (`deploy-panel.yml`)

```yaml
http:
  routers:
    deploy-panel:
      rule: "Host(`deploy.example.com`)"
      service: deploy-panel-service
      entryPoints:
        - websecure
      tls:
        certResolver: letsencrypt

  services:
    deploy-panel-service:
      loadBalancer:
        servers:
          # Bounds to Docker internal gateway IP (172.17.0.1:8099)
          - url: "http://172.17.0.1:8099"
```

---

## Standardized Makefile and Script Contract

To enforce clean separation of concerns, `deploy-panel` contains no inline application build logic. Instead, it delegates to a standardized script or `Makefile` entrypoint in each application repository.

### Shared Runner Entrypoint (`scripts/make-deploy.sh`)

```bash
#!/usr/bin/env bash
  # Shared deploy-panel entrypoint: run make deploy in project workdir.
set -euo pipefail
exec make deploy
```

### Canonical Production Makefile (`Makefile`)

```makefile
.PHONY: deploy build pull healthcheck

deploy: pull build up healthcheck prune

pull:
	git pull --ff-only

build:
	docker compose -p prod pull

up:
	docker compose -p prod up -d --remove-orphans

healthcheck:
	@echo "Verifying application health..."
	@curl --fail --retry 5 --retry-delay 2 http://127.0.0.1:8080/health || exit 1

prune:
	docker image prune -f
```

---

## Code Mechanics and Safety Systems

### Go Process Teeing and Path Sanitization

A critical vulnerability in deployment tools is **Directory Traversal** (`../../etc/passwd`). `deploy-panel` mitigates path traversal by validating user-supplied run IDs against an exact regular expression before touching the filesystem:

```go
// Enforces ISO-8601 UTC timestamp format paired with 4-byte random hex suffix
var runIDPattern = regexp.MustCompile(`^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{8}$`)

func (m *Manager) RunLog(projectID, runID string) ([]byte, error) {
    if _, ok := m.projects[projectID]; !ok {
        return nil, ErrProjectNotFound
    }
    // Reject untrusted path inputs before filepath.Join
    if !runIDPattern.MatchString(runID) {
        return nil, ErrRunNotFound
    }
    return os.ReadFile(m.logPath(projectID, runID))
}
```

To stream live ANSI terminal output without memory leaks, `deploy-panel` implements a custom `teeWriter` which fans out bytes synchronously to the log file on disk and active WebSocket subscribers:

```go
type teeWriter struct {
    file *os.File
    bc   *broadcaster
}

func (t *teeWriter) Write(p []byte) (int, error) {
    n, err := t.file.Write(p)
    t.bc.publish(p) // Non-blocking broadcast to WebSocket channels
    return n, err
}
```

---

## Self-Deployment and Deadlock Avoidance

One key feature of `deploy-panel` is its ability to deploy itself (`deploy-self.sh`).

### The Systemd Deadlock Problem
If `deploy-panel` calls a script that invokes `systemctl restart deploy-panel`, a systemd deadlock occurs:
1. `systemctl restart` issues a synchronous request to systemd.
2. Systemd waits for `deploy-panel` (`MainPID`) to exit.
3. `deploy-panel` blocks while waiting for its child process (`deploy-self.sh`) to finish.
4. The system hangs indefinitely.

### Process Tree Inspection and Deferred Restart
`build-and-run.sh` inspects its process ancestor chain (`ppid`) to check if `deploy-panel.service` spawned it. If detected, it sets `SELF_DEPLOY=1` and delegates the service restart to an asynchronous single-shot systemd timer:

```bash
running_under_deploy_panel_service() {
    local main pid i=0
    main="$(systemctl show -p MainPID --value deploy-panel 2>/dev/null)"
    pid=$$
    while [[ "$pid" -gt 1 && "$i" -lt 64 ]]; do
        [[ "$pid" == "$main" ]] && return 0
        pid="$(ps -o ppid= -p "$pid" | tr -d ' ')"
        i=$((i + 1))
    done
    return 1
}
```

When `SELF_DEPLOY=1` is active, the script builds the new binary, updates the executable on disk, and schedules a deferred restart (`systemd-run --on-active=2s systemctl restart deploy-panel`). This lets `deploy-self.sh` return HTTP 200 OK cleanly before the server cycles.

---

## Engineering Decision Matrix

```
                          ┌─────────────────────────────────────────┐
                          │ Is your app hosted on single/few nodes? │
                          └────────────────────┬────────────────────┘
                                               │
                       ┌───────────────────────┴───────────────────────┐
                       │                                               │
                      YES                                              NO
                       │                                               │
       ┌───────────────┴───────────────┐               ┌───────────────┴───────────────┐
       │ Do you need complex parallel  │               │ Use Distributed CI/CD System  │
       │ CI build matrices / K8s fleet?│               │ (GitHub Actions, GitLab CI,   │
       └───────────────┬───────────────┘               │ ArgoCD, Kubernetes)           │
                       │                               └───────────────────────────────┘
               ┌───────┴───────┐
              YES              NO
               │               │
       ┌───────┴───────┐   ┌───┴───────────────────────────────────────────┐
       │ Use SaaS CI   │   │ IMPLEMENT MINIMAL DEPLOY PANEL (`deploy-panel`)│
       │ (GitHub/GitLab│   │ - Zero JVM bloat                              │
       │ Cloud Runners)│   │ - Instant host execution                      │
       └───────────────┘   │ - Systemd sandboxed bash scripts              │
                           └───────────────────────────────────────────────┘
```

---

## Enterprise Hardening and Compliance Extensions

For teams requiring SOC 2 Type II or ISO 27001 compliance, three lightweight extensions upgrade this architecture without introducing heavy CI software:

### 1. Off-Host Audit Log Shipping (CC6.8 Compliance)
Local disk logs (`logs/<project>/<run-id>.log`) provide instant debugging, but enterprise compliance requires immutable off-host audit trails. A lightweight Vector or Promtail daemon forwards run logs to Grafana Loki or AWS CloudWatch:

```yaml
  # Vector agent configuration for log streaming
sources:
  deploy_panel_logs:
    type: file
    include: ["/opt/deploy-panel/logs/*/*.log"]

sinks:
  loki_audit:
    type: loki
    inputs: ["deploy_panel_logs"]
    endpoint: "https://loki.internal.company.com"
```

### 2. OIDC and Tailscale Single Sign-On
When scaling beyond a single operator, replace the single-admin bcrypt authentication with an OIDC proxy like OAuth2 Proxy or Tailscale WhoIs headers. This binds deployment triggers to individual developer identity providers such as Okta or Google Workspace with hardware MFA.

### 3. One-Command Disaster Recovery (RTO Under 60 Seconds)
Since `deploy-panel` stores state strictly as code and filesystem logs, server recovery takes under 60 seconds on a fresh Linux VPS:

```bash
git clone git@github.com:org/infra.git /opt/infra
cd /opt/infra/deploy-panel
./install-go.sh && ./build-and-run.sh
```

---

## Staff Engineer Conclusion

As Staff Engineers, our duty is to remove unnecessary complexity, not celebrate it. Applying enterprise CI tools like Jenkins or ArgoCD to single-server setups is an anti-pattern driven by Resume Driven Development.

`deploy-panel` proves that less is often far more. A 15MB Go binary, standard Systemd sandboxing, Git-tracked config, and simple Bash scripts deliver a secure, fast deployment system that lasts for years with zero maintenance.