Diogo Peralta Cordeiro

Self-Hosting a Small Production Stack: Linux, Backups, Hardening and Recovery

· updated · self-hosting, linux, homelab, devops, backups, recovery, cybersecurity, nginx, docker, matrix, nextcloud, monitoring

I have been self-hosting for many years. What started as a way to learn Linux, web servers and networking gradually became a long-running personal infrastructure environment: websites, personal cloud storage, code hosting, documentation, communication tools, file services, monitoring and automation.

I do not see this setup as a toy server. I treat it as a small production environment: documented, recoverable, monitored and hardened enough that failures become engineering problems rather than emergencies.

This post is a summary of that approach. The point is not to publish my infrastructure map. The point is to show the engineering discipline behind running personal services over a long period of time.

Architecture overview

The infrastructure follows a simple layered model:

Public Internet
    │
    ▼
DNS + TLS
    │
    ▼
Nginx reverse proxy
    │
    ├── Public websites
    ├── Personal cloud services
    ├── Documentation/wiki services
    ├── Code hosting
    ├── Communication services
    ├── Notification services
    └── Health/status endpoints

Storage layer
    ├── OS disk
    ├── bulk data disk
    ├── encrypted service storage
    └── external/offline backup media

Operations layer
    ├── backup scripts
    ├── restore procedures
    ├── service maintenance scripts
    ├── cron/systemd scheduled jobs
    ├── health checks
    └── notifications

The stack mixes traditional Linux services and containerised applications. I do not containerise everything by default. Some services are easier to run and maintain as normal Linux services; others benefit from Docker or Docker Compose because they have multiple moving parts, isolated dependencies or a repeatable deployment model.

The reverse proxy is the main public entry point. Internally, services are separated by application, user, directory, port and deployment model. This keeps the public surface smaller and allows each service to evolve without exposing every internal component directly.

Services I operate

The environment has included several categories of service:

This combination is useful because it exercises many different parts of real systems work:

A homelab becomes much more valuable when it stops being only a collection of installed applications and becomes an operated system.

Reverse proxy and web serving

Most public traffic is handled through Nginx. The reverse proxy layer is responsible for:

A simplified public-safe reverse proxy pattern looks like this:

server {
    listen 80;
    server_name example.com www.example.com;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name example.com www.example.com;

    ssl_certificate     /etc/letsencrypt/live/example/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example/privkey.pem;

    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    location ~ /\.(env|git|svn|hg) {
        deny all;
    }

    location / {
        proxy_pass http://127.0.0.1:PORT;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

The important idea is not that every vhost looks exactly like this. The important idea is that the proxy becomes a security and operations boundary. Services can bind locally, expose only what they need, and move internally without changing the public contract.

Web development and publishing stack

My own websites are also part of the infrastructure. They are not only static pages; they involve application code, styling, content management, CV generation, blog publishing, deployment and server administration.

The stack has involved:

This is useful because it connects software development to operations. A personal website is not just a design exercise. It needs routing, backups, deployment, caching, permissions, TLS, error handling, content editing and long-term maintainability.

Stateful data and filesystem layout

One of the main lessons from self-hosting is that state matters more than the application binaries.

Applications can usually be reinstalled. Stateful data is what must be protected:

For that reason, I organise the system so that stateful data, service configuration and operational scripts are easier to identify and back up. A clean layout makes recovery possible because it answers a basic question: “What do I need in order to rebuild this service?”

A public-safe example of that mental model is:

/system
    OS packages, users, base configuration

/services
    application code, containers, service definitions

/data
    uploaded files, cloud storage, repositories, service state

/databases
    SQL dumps or database-backed service state

/secrets
    keys, tokens, credentials, encryption material

/operations
    backup scripts, maintenance scripts, restore notes

The real system is more specific than this, but the principle is stable: state should not be scattered randomly across the server without documentation.

Encrypted storage for sensitive services

Some services benefit from encrypted storage. In my case, part of the communication stack is stored inside a LUKS-backed encrypted volume. That encrypted filesystem contains the service data and its database state.

The important design points are:

A simplified version of the idea is:

encrypted-service.luks
    ├── service-data/
    ├── database/
    ├── config/
    └── docker-compose.yml

The backup contains the encrypted image. The sensitive key material is copied separately and should be moved to safer storage, such as an offline disk or encrypted vault. This separation matters: a backup of encrypted data is not enough if the key is lost, but storing the key casually next to the data weakens the whole design.

Backup strategy

My backup approach is built around the idea that backups must be restorable, not merely present.

A typical full backup needs to cover:

For SQL-backed services, the backup should include proper dumps rather than relying only on copying live database files. For file-backed services, rsync-style backups work well, especially when combined with a dated retention area for changed or deleted files.

A simplified backup flow looks like this:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_ROOT="/mnt/backup/current"
DATE="$(date +%F)"

mount_backup_disk

notify "backup started"

dump_databases "$BACKUP_ROOT/dbs.sql"

enable_maintenance_mode_for_stateful_web_apps

rsync -aAX --delete \
  --backup --backup-dir="$BACKUP_ROOT/OLD/$DATE" \
  /data/ "$BACKUP_ROOT/data/"

rsync -aAX --delete \
  /services/ "$BACKUP_ROOT/services/"

rsync -aAX --delete \
  /operations/ "$BACKUP_ROOT/operations/"

copy_selected_system_config "$BACKUP_ROOT/system-config/"

disable_maintenance_mode_for_stateful_web_apps

write_backup_marker "$BACKUP_ROOT/when.txt"

unmount_backup_disk

notify "backup completed"

This is not intended to be a copy-paste script. It shows the operational pattern:

  1. mount backup media;
  2. notify start;
  3. dump databases;
  4. pause or stabilise stateful services where appropriate;
  5. copy state and configuration;
  6. preserve deleted/changed files in a dated area;
  7. write a backup marker;
  8. unmount and power off backup media;
  9. notify completion.

The backup disk should not be treated as always-online storage. Keeping it mounted all the time increases exposure to accidental deletion, filesystem corruption and ransomware-style failure modes.

Sensitive material backup

Sensitive material is handled separately from ordinary backups. This includes things like:

The operational rule is simple: ordinary backup data and sensitive key material should not have exactly the same handling.

A simplified sensitive-backup flow looks like this:

#!/usr/bin/env bash
set -euo pipefail

SENSITIVE_BACKUP="/mnt/backup/sensitive"

install -d -m 700 "$SENSITIVE_BACKUP/luks"
install -d -m 700 "$SENSITIVE_BACKUP/env"
install -d -m 700 "$SENSITIVE_BACKUP/ssh"

install -m 600 /secure/keyfile "$SENSITIVE_BACKUP/luks/service.key"
install -m 600 /root/.config/service.env "$SENSITIVE_BACKUP/env/service.env"

sync

After that, the sensitive backup should be moved to safer storage, such as an offline encrypted medium or an encrypted vault. This makes the recovery process possible without casually exposing the keys that make recovery possible.

Recovery testing and offline inspection

The best backup strategy is one that can be tested without touching production.

For encrypted service storage, a useful pattern is read-only inspection from backup:

cryptsetup open --readonly \
  /path/to/backup/encrypted-service.luks \
  service_inspect \
  --key-file /path/to/secure/keyfile

mount -t ext4 -o ro,norecovery \
  /dev/mapper/service_inspect \
  /mnt/service_inspect

ls -lah /mnt/service_inspect

umount /mnt/service_inspect
cryptsetup close service_inspect

The ro,norecovery part is important for an ext4 filesystem that may look unclean because it was copied while not formally unmounted. The goal is inspection, not mutation. I want to be able to verify that the expected service data, database files and configuration are present without replaying a journal or writing to the backup image.

This is the kind of operational detail that makes a backup plan real. A restore checklist should explain not only “what is backed up” but also “how to inspect it safely”.

Service administration scripts

Over time, I have built and maintained scripts for recurring administrative tasks.

Examples of what these scripts handle:

This matters because scripts turn one-off server knowledge into repeatable operations. A good admin script captures:

For example, a hosting-management script can make the difference between manually editing Nginx, PHP-FPM, users and directories every time, and having a repeatable workflow such as:

hosting-admin create project community-only
hosting-admin php-enable project
hosting-admin add-domain project example.com www.example.com
hosting-admin disable project
hosting-admin purge project

The exact script names are not important publicly. The engineering habit is important: reduce undocumented manual operations, make repeated tasks explicit, and keep destructive actions clear.

Cron and systemd timers

Scheduled maintenance is another important part of the stack.

Typical scheduled tasks include:

For simple jobs, cron is often enough. For jobs that benefit from dependency management, logging and service integration, systemd timers are cleaner.

A simplified systemd timer pair might look like this:

# /etc/systemd/system/example-health.service
[Unit]
Description=Run example health check

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/example-health.sh
# /etc/systemd/system/example-health.timer
[Unit]
Description=Run example health check periodically

[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
Persistent=true

[Install]
WantedBy=timers.target

The value is not just automation. It is visibility. Failed systemd units are easier to inspect than silent manual habits.

Health checks and monitoring

For a personal infrastructure stack, monitoring should be lightweight but real.

The monitoring layer should answer questions such as:

A simple health script can output JSON for a private or public status page:

{
  "host": "server",
  "checked_at": "2026-06-22T18:00:00Z",
  "services": {
    "nginx": "ok",
    "php-fpm": "ok",
    "database": "ok",
    "backup_age_hours": 12,
    "disk_usage_percent": 71,
    "tls_days_remaining": 64
  }
}

Even a small status page is useful because it turns vague worry into concrete signals. It also makes it easier to notice when a maintenance task silently stopped working.

Hardening principles

The hardening approach is pragmatic rather than theatrical. The goal is to reduce obvious risk and make compromise or misconfiguration less likely.

The principles I apply include:

Security is not a single tool. It is the combination of boundaries, defaults, permissions, updates, observability and recovery.

Example: preventing accidental secret exposure

A recurring class of web-server mistakes is accidentally serving files that were never meant to be public: .env, .git, backups, lockfiles, local configs or editor artefacts.

A generic Nginx defensive block can help:

location ~ /\.(?!well-known) {
    deny all;
}

location ~* \.(env|ini|log|sql|bak|old|orig|swp|lock)$ {
    deny all;
}

location ~* /(composer\.(json|lock)|package-lock\.json|yarn\.lock|Dockerfile|docker-compose\.ya?ml)$ {
    deny all;
}

This does not replace correct deployment practices, but it adds a useful guardrail. Public web roots should contain public assets, not source repositories, secrets or operational files.

Example: backup freshness check

Backups should produce machine-checkable evidence that they ran.

A simple marker-based check can be enough:

#!/usr/bin/env bash
set -euo pipefail

MARKER="/backup/when.txt"
MAX_AGE_HOURS=36

if [[ ! -f "$MARKER" ]]; then
    echo "backup marker missing"
    exit 2
fi

last_backup_epoch="$(stat -c %Y "$MARKER")"
now_epoch="$(date +%s)"
age_hours="$(( (now_epoch - last_backup_epoch) / 3600 ))"

if (( age_hours > MAX_AGE_HOURS )); then
    echo "backup too old: ${age_hours}h"
    exit 2
fi

echo "backup fresh: ${age_hours}h"

This is not sophisticated monitoring, but it catches a very common failure: assuming backups are running because a script exists.

Database handling

Database-backed services need special treatment.

Copying a live database directory is not always enough. Depending on the database and service, the backup plan may need:

For MariaDB/MySQL-style services, a logical dump can be part of the normal backup process:

mysqldump --single-transaction --routines --triggers --events \
  --all-databases > dbs.sql

For PostgreSQL-backed services, the equivalent might be service-specific dumps or a documented filesystem-level backup strategy if the database lives inside a service container or encrypted volume.

The key point is that “database exists somewhere under the service directory” is not a backup strategy. The backup method should match the database engine and the consistency requirements of the service.

Documentation as operational memory

The most valuable improvement in the whole setup has been documentation.

Good infrastructure notes answer:

This kind of documentation is not glamorous, but it is what turns a personal server from a fragile pet into an understandable system.

What I learned from operating it

Self-hosting has taught me practical lessons that transfer directly to professional software engineering:

This mindset also changes how I build applications. I care about configuration, logs, migrations, database backups, deployment paths, service boundaries and failure modes because I have had to operate systems myself.

Technologies and practices involved

The stack has changed over time, but the work has involved:

Conclusion

This infrastructure is personal, but the engineering practice is real.

It has required me to design, deploy, secure, monitor, document and recover systems over a long period of time. It has involved web development, Linux administration, databases, reverse proxying, encrypted storage, automation, backups, monitoring and operational discipline.

That is why I consider self-hosting part of my technical experience. It is not just “running a server”. It is a continuous exercise in full-stack engineering, DevOps, cybersecurity fundamentals and long-term maintainability.