Websites

nginx (Ubuntu/Debian package) + one PHP-FPM master per account and PHP version. Spec: docs/specs/web.md. Runtimes beyond PHP, presets, deploys, WordPress toolkit are the next card.

How a change is applied (AC-web-12/13/16)

The agent keeps immutable numbered generations under /var/lib/respirecloud/web/gen/<N>/ and a symlink /etc/respirecloud/web/current -> gen/<N>. nginx and every PHP-FPM unit read their config through that symlink (nginx -c /etc/respirecloud/web/current/nginx/nginx.conf, nginx uses only relative includes). web.config.apply = load generation N, apply the change, render N+1 into a scratch dir, run nginx -t -c and php-fpm -t -y on the exact bytes that would go live, diff, create docroot/tmp/log dirs (symlink-safe), swap the symlink (atomic rename), start/reload units, health-check every changed site through the local nginx (refused connection or HTTP 500; 502/504 for PHP sites) and on any failure flip back to N and reload again. A rejected snippet never touches the live config (hash unchanged). History = the generation list (web.config.history, who/when/why also in m_web.history); web.config.rollback activates an older generation with the same validate/health/auto-revert flow. 30 generations are kept. dry_run on create/update/delete/layers.set/ rollback returns the per-file unified diffs as a module.Plan.

Layers (AC-web-14/15)

global -> server (node) -> plan -> reseller -> user (account) -> site. Typed settings (http.*, php.*, pool.*, proxy.allow_internal, site.auto_create) are merged lowest-authority-first; a layer may lock keys so lower layers cannot change them (checked on save and on resolve). Admin-only keys cannot be set from reseller/user/site layers. web.layers.effective shows value + source layer. Snippet hooks: http (admin layers only; users may only declare limit_req_zone/limit_conn_zone prefixed with their account name), server, location, php_ini, php_pool. Plan/reseller ids of an account come from core.principal.get; when the caller may not read them the last value seen (account_ctx) is used so everybody resolves the same layers.

Snippet allow-list (non-admin layers)

Parsed with a real tokenizer; every statement is checked, errors carry the line number. Allowed: timeouts, gzip, add_header, expires, index, try_files, return, rewrite, error_page, allow/deny, auth_basic(+file inside home), root/alias inside the account home only (no variables, no ..), set (not $rc_*), proxy_pass to non-internal http(s) hosts, proxy_set_header, limit_req/limit_conn (own zones), access_log off, location and a restricted if. Rejected with a reason: load_module, lua/perl/js, include (all), fastcgi_pass/uwsgi/scgi, server_name, listen, ssl_*, error_log, fastcgi_param, disable_symlinks, resolver, real_ip*, anything unknown. php.ini keys are allow-listed with value checks; open_basedir must stay inside home; pool keys are bounded (pm.max_children 1..200). The agent re-validates everything (control plane validation is only for early, friendly errors).

Isolation (AC-web-18)

• PHP workers run as the account user, in a per-account master unit rc-fpm-<ver>-<user>.service with Slice=rc-acct-<user>.slice (limits from system.slice.apply apply). Socket /run/rc-php/<user>-<ver>.sock is user:www-data 0660: other accounts cannot connect; fastcgi_pass cannot be written by users.
• Pool-level php_admin_value[open_basedir] = own home (+tmp, system php dirs); upload/session/tmp dirs in ~/tmp.
• nginx runs as www-data, which is added to each account's group to read docroots; every server block has disable_symlinks if_not_owner from=$document_root, so a symlink to another account's files is not served.
• Docroot/tmp creation by root uses openat(O_NOFOLLOW) per component: a symlink planted in the home cannot redirect it.
• Logs live in root-owned /var/log/rc-web/<account>/ (a user cannot symlink a log file onto a root-opened path).
• Unknown Host headers get a neutral 404 page (00-default.conf); the https default server rejects the handshake.
• Proxy sites to loopback/private hosts are refused unless the admin sets proxy.allow_internal=true (SSRF to the panel). A vouched app/container resource belonging to the same account is the narrow exception; Web checks its exact port.

Actions

web.server.ensure (prepare node: nginx, PHP versions from the distro or the Ondrej Sury repo with pinned key fingerprint, drop-in for nginx.service, generation 1), web.php.versions, web.sites.create|list|get|update|suspend| unsuspend|delete|reconcile, web.layers.get|set|effective, web.config.history|rollback, web.sites.cert.install|remove, and event handlers web.on_domain_created (auto site per plan's site.auto_create; alias/parked join the parent's site), web.on_domain_deleted, web.on_account_deleted. Site types: static, php, proxy. Limit key web.sites. Resource kind site for sub-user grants.

Contract for the ssl module

• HTTP-01: every plain-HTTP server block (and the default vhost) serves /.well-known/acme-challenge/ from /var/lib/rc-acme (agentop.WebAcmeDir), before any redirect rule. The ssl agent writes the token file to /var/lib/rc-acme/.well-known/acme-challenge/<token> (dir exists, root 0755). Works for any domain of any site, and unknown hosts.
• Install a certificate: call action web.sites.cert.install {site_id, fullchain_pem, privkey_pem}. The agent validates the pair (tls.X509KeyPair, not expired), stores them in /etc/respirecloud/web/certs/<site id>/ (fullchain.pem, privkey.pem 0600), re-renders the site with a listen 443 ssl block (HTTP/2, HSTS when the http.hsts_max_age setting is > 0) through the normal validate/swap/health/rollback path, and restores the previous certificate if that fails. web.sites.update {force_https:true} turns plain HTTP into a 301 to https (ACME path stays on HTTP). web.sites.cert.remove reverses it. Certificates live outside the generations, so a rollback never loses them. All names of the site (primary, serve and redirect aliases) share the one certificate; ssl should request a SAN cert for web.sites.get -> domains[].name.
• Events: web.site.created|updated|deleted carry site id and account for ssl to react (issue after create).

Verified

See docs/tasks/w2/w2-02-web.handover.md for the lab run (curl proofs, failed-config rollback, isolation).

Not done / next

PHP version install/remove by admin after first ensure (only additive via web.server.ensure), per-site pool tuning beyond the first site of a pool, extension manager, error/slow log viewers, permission fixer, site manifest export, UI bundle, basic-auth/IP/hotlink consumption from security, FrankenPHP, runtimes.

Protection per site (w11-21)

Firewall, request limits, security-header presets, bot challenge and hotlink protection are typed layer settings (webgen/protection.go): waf.mode|paranoia|exceptions|body_limit|fail, rate.preset, headers.*, http.security_headers, http.hsts_max_age, bot.mode|level|paths, hotlink.mode|allow. Customers pick presets; every value is a member of a fixed set or matches a strict pattern (a rule id, a path from a closed alphabet, a host name), and nothing is ever copied from a setting into nginx or rule syntax unchecked, so a customer cannot inject a directive or a SecLang rule. waf.body_limit and waf.fail are staff-only (AdminOnly). The site page has a Protection tab; staff set server-wide defaults on Web protection (/config/web/protection, the global layer).

Engine choice and why (Coraza service, not the ModSecurity connector)

• nginx.org nginx cannot load a distro's ModSecurity connector. The web module installs nginx from nginx.org mainline (see versionsNginxRepo); Debian / Ubuntu's libnginx-mod-http-modsecurity and EPEL's nginx-mod-modsecurity are built for the distro's own nginx and are refused by any other build ("module is not binary compatible"). nginx.org ships no ModSecurity module. Using the connector would mean switching the web server package for every site or compiling the connector on the customer's server, and both defeat "installs cleanly from packages on both families".
• Rule sets in distro packages are older and differ by family: CRS 3.3.x on Debian 12 / Ubuntu 22.04 and 24.04, 4.x only on newer releases, EPEL's mod_security_crs at its own pace and path. The score of a request would depend on the distro.
• So the engine is Coraza (Go, Apache-2.0) with the OWASP Core Rule Set 4, compiled into respirecloud-agent from go.sum-pinned module sources and run as the separate service rc-waf.service (respirecloud-agent waf, as the web user, no capabilities, read-only filesystem except its log and socket directories). Identical on Debian, Ubuntu, AlmaLinux and Rocky, the same rule set everywhere, nothing downloaded at install time (no unsigned binaries), updated with the panel. Components are listed in THIRD_PARTY_NOTICES.md as separate services.

How a protected site is served

For a site that has the firewall in detect/block mode the generated nginx file has two server blocks: the public one (listen 80/443, access log, request limit, bot check, ACME, location / -> proxy_pass to the firewall service over a unix socket) and an inner one that holds the real content and listens on /run/rc-waf/up-<site>.sock (real client address and scheme come from headers nginx sets and the service forwards; the inner block only accepts connections on that socket). Request bodies are therefore inspected too (up to waf.body_limit, the rest is passed on). The upstream group has the inner socket as a backup server: if the firewall service is down the site keeps serving (fail-open, the default; staff can choose fail-closed, on the Protection tab with the plain consequences next to the switch). Upgrade (WebSocket) requests are passed on after nginx's own checks. A node without the service renders none of this: waf.* settings are stored but have no effect until web.waf.install ran (the default mode is detect).

• Config files: the agent copies waf/<site>.json of the live generation to /var/lib/rc-waf/gen/<N>/ and moves /var/lib/rc-waf/current inside the same step that flips /etc/respirecloud/web/current (apply, rollback and automatic revert alike; the service runs as the web user and cannot enter /etc/respirecloud). The service follows that symlink by itself, so a rollback flips the firewall with nginx at the same moment; a request for a site it has not seen yet triggers a re-read, so the health check of a new generation always finds its rules.
• Safe by construction: a generation is built in a scratch directory; nginx -t and a compile of the changed sites' rules (wafd.Validate) must pass before the symlink moves; then the usual health check and automatic rollback apply. A bad exception is refused before anything goes live. Rules 949xxx / 959xxx / 980xxx (the blocking evaluation) and ids outside 911000-999999 can never be excepted. Kill switch: web.protection.kill (firewall + bot challenge off through the same safe apply) and the red button on the tab; staff can set waf.mode=off server-wide in the global layer.
• Footprint: the compiled rule set costs roughly 30-60 MB of memory per protected site; the service is capped at 2 GB.

False positives

web.protection.events groups the rule matches of a site's log (/var/log/rc-waf/<site>.events.jsonl, rotated at 8 MB) by request: refused requests (block) and requests detect mode would have refused, with rule id, message and a redacted, truncated excerpt. Each rule on a request offers Allow for "this page" or "whole site": web.protection.exception.add with dry_run returns the plan (what stops being checked, a warning for a site-wide exception); applying it adds <rule> [<path>] to the site's waf.exceptions, rendered as ctl:ruleRemoveById rules placed before the rule set. Each request carries an assistant entry point ("Why was this request blocked?", job web.waf_blocked).

Rate limits, headers, bot challenge, hotlink

• rate.preset = off / log (counts, limit_req_dry_run) / gentle (30 r/s) / standard (10 r/s) / strict (3 r/s) per visitor address with burst and connection caps; refused visitors get 429.
• Header presets (webgen.SecurityHeaders feeds both the template and web.protection.preview): nosniff, Referrer-Policy, X-Frame-Options, Permissions-Policy, Content-Security-Policy (off / report-only / enforce, relaxed or strict preset), HSTS with includeSubDomains and preload. Headers are repeated in the cached PHP location, where nginx would otherwise drop them.
• Bot challenge: nginx auth_request asks the service on every request of a site that enables it; paths are matched in the service after the same normalisation nginx applies (decode, merge slashes, resolve dots, lower-case). A gated path without a valid cookie gets a 403 page that solves a SHA-256 proof of work (12/14/17 leading zero bits) in plain JavaScript (works on http too) and receives an HMAC cookie bound to site and client address (12 h). Challenge and cookie are stateless HMACs under a secret in /var/lib/rc-waf/state/secret.
• Hotlink: valid_referers none blocked server_names <allowed hosts>, applied to image, video, audio and font files.
• Directory passwords: see the next section (w11-92). Basic-auth snippets inside the home stay allowed by the snippet allow-list.

Actions

web.protection.get|set|preview|events|posture, web.protection.exception.add|remove, web.protection.kill, web.waf.install|status. Agent ops web.waf.install|status|events.

Directory passwords (w11-92; cPanel's "Directory Privacy")

A path of a site asks for a name and password (HTTP Basic). Actions web.dirpass.list (read), protect, unprotect (medium risk: a customer without an authenticator can still use it; the plan is shown first), person.add, person.password, person.remove (all with dry_run, permission web.config.edit; the list needs web.view). The screen is a card on the site's Protection tab.

• Rules (webgen/dirauth.go): a path /admin protects /admin and everything below it, never /administrator; / is the whole site. Paths use letters, digits and . _ ~ -, no .., at most 20 per site, 100 people per path; /_rc and /.well-known cannot be protected. When protected paths overlap the longest wins. Names: letters, digits, . _ @ + -. Passwords: 8-72 bytes (bcrypt reads 72), not the name, not one repeated character. The last person of a path cannot be removed (unprotect instead).
• Rendered as two map $uri blocks per site (the realm text and the password file per path, off and none elsewhere) and auth_basic $rc_dpr_<site>; auth_basic_user_file $rc_dpf_<site>; in the content server (behind the firewall that is the inner server, so the firewall still inspects first). No location is added or replaced, so PHP, redirects, error pages, the firewall, the bot challenge and the certificate challenge (auth_basic off there) keep working; a redirect is answered before the password is asked. The decision uses the normalised request path, so a script, a static file or a try_files fallback below the path all ask. A user snippet that sets its own auth_basic in a location wins for that location.
• Storage: the control plane keeps dirpass_paths / dirpass_users (bcrypt cost 10, hash only; the clear password is hashed at once, never stored, logged or returned, and the host redacts every param named password). The agent keeps the files in the generation (auth/<site>/<hash of path>.htpasswd) and publishes them, before the web server is told to reload (apply, rollback and automatic revert alike), to /var/lib/rc-web-auth/gen/<N>/ with current pointing at it (the web workers cannot enter /var/lib/respirecloud). Every folder is 0750 and every file 0640, group = the web server's user: the workers read them and no hosting account can open or list them. They are kept out of sites/<id>.json, of every diff, of readTree and of web.config.get; loadGen reads the hashes back, so a certificate install, another site's change, a re-generation or a PHP version change keeps the protection, and a rollback restores the people of that generation. (Between the flip and the end of the reload, workers still on the old configuration read the new files: a path unprotected a moment ago answers 500 to a request in those milliseconds, never open.)
• The web server checks the password again on every request below the path (bcrypt: about 60 ms of one worker per request at cost 10), so protect folders, not busy public pages, and use the request limit on the site.
• Without a certificate the password crosses the internet in clear: the list says so and the screen shows a warning.

Site logs and site health (w11-33)

• Site logs reach the log store. The agent's log shipper reads /var/log/rc-web/<account>/<site>.access.log, <site>.error.log and php-fpm-<version>.log (root-owned directory, files opened without following symlinks) and tags every line with the account (from the directory the agent created, never from file content) and site=<site id>. Sources: site.access, site.error, site.php. They are customer data: the logs module stores them in the account's own tenant, so only that account (and its collaborators who hold logs.view) and staff can read them; the same masking as every other log applies before storage; retention is the logs module's.
• Site health. web.health.scan runs every 2 minutes and asks each node (one web.site.health agent call) whether each PHP site's pool service is running and accepts connections on its socket, whether a proxied site's loopback target is listening (other hosts are never dialled), and whether the installed certificate is still valid. The result is health (ok, degraded, down) with a plain health_reason on every site read; the Websites list and the site page show it. A state changes only after two scans in a row agree (about 4 minutes), so a restart never raises anything, and one bell entry is raised when a site goes down and one when it is back (events web.site.unhealthy / web.site.recovered).
• Firewall (w11-21). A protected site whose firewall service is stopped keeps being served (fail-open): its health stays ok and it carries a health_notice. With the staff setting waf.fail=closed the same condition is down.

Firewall defaults and events (w11-50)

• Default mode is detect; a site whose app.kind is a recognised CMS (wordpress, joomla, drupal, magento, prestashop) resolves to block with source "app" unless some layer set waf.mode (webgen.Resolve). app.kind is a protection key: whoever installs the app (the Apps module) sets it with web.protection.set: since w11-64 the Apps module does it when it links a WordPress or Drupal app to a site (modules/apps/cp/appkind.go). A WordPress site also gets its preset exclusions in the service (wafd.appProfile): for requests under /wp-admin/ and /wp-json/ that carry the signed-in cookie, the attack rules skip the post body fields (content, title, excerpt, JSON content.raw, ...) and WordPress's own cookies; every other field, the login page and anonymous requests keep the full rule set.

• Two instances, no gap on restart (w11-64). web.waf.install runs two copies of the service, rc-waf (socket waf.sock) and rc-waf-b (waf-b.sock), Type=notify: a copy compiles its rules first, binds its socket, then says READY. nginx lists both (max_fails=0: a copy that is restarting is skipped per request, not held out for ten seconds) and only when BOTH are down does the backup (the site itself, uninspected) answer. Installing again restarts them one after the other, each only after the previous one is READY, so a restart or an update never leaves a site uninspected. nginx keeps connections to the service open (keepalive), and the service closes idle ones later than nginx does. A GET or HEAD for an image, style, script or font whose path holds no script extension and whose query is at most 64 plain characters skips the rules (wafd.plainStaticFile): the web server answers it from disk. JSON and XML bodies are parsed into arguments (rule 200000/200001), as the Core Rule Set expects.

• Firewall events are shipped to the Logs page as source site.firewall (agent siteFirewallFiles): owner = the account whose /var/log/rc-web/<account>/<site>.access.log exists; masked by the logs module's masker.
• Measured (docs/distros.md): w11-50 on AlmaLinux 10 saw about 12 ms at p95; w11-64 on a loaded Ubuntu 26.04 lab saw 3.3-3.7 ms for a small GET (static files 1 ms), POST bodies not measured. Restart gap: 26 % of requests uninspected before, 0.05 % (8 of 14958) after; not zero.