Key takeaways
- 856 of the 1393 actions in RespireCloud have a passing live result recorded on a real server; 6 currently fail.
- Two security audits in October produced 28 findings between them, none critical, and all are fixed.
- 303 actions that change something have neither a unit test nor a live result yet, and we list them.
- An independent external penetration test has not started, and we say so on the security page.
Most software announcements say "thoroughly tested" and stop. We would rather show the numbers, including the ones that do not look good. RespireCloud has a generated test checklist that lists every action, and what it says today is below.
What the checklist counts
RespireCloud is built from modules, and every module declares its actions in a manifest. A script reads the manifests, the panel routes, the command line and the test files, and writes one row per action. For each row it records two things. A unit test exists or not. And the newest live result: a real call on a real server, recorded as PASS or FAIL with the date, the server and the release.
Nobody edits the file by hand. If it says something, a test file or a result file says it first.
The numbers
As of this post the checklist has 1393 actions, of which 906 change something.
| Measure | Count |
|---|---|
| Actions in all modules | 1393 |
| Actions with a unit test | 383 |
| Newest live result is PASS | 856 |
| Newest live result is FAIL | 6 |
| Mutating actions with no test at all | 303 |
The 856 passing results were recorded on real test servers, not on mock objects. Dry run and chained runs are marked as such in the results, so a plan is never counted as a full run.
What the live runs found
Running real calls finds bugs that unit tests miss. In the last weeks, live runs found and fixed problems in the cloud server drivers, among them a request limit that was too small for the start up data a server needs. The bug list has 11 open entries, one of medium severity. We fix them in order, and the list is not hidden.
Audits
We ran two security and code audits in October. The first, on 19 October, found 15 findings: 3 high, 5 medium, 5 low and 2 informational. The second, on 20 October, covered everything built after the first and found 13: 3 high, 4 medium, 4 low and 2 informational. None was critical, and every finding is fixed. Four white box reviews on 10 and 11 October had earlier produced 15 findings, also fixed, each with a test.
The security page has the current state and the two findings we have not closed.
What we have not done
- 303 untested mutating actions. They are listed. Many are rare or administrative, but they are gaps.
- An external penetration test. It has not started. For a hosting panel we think an independent test is necessary, and we will say when it happens.
- Customer backups. The importers have run on archives we built, not on production backups from customers.
Why publish this
A table with gaps in it is more useful than a claim without one. If you are deciding whether to trust a panel with your servers, you can read what has been run and what has not. Read more about the design in the security model, or see the performance numbers from our load test.