Improve logging and error handling #888
Description
The problem
Users have a hard time setting up our CLI and plugins on their own.
Error handling
- Users have a hard time understanding errors from the CLI.
- Errors don't give hints on how to fix them.
- It often isn't clear in which step an error occurred.
- Debugging errors using
--verboseflag is of very little help, as this log level is barely used. - Some errors are unnecessarily verbose (e.g. Zod schema validation).
- Output and error streams from executed processes are often silenced unless there's an error, hiding potentially valuable information.
Logging
- The only progress indicator is a progress bar which is supposed to update discretely whenever a plugin is finished running (but has actually been broken for a long time, so it just skips to 100% immediately).
- For long-running plugin execution, there's no way for a user to tell if something is still happening or if it's hanging.
- Aside from the final report, very little information is logged about what the CLI is doing. This lack of transparency contributes to users struggling to understand where an error came from, as well as how the CLI is supposed to work in general.
- The progress bar also causes strange UI side-effects (scrolling logs out of viewport), is difficult to test and isn't suitable for logging in CI pipelines.
Proposed solution
- replace progress bar with spinners
- spinners should be shown whenever some potentially long-running process is running
- CLI should log high-level steps being performed (e.g. parsing config, executing plugin 2 out of 4, formatting report)
- all plugin logs should be prefixed by plugin which emitted them
- plugins should use logger with same verbose settings as CLI
- Zod errors should be made more compact
- ensure every error is given proper context (in which step is occurred)
- distinguish log levels (error, warning, info, debug)
- add E2E test for what's being printed to stdout in
cli-e2e(part of 1st contact) - improve errors & logs coming from upload command & portal-client #914
- improve logging in tests #915
- create environment variable to toggle logging (verbose) #913
- fix(ci): always show execute process errors and log stdout if verbose #952