What steps can you take to ensure system logs are accurate and useful for troubleshooting?

El primer punto antes de el registro es tener muy en claro a que al organización define Riesgo, puede ser riesgo a la salud, a la perdida de clientes, al incumplimiento de procesos, etc.

Applications, infrastructure and security logging and traceability is the main weapon for metrics, troubleshooting and security standards. On each log flow we will need to differ between multiple log levels and in each case it will have different audience and events. Considering this approach, logging is the main weapon, but the configuration and services used for each case should be the main focus of a logging system. Lesser logs with full observability is better than all the logs without it.

El segundo punto debe ser la categorización de los riesgos, es decir utilizando por ejemplo la ISO 31.000 o IRAM 3.801 debemos definir claramente como lo vamos a definir. Luego vendrá la registración. Donde como se menciona debe definirse la estructura de los registros. Pero antes de registrarlos deben estar correctamente definidos.

Multiples teams will read the same logs with multiple purposes like security, implementers, customers, stakeholders, quality assurance and many others audiences. Using a full standard logging format will allow to all the teams use the same stream with having multiple integrations or tools.

To standardize log formats for accurate troubleshooting, establish a consistent structure across systems, including key information such as timestamp, severity level, source component, and specific event details. Ensure uniformity in log message syntax and use clear, descriptive labels for each field. Implement a logging framework or tool that enforces these standards automatically, reducing manual errors. Regularly review and update the standardized format based on evolving needs and feedback from troubleshooting experiences. Training team members on the importance of adhering to standardized log formats reinforces their effectiveness in identifying and resolving issues efficiently.

Logging is our first audit system on all the fronts like security, development, business sustainability and others. Considering this we can't lose any data of logging or we will also lose the truth of history of events, applications and tools.

Los datos deben tener una taxonomía normada siguiendo normativas y buenas prácticas, estos deben Integros, Confidenciales, Auténticos, estar siempre disponibles y consistentes.

Logging system are the core observability inside all companies, but the veracity in all the lines is fundamental. If somehow we have a logging system polluted with wrong data, we will base our decisions and troubleshooting in lies and this can impact the entire business.

One thing I have found helpful is to speak to those taking the logs. In this new world of automation not all logs are automated and are still manual. Debriefing with those who filled out the manual logs can add much needed context for future troubleshooting efforts. Even with automated logs there are usually human observers to add additional context.

Any logging systems receive millions lines of data every day, minute or even second, we should automate the reading of logs for security, observability, metrics and others. Teams should use summary log reports to analyse the macro logging system to check for improvements, new features or problems.

Time, time, time. Synchronize your log timestamps with physical clocks as much as possible, at the best using PTP or NTP protocols, as logs with wrong timestamps are very hard to analyze. And don't forget about timezones!