SPIKE - Wazuh Ansible

[teddytpc1]

Objective
https://github.com/wazuh/internal-devel-requests/issues/1319

Description

As part of the DevOps overhaul objective we need to conduct research, analyze alternatives, and design how to implement the following changes.

1. Installation process:

• The Wazuh Ansible deployment alternative must use the Wazuh package URL instead of repositories.
• For the development packages, the download method will be defined here.

2. Installation configuration:

• Ensure the Wazuh Ansible deployment uses out-of-the-box Wazuh configurations.

3. Testing Improvements:

• Enhance deployment tests with additional checks, including log validation for errors and warnings.
• Ensure tests are compatible with both production and development packages.

4. Documentation Updates:

• Simplify Ansible installation documentation.
• Focus on prerequisites for Wazuh Ansible deployment, using the Wazuh Kubernetes documentation as a reference.

Implementation restrictions

• Testing Environment:
  • Tests must be implemented using GitHub Actions (GHA).
• Compatibility:
  • Ensure compatibility with both production and development package environments.
• Logs Validation:
  • Include robust log analysis to detect and report errors or warnings clearly.
• Minimal Maintenance:
  • Prioritize simplicity and minimize ongoing maintenance requirements.

Plan

1. Research & Analysis:

• Analyze the current Wazuh Ansible deployment process to identify required changes for using package URLs.
• Review and integrate the development package download method outlined in #1725.

2. Configuration Updates:

• Define how to modify the Wazuh Ansible deployment to use out-of-the-box configurations.

3. Test Enhancements:

• Define how the new GHA workflows will validate deployments with production and development packages.
• Define how to integrate log analysis into deployment tests to identify errors and warnings.

4. Documentation Updates:

• Define how to simplify the Ansible installation documentation.
• Rewrite Wazuh Ansible installation documentation, focusing on prerequisites and using Wazuh Kubernetes documentation as a style guide.

[YisDav]

Plan

The proposal of this plan aims to outline in a general way the steps and considerations necessary to carry out the above proposed actions. It begins with the definition of the starting points for the Ansible project structure. Subsequently, it considers the creation of workflows in GitHub Actions, which will be aligned with the established key points. In addition, a plan will be developed for the Out-of-the-Box (OOB) code creation stage for each Wazuh component, ensuring that each part is consistent with the goals and constraints proposed in this Issue. Finally, improvements in test implementation and log validation are addressed, as well as adjustments to the documentation.

Steps

1. Define starting points for the Ansible project structure (SPIKE - Wazuh Ansible #1461 (comment)).
2. Create the workflows for AIO and distributed deployments, aligned with the mentioned key points (SPIKE - Wazuh Ansible #1461 (comment)).
3. Create the Out-Of-the-Box (OOB) code for each Wazuh component (SPIKE - Wazuh Ansible #1461 (comment)).
4. Improve test implementation and validation of logs, errors and warnings (SPIKE - Wazuh Ansible #1461 (comment)).
5. Adjustments to the documentation (SPIKE - Wazuh Ansible #1461 (comment)).
6. Other considerations (SPIKE - Wazuh Ansible #1461 (comment)).

[YisDav]

Ansible Project Structure

Defining the overall structure of the Ansible project:

• Determine if roles will be used. Update: roles will be used.
• Define the certificate generation tool (related issue here).
  • Plan is designed according to the current setup
• Establish the general organization of the project (directories, files, etc.)
  • Chosen proposal: Simpler Approach

Proposals

Current Structure:

• It is proposed to maintain an organization similar to the current one. This means having two main directories: one for playbooks and another for roles.
• The playbooks directory will primarily store playbooks for Wazuh distributed (wazuh-distributed.yml), Wazuh AIO (wazuh-aio.yml), and Wazuh agent (wazuh-agent.yml).

Simpler Approach:

• Other projects like CloudBox or Plone manage their roles directory similarly but keep the playbooks in the main directory.

[YisDav]

Workflows

Both deployment workflows, single node and all-in-one, will be required to be parameterizable. In particular, each Wazuh component must be allowed to be downloaded using packages instead of repositories, via a URL (or pre-signed URL, for Development packages). Additionally, there is a compatibility table. That is, a list of operating systems where both workflows must be executed.

Wazuh Package URL Retrieval

• [Workflows] Obtain URLs for Wazuh packages from their respective buckets
  1. Generate the pre-signed URL for Development packages
     • Configure the role with permissions to access the package
     • Both, Production and PreRelease packages are public accesible, so they will not require a pre-signed URL. However, Development packages will require to generate a pre-signed URL (either for the latest uploaded package or packages identified with commit-sha in their naming).
     • The URLs for downloading the packages (pre-signed or public) are generated based in three downloaded list file of the S3 URIs or URLs.
  2. Store each URL in a variable that can be used later to instruct Ansible on where to download the packages.
  3. Verify that Ansible can download the package. Ensure the package's availability and confirm that no permission issues arise.

URL parametrization

• Wazuh component packages will be downloaded, by default, from the production environment.
• However, the download URL must be parameterizable or modifiable, so that you can define where you want to download the package from (either Production, PreRelease, or Development [PreSigned URLs])

Compatibility Matrix with Operating Systems

💡 Note: Support for ARM architecture will now also be included.

Supported Operating Systems

Name	Version(s)	Architecture(s)	Wazuh Components (supported)
RedHat	8, 9	x86_64, aarch64, arm	Indexer, Manager, Dashboard, Agent
Ubuntu	22.04, 24.04	x86_64, aarch64, arm	Indexer, Manager, Dashboard, Agent
Amazon Linux	2, 2023	x86_64, aarch64, arm	Indexer, Manager, Dashboard, Agent
CentOS	8	x86_64, aarch64, arm	Indexer, Manager, Dashboard, Agent
Windows	-	-	Agent
MacOS	-	-	Agent

[YisDav]

OOB code for each Wazuh component

Certificate Generation

• Utilize the previously defined tool: Learn more

Create Roles/Playbooks to Deploy Each Wazuh Component

1. Provisioning and Installation of Wazuh Indexer

• Sub-steps:
  • Download and install package(s)
    • Download and install missing dependency(ies)?
      • YUM
        • coreutils
      • APT
        • debconf
        • adduser
        • procps
  • Configure OpenSearch at /etc/wazuh-indexer/opensearch.yml
  • Deploy and configure certificates
  • Test installation and functionality of the component

2. Provisioning and Installation of Wazuh Server

• Sub-steps:
  • Download and install package(s)
    • Download and install missing dependency(ies)?
      • YUM
        • coreutils
      • APT
        • debconf
        • adduser
        • procps
        • gnupg
        • apt-transport-https
  • Configuration of Filebeat
    • Filebeat configuration file at /etc/filebeat/filebeat.yml
    • Store credentials in keystore
    • Download alert template for Wazuh Indexer
    • Install Wazuh module for Filebeat.
    • Deploy and configure certificates
  • Configure and test connectivity with Wazuh Indexer
  • Test installation and functionality of the component

3. Provisioning and Installation of Wazuh Dashboard

• Sub-steps:

  • Download and install package(s)
    • Download and install missing dependency(ies)?
      • YUM
        • libcap
      • APT
        • debhelper
        • tar
        • curl
        • libcap2-bin
  • Configure the dashboard
    • Configuration files:
      • /etc/wazuh-dashboard/opensearch_dashboards.yml
      • /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml
  • Deploy and configure certificates
  • Enhance installation security? Learn more
    • Note: This is not a production environment
  • Test installation and functionality of the component

• Create Playbook for Single Node Using Roles/Code of Each Component

• Create Playbook for Distributed Deployments Using Roles/Code of Each Component

[YisDav]

Test Improvements

Proposals

Utilize Ansible Lint in PR Checks

Implement linting using the ansible-lint action during pull request checks to ensure code quality.

Ansible Lint GitHub Action

Implement Log Parsing Post-Execution

• Log Parsing Script

  Develop a script (e.g., in Python) to parse logs for specific error or warning keywords such as "ERROR", "WARNING", or custom patterns.

• Log Analysis System Proposal

  The goal is to establish a log analysis system that facilitates review and debugging tasks, especially in case of errors or failures during any phase. This system will collect multiple logs generated by Wazuh and its components, making them available through a designated artifact.

  1. Collection of Key Log Sources for Debugging and Diagnosis During Deployments

     Wazuh Component	Logs
     Indexer	/var/log/wazuh-indexer/ /var/ossec/logs
     Manager	/var/ossec/logs/ /var/ossec/logs/ossec.log
     Dashboard	/usr/share/wazuh-dashboard/data/wazuh/logs
     Agent	/var/ossec/logs/ /var/ossec/logs/ossec.logs

  2. Automated Log Analysis

     Conduct an automated scan for errors and warnings after execution.

     • Execution Type:

       • Post-Execution: This occurs when other components have completed their deployment.

     • Statistics Generation:

       • A plain text file will be created containing:
         • Number of errors per component
         • Number of warnings per component
         • Total number of errors
         • Total number of warnings

  3. Artifact Generation:

     Once the statistics and logs for each component are collected, they should be uploaded as artifacts under a common directory while maintaining their full path for easy identification.

     Example structure:

     logs-artifact/
     ├── wazuh-manager/
     │   └── var/
     │       └── ossec/
     │           └── logs/
     │               └── ossec.log
     ├── ...
     │ 
     └── summary.txt

Implement Check and Diff Before Actual Deployment

Utilize the --check and --diff modes in Ansible to validate configuration changes before applying them. This is useful for verifying expected behavior without making actual modifications.

Example

ansible-playbook playbook.yml --check --diff

[YisDav]

Adjustments to the Documentation

Relocate Documentation Section

It would be more convenient to have this section on the Wazuh Agent documentation page.

• Remove this section from the documentation and replace it with a prerequisites section.

Prerequisites

• Ansible (core) ≥ 2.16
• SSH
• CentOS / RHEL 7 / Fedora
  • epel-release
• Debian / Ubuntu
  • lsb-release
  • software-properties-common
• Windows
  • Windows versions under current and extended support from Microsoft:
    • Ansible can manage desktop OSs including Windows 7, 8.1, and 10, and server OSs including Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016, and 2019.
  • PowerShell 3.0 or newer.
  • At least .NET version 4.0 should be installed on the Windows endpoint.
  • A WinRM listener should be created and activated.

[YisDav]

Considerations

Ansible Roles

Compatibility Issues

• First Research Process

  The syntax and structure of Ansible roles were first introduced in Ansible version 1.2, released in October 2013. Since their introduction, roles have become a cornerstone of Ansible's approach to modularity and reusability. The structure includes directories like tasks, vars, defaults, and handlers, which are still in use today.

Key Milestones and Changes in Role Syntax and Structure

1. Initial Definition (Ansible 1.2, 2013):

   • Roles were introduced with the now-familiar directory structure:

     roles/
       └── my_role/
           ├── tasks/
           ├── handlers/
           ├── templates/
           ├── files/
           ├── vars/
           ├── defaults/
           └── meta/
     
     

   • This structure allowed users to organize their playbooks and associated files into reusable components.

2. Introduction of Ansible Galaxy (Ansible 1.4, 2014):

   • Ansible Galaxy was launched as a platform for sharing and discovering roles. This encouraged the community to adopt roles widely and provided a standard for role structure and metadata.

3. Transition to Collections (Ansible 2.10, 2020):

   • While the role structure remained unchanged, roles became part of the larger concept of collections, which grouped roles, plugins, and modules together. This change was backward-compatible but required updates to workflows for publishing and consuming roles.

4. Minor Adjustments Over Time:

   • Updates to the meta/main.yml file to include dependencies and compatibility information.
   • Support for new features in tasks (e.g., block/rescue/always introduced in Ansible 2.0).
   • Enhanced linting tools like ansible-lint to enforce best practices in roles.

How Many Times Has the Role Structure Changed?

The core structure of roles has remained largely unchanged since its introduction in 2013. However, the ecosystem around roles has evolved:

• The introduction of collections in Ansible 2.10 was the most significant shift, but it did not directly change the role structure itself.
• Minor enhancements, such as metadata updates and new task features, have been added incrementally.

Conclusion

The role syntax and structure have proven to be stable over the years, with only minor adjustments to accommodate new features and improve usability. This stability makes roles a reliable choice for long-term projects. However, keeping up with ecosystem changes (like collections and module updates) is essential to ensure compatibility.

---

Hypothetical case: Multiple OS with multiple architectures

💡 Note: Keep in mind that there will be a different package for each Wazuh component, each architecture (AMD64 or ARM64) and each operating system family (Debian [.deb] or RHEL [.rpm], mainly). In addition, each package released in development (S3), will require the generation of a pre-signed URL in the corresponding workflow.

Assume a case in which Wazuh must be deployed using Ansible, with the particularity of having to install a given component on multiple hosts, with different operating systems and different architectures. For example, if a cluster of indexers must be installed on 3 different hosts. However, one of these hosts is Debian with AMD64 architecture, the other is a Debian with ARM64, and the remaining host is a RHEL with ARM64.

If it is necessary to download the packages from Production or PreRelease, there should be no problem, since it is possible to download the appropriate versions without the need for pre-signed URLs. What will be necessary, however, is to set up a URL to download each package, making sure that it is the version compatible with the corresponding operating system.

But on the other hand, if it is required to download the packages from Development, it will be necessary to have the pre-signed URLs for each needed package (debian_amd64, debian_arm64, and for rhel_arm64).

Case Solution Proposals:

1. Collect information from each host using ansible_facts, prior to the main deployment. Once we have the information of the operating systems and their architectures for each host, we create the required pre-signed URLs for each required package.

2. Create pre-signed URLs for all likely required packages. That is, for this example, URLs would be created for both operating systems (.deb, .rpm) and for both architectures (amd64 and arm64). All these URLs would be passed to the Ansible runtime, and there it will be chosen which one is suitable for each host. So, for any system and for any architecture there is always a pre-signed URL available from which to download the package.

[YisDav]

Implementation Plan

The initial implementation plan covers the following steps in a sequential manner:

1. Cleaning of the repository and new project initial setup
   • Follow the Ansible Project Structure (SPIKE - Wazuh Ansible #1461 (comment)) guidelines
   • Keep the .github/ directory
2. Creation of the playbook to collect information from the hosts.
   • Check the Hypothetical case: Multiple OS with multiple architectures for more information (SPIKE - Wazuh Ansible #1461 (comment)).
   • It will be required to gather both operative system family [host_os_family] (i.e., rhel or debian-based) and its architecture [host_architecture] (i.e., amd64 or arm64)
   • The information of each host_os_family and host_architecture must be handled in a way that the workflow can later identify, iterate and generate the pre-signed URL for each required package.
3. Workflow adaptations: Generation of pre-signed URLs for the necessary packages.
   • Must be able to generate pre-signed URL of each required S3 object (package) in case it isn't a Production or PreRelease package.
   • There will be three different URL list files, and will be eventually downloaded and used to generate URLs ansible variables file. Those three correspond to Development, PreRelease, and Production (qa-maintained)
   • For development packages, once each package URL has been created (based in a given, downloaded, S3 URIs list file), it must be saved and passed as a parameter for the Ansible runtime.
     • Options:
       • Ansible variable
       • Environment variable
     • Include the option to deploy Wazuh agent in the current workflows
     • For more information check the Workflows section (SPIKE - Wazuh Ansible #1461 (comment)).
4. Development of the main installation playbook(s).
   • For: distributed, all-in-one, and agent.
   • Must be able to gather the custom packages URL variables (in case it exists), otherwise default values must be used.
5. Creation of the roles: wazuh-indexer, wazuh-manager, wazuh-dashboard, wazuh-agent
   • Must be able to download and install its package from a given URL.
     • First, it will be needed to download the package
     • Then, install it using the usual package manager (i.e., apt or yum)

[YisDav]

Diagrams