Understanding the ComplianceAsCode build system
Introduction
This section aims to provide an introduction to the ComplianceAsCode build system to developers interested in extending or debugging it.
Before beginning, it is generally expected that some familiarity with the relevant standards in this space are understood. Among others, these are:
XCCDF, the eXtensible Configuration Checklist Description Format; this is a textual representation format of various steps in hardening a particular system.
OVAL, the Open Vulnerability and Assessment Language; this is a standardized mechanism for auditing various compliance checks.
OCIL, the Open Checklist Interactive Language, an expressive language for handling manual compliance checks.
CPE, the Common Platform Enumeration; a scheme for identifying software and systems.
SCAP source data stream format, a mechanism for combining the above into a single redistributable file.
Additionally, some familiarity with the content layout (as discussed in previous chapters) is also implied.
However, while this document serves as a guide, ultimately the build system is changing and thus inspecting the code is the only way to find the answers to many questions.
High-Level Overview
ComplianceAsCode’s content project is ultimately the combination of three things:
A collection of content in a format-agnostic manner,
A build system for collecting this content and combining it to form artifacts understood by other systems,
A test system for validating both the compliance of these artifacts to various standards and the correctness of the content in the repo.
As previous sections describe in detail the expectations around content in the
repo, this section aims to describe the build system. For understanding of the
test systems, it is suggested to look at the README under tests/
in the
repo.
The build system is generated by CMake and combines local Python utilities
with XML tools (such as xmllint
and xsltproc
) and OpenSCAP’s oscap
CLI executable. These Python utilities transform the various input files
into a more standardized format and apply Jinja macros to them. Ultimately
many of the artifacts we generate are XML-based so extensive XSLT processing
occurs after building the initial structure in Python. Finally, OpenSCAP
combines and references several files for us to build the finished artifacts.
CMake Structure
CMake requires projects have an entry point called /CMakeLists.txt
. This
uses the CMake language and drives building and installing the project. This
file contains several things:
The many build-time options for customizing the types of content generated,
The hand-off for generating each product’s specific content,
Common installation, testing, and distribution targets.
However, the specifics of building a particular product are contained in the
shared module located at cmake/SSGCommon.cmake
. This file contains all of
the CMake logic to build a particular product and exposes the top-level macro
ssg_build_product(...)
. This macro generates per-product build, installation,
and testing targets. While the specifics should be understood from this file
directly, in general this takes the following outline of steps in rough order
of occurrence:
Generate SCE content and metadata,
Generate a shorthand XML from various YAML files (like rules, variables, and profiles) – this is a single document that resembles the structure of the XCCDF but isn’t necessarily standards compliant,
Generation of templated content (such as Bash, OVAL, and Ansible) from the metadata contained in the rule YAML files,
Transformation of the shorthand into unlinked and linked XCCDF documents,
Combining the various OVAL, OCIL, and XCCDF documents into a single Datastream document,
Generating any derived products (such as CentOS and Scientific Linux).
Python Build Scripts
Various Python utilities under /build-scripts
contribute to this process;
refer to their help text for more information and usage:
add_stig_references.py
– add STIG reference identifiers into the unlinked XCCDF.build_all_guides.py
– generates separate HTML guides for every profile in an XCCDF document.build_profile_remediations.py
– generates separate remediation content for each profile.build_rule_playbooks.py
– generates per-rule per-profile playbooks in Ansible content.build_sce.py
– outputs SCE content and combined metadata.build_templated_content.py
– generates templated audit and remediation content.combine_ovals.py
– combines separate (per-rule, shared, and templated) OVAL XML trees into a single larger OVAL XML document.collect_remediations.py
– finds the separate (per-rule and templated) remediations and places them into a single directory.compile_profiles.py
– pre-processes profiles to handle inheritance and the product-independent controls format.cpe_generate.py
– generates the product-specific CPE dictionary and checks.enable_derivatives.py
– generates derivative product content from a base product.expand_jinja.py
– helper script used by the BATS (Bash unit test framework) to expand Jinja in test scripts.generate_fixes_xml.py
– combines the remediations (from the source directory) into a single XML document to be merged with the unlinked XCCDF.generate_man_page.py
– generates the ComplianceAsCode man page.oscap_svg_support.py
– helper script to check if OpenSCAP supports inline SVGs (used by the build system for feature detection).profile_tool.py
– utility script to generate statistics about profiles in a specific XCCDF/Datastream file.relabel_ids.py
– updates various internal identifiers to their final resolved values (e.g., with thexccdf_org.ssgproject.content_
prefix).sds_move_ocil_to_checks.py
– updates the datastream to have the correct structure for OCIL checks; only necessary for old OpenSCAP versions.unselect_empty_xccdf_groups.py
– updates the XCCDF document to remove selections of groups without any rules.update_sds_version.py
– used to build XCCDF v1.2 (instead of XCCDF v1.3) compatible content.verify_references.py
– used by the test system to verify cross-linkage of identifiers between XCCDF and OVAL/OCIL documents.yaml_to_shorthand.py
– generates the shorthand XML document from the various XML pieces.
Many of these utilities are simply front-ends over code in the SSG Python
module located under ssg/
.