Skip to content

architecture.md

Latest commit

 

History

History
104 lines (85 loc) · 8.6 KB

architecture.md

File metadata and controls

104 lines (85 loc) · 8.6 KB
Raw

Architecture

The project is written as a set of communicating containers. It allows to scale different parts of the application according to needs. It also increases application robustness, because even if one component have some issues (errors, down times), the others are not impacted with that work.

The components are manager, listener, evaluator-{upload,recalc}, aggregator, vmaas_sync, database and database_admin.

Components

  • manager - serves information about evaluated systems for a given account via REST API. It allows to display:

    • list of all systems registered for a given account with advisory related stats,
    • list of advisories evaluated for any of these systems,
    • list of all advisories, related to at least one system, with stats (e.g. related systems count). Manager component depends on database storage only and it's also the only application component directly accesed by application frontend. So it ensures high application stability and availability. It supports RBAC and so it depends on this service, however it can be disabled by setting ENABLE_RBAC=no. See component environment variables

  • listener - connects to the Kafka service, and listens for messages about newly uploaded archives. When a new archive is uploaded, it upserts system_inventory (including vmaas_json with installed packages, repos, and modules) and the matching system_patch row. Upload locks and checksum logic apply to system_inventory. It registers repositories for the system by pairing repo with the internal system id (system_inventory.id) through system_repo. After that it sends a Kafka message (patchman.evaluator.upload topic) to evaluate the system with the evaluator-upload component. This component also handles system deleting events (platform.inventory.events Kafka topic). It consumes template events from Content Sources (platform.content-sources.template), upserts template metadata, and on template-updated syncs template_advisory rows from the Content Sources API. When template advisories change and template_advisory_eval is enabled, it sends a bulk re-evaluation message to patchman.evaluator.user-evaluation. See component environment variables

  • evaluator-upload - connects to the Kafka service (patchman.evaluator.upload topic) and listens for evaluation requests from the listener component. For each received Kafka message it evaluates system with ID contained in the message. It loads each system by joining system_inventory and system_patch. As an evaluation result it updates system_advisories (referencing system_inventory.id), system_patch (evaluation caches, last_evaluation, and related fields), and advisory_account_data. Evaluation is scaled on two levels, firstly with multiple replicas (more pods) and secondary with multiple goroutines within single pod (set by CONSUMER_COUNT environment variable). See component environment variables

  • evaluator-recalc - same as the -upload instance but receives Kafka messages from vmaas-sync component (patchman.evaluator.recalc Kafka topic) after advisories list is updated in database (advisory_metadata database table). After this event all registered systems have to be re-evaluated because the input data for the system evaluation has changed. See component environment variables

  • evaluator-user-evaluation - same as the -upload instance, but listens for re-evaluation requests from the manager component when user adds or removes systems from a template, and from the listener when template advisory content changes (patchman.evaluator.user-evaluation topic). Recalc messages carry explicit system_ids; the listener resolves template-assigned systems before sending. The requests are separated as when there is a heavy load from inventory at the time it may take very long for systems to be recalculated/updated. See component environment variables

  • aggregator - maintains per-account, per-workspace advisory counts. When the evaluator processes a system upload or recalculation and updates system_advisories, it publishes an AdvisoryUpdateEvent to the patchman.advisory.update Kafka topic listing which advisory IDs changed for a given account. The aggregator consumes these events and recounts how many systems have each advisory applicable or installable, writing the results to account_advisory. This is the workspace-aware replacement for advisory_account_data (previously maintained by the evaluator). Incoming events are batched before processing. When enable_notifications is set in POD_CONFIG, the aggregator also publishes new installable advisories to platform.notifications.ingress and marks them as notified in account_advisory. See component environment variables

  • vmaas-sync - connects to VMaaS, and upon receiving notification about updated data, syncs new advisories into the database, and requests re-evaluation for systems which could be affected by new advisories. It's done via messaging to the patchman.evaluator.recalc Kafka topic and receiving by the evaluator-recalc component. Recalculation can be done in two modes. It's either done for all active systems or based on updated repositories. In this case only systems paired with updated repositories are selected and sent to recalculation. This mode is enabled by setting ENABLE_REPO_BASED_RE_EVALUATION=true. This component also performs system culling. Using container CLI it's possible to manually trigger advisories update (./scripts/sync.sh) and systems recalculation (./scripts/re-calc.sh). See component environment variables

  • database - Stores data about systems, advisories, system advisories and different related data. Detailed description of the component and data layout are in separate page.

  • database-admin - Executes database initialization and migrations. It needs all rights for the database. It also creates database users for all components and updates passwords for them, so it reads passwords for admin and all components from environment variables. In production a db-migration Job runs migrations once per deploy; other pods wait in check-for-db init until the schema is current. Before DDL it sets app users (listener, evaluator, manager, vmaas_sync) to NOLOGIN, optionally terminates lingering sessions when terminate_db_sessions=true, waits until no app sessions remain, runs migrations, then restores LOGIN. See Database migrations and the major migration runbook. Using container CLI it's possible to manually manage database (./scripts/psql.sh). See component environment variables

Components cooperation schema

Template content update flow

When Content Sources publishes a template-updated event:

  1. listener upserts template metadata and calls Content Sources GET /templates/{uuid}/advisories/ids.
  2. listener diffs the response against template_advisory and inserts or deletes rows.
  3. If advisories changed and template_advisory_eval=true, listener looks up systems assigned to the template and sends PlatformEvent{system_ids} to patchman.evaluator.user-evaluation.
  4. evaluator-user-evaluation re-evaluates each system in the message.
  5. For template-assigned systems, evaluator skips yum updates, calls VMaaS, and marks advisories in template_advisory as installable; other applicable advisories remain applicable.

template_advisory_eval rollout

One flag controls the full template-advisory feature. Set template_advisory_eval=true in POD_CONFIG on:

  • listener — triggers re-evaluation when template advisory content changes or a template is deleted
  • evaluator-upload, evaluator-user-evaluation, and evaluator-recalc — applies installability from template_advisory for template-assigned systems

Default is false everywhere. Enable listener and evaluator pods together; listener-only leaves advisories synced but installability unchanged, evaluator-only leaves content changes without automatic recalc.

This is separate from manager template_change_eval, which still controls recalc when users assign or remove systems via the REST API.