Here’s a high-level overview of the components within Lyra.
A manifest will typically consist of one workflow definition written in a declarative or compiled language of choice. That workflow defines the desired state of an infrastructure or piece of an infrastructure.
The currently supported languages are Puppet, Go, TypeScript and YAML. Manifests written in Go must either be compiled and provided as a service or executed using a Lyra link that performs a
go run on the source.
The Lyra Command Line Interface. Provides commands like apply, validate, and version.
Performs evaluation of workflow step definitions using services. Services and definitions are obtained from the Loader.
Loads and manages all services, workflow steps, value types, and functions available to the system.
Parses workflow manifests written in a specific language into step definitions that the workflow engine can understand. Executes imperative steps (actions) on demand using an evaluator native to the language.
A Hiera 5 compatible framework for looking up various values such as environment settings, command line options, secret credentials, and configuration values. A workflow or a step can declare that input should be fulfilled by lookup.
The lookup framework is easy to extend with new functionality by adding special backend functions tailored to read from a file or use vault, k8s, etc.
A persistent store that maps Lyra internal step identifiers to external resource identifiers.
An external process described by a service definition. Examples of services are:
Identity - The Identity Storage
Aws - Provides handlers for a number of Aws resource types
Puppet-Workflow - Language Front-end for Puppet. Produces step definitions from Puppet manifests
The Lyra architecture relies heavily on dynamically typed values. A dynamically typed value can hold data of any kind and is described using a dynamic type. Important features of dynamically typed values are:
Efficiently serialized/deserialized without loss of type information.
Reflection to and from Go values.
Readable string representation.
Well defined comparison and arithmetic.
Type assertion. Enables checks for type correctness.
Unified way for accessing properties and invoking methods.
Immutability, which eliminates a lot of potential concurrency problems.
The Type system used in Lyra is the same as in Puppet. This vouches for future integrations that communicate using complex data types (RichData).
The Loader framework provided by the pcore module is used to form a loader hierarchy in Lyra. The basic loader hierarchy finds all the well-known types such as String, Integer, Array, etc. It also allows for new types and functions to be added. Lyra then adds a customized loader that finds services and definitions.
All types, functions, workflow steps, etc. are loaded on demand. The loader finds many things by looking for named files in well known directories such as “workflows” and “types”. When a user invokes the Lyra CLI with an apply command with the name of a manifest, the loader will look for a file with that name that has a known extension such as “.yaml” or “.pp”. If found, it will load it and hand it over to the workflow engine for execution.
The ability to send complex data structures between processes is very important to Lyra since much of its tasks are performed using services. All communication with such services are currently based on Protocol Buffers over gRPC. The evaluator provides efficient bi-directional conversion between dynamically typed values and the Data protocol buffer provided by the Lyra *data-protobuf *module, enabling data of arbitrary complexity to be sent over the wire in a type-safe and efficient manner.
Lyra covers both programming models by providing specializations of a Step, such as Resource, which is pure declaration of a desired state.
Lyra also allows generic Action steps that are very similar to functions. I.e. code that is called with parameters and returns values. Such actions can do virtually anything. When controlled by Guards they can mimic state-changes that are not easily expressed using resources.
Lyra makes ordering easy without enforcing a specific model onto the user. As mentioned, all order of execution can be determined by the workflow engine, either by just looking at the input and output declarations of activities contained in a workflow, or by running all activities sequentially.
Order isn’t applicable within individual steps because they are either a declaration of state, a block of code to execute, or a workflow in which case it manages its own order internally. If it is code, then that code is in charge of its own actions. The workflow engine treats an invocation of the code as one-off execution of something opaque.
A special collect step enables the declaration of a step that should be invoked multiple times. The declared step will run multiple times, each time with a different value of a given variable or variables. The invocations will happen in parallel. Lyra has two collect steps, “times” and “each”.
Create, read, update and delete are distinct functions that Lyra aims to provide out of the box. In many cases, all of them can be performed using one single workflow that declares a set of resources and a set of sane rules. The CRUD methods are:
Create - create an external resource based on a desired state.
Read - collect data and optionally report diffs between the desired and actual state.
Update - update an external state with new data.
Delete - delete the external state.
It is expected that more elaborate schemes will be architected and implemented. There will however still be cases where automated rules do not apply. In Lyra, this is solved by writing special workflows for the non-default situations.
Lyra is built using Go modules. The suite of repositories it uses can be found at the Lyra Project at GitHub. Here’s a brief description of each module (a module maps 1-1 to git a repository).
Contains the Lyra CLI, Lyra Loader, and a suite of resource handlers.
Makes Terraform providers usable to Lyra as resource handlers.
Lyra Software Development Kit. Intended for authors of new Lyra services. It provides what’s needed to register new services, handles, definitions, and types to the Go author. An sdk will be provided for each of the supported languages.
A Hiera 5 compatible system used to lookup data from various sources such as yaml files, environment variables, or databases.
Manages the Puppet Type System, dynamic typed values, Puppet compatible serialization of RichData, and bidirectional reflection between Go and Puppet types and values.
The Workflow Engine.
Formalized Warning and Error reporting.
Protobuf definition for transferring generic Data structures.
The YAML Front-end
The Puppet Language Front-end
Parser used when parsing types and manifests. The parser is a full implementation of the Puppet DSL.
The Manages the evaluation of ASTs (Abstract Syntax Trees) produced by the Puppet Parser.
Contains lots of tests written in Puppet Spec language for parser, evaluator, and hiera modules.