Author: Seth Cooper

EITR Technologies, LLC (EITR) announced today that through their collaborative working relationship with quantitative finance company Hudson River Trading (HRT), their developers and thought leaders have made significant contributions to the latest version of Salt (version 3005). EITR, as staunch proponents of automation and experts in cloud services, DevSecOps platforms, and cybersecurity took inspiration from HRT’s use cases, bug listings, and need for newly developed features. HRT has built one of the world’s most sophisticated computing environments for trading financial product research and development. Their researchers are at the forefront of innovation in the world of algorithmic trading, and their workflow is demanding. These EITR-contributed enhancements will make their workflow, transactions, and customer service more robust.

Salt Project is Python-based, open-source software used extensively throughout IT automation for remote task execution and configuration management. EITR hosts a large clientele, including large healthcare companies, cybersecurity integrators, publicly-traded SaaS companies, the Department of Defense, and finance firms such as the aforementioned Hudson River Trading. EITR’s commitment to creating and contributing to open-source repositories continues to provide exceptional value to their customers via cost savings, reduced solution development time, and the ability to remain vendor agnostic.

EITR Technologies’ CEO, Nicholas Hughes worked on many of the critical updates in the upcoming Salt version 3005. Hughes strongly believes in an “everything as code” methodology and is a regular contributor to many open-source projects hosted on GitLab and GitHub, where the Salt Project codebase is available free of charge. “As we focus on our engagement with important clients in both public and private sectors, we can more easily expose our core values to industry leaders like Hudson River Trading. As frequent contributors to a trusted technology like Salt, we can help to drive cutting-edge automation advances and stay abreast of ever-changing technology trends” said Nicholas Hughes, CEO, EITR Technologies.

While often summarized as a configuration management solution, the Python-based open-source software is a critical component of many organizations broader IT automation and orchestration strategy. The initial release of Salt took place more than 10 years ago, in March of 2011, with SaltStack forming as a company around the technology shortly thereafter. And due to ongoing popularity since that time, VMware acquired SaltStack in September of 2020.

Release notes for Salt 3005 include more than 30 EITR-developed enhancements, most of which were funded by Hudson River Trading and inspired by their many valuable use cases. Said enhancements include fixes to existing features, the addition of new features, and general “quality of life” enhancements.

Upcoming Changes in Salt 3005

Bugs Fixed

#62265 Fix variable reuse causing requisite_in problems

The PR fixes some variable reuse within a loop in a certain section of the code that overwrites a critical variable holding the state name for requisites. In very specific circumstances, requisites were not found due to receiving the wrong state module name when not explicitly specified.

#62259 Fix service.enabled unavailable service error in test mode

This PR fixes an issue with the service.enabled state where it would error in test mode when a service was unavailable (not installed). The fix now assumes that a service will be installed by a state if not present while in test mode.

#62241 Fix postgres views

The postgres_privileges.present state was unable to verify existing privileges on views because it filtered on ordinary tables only. This PR adds the ability to verify and properly set privileges on views.

#62088 Fix useradd hard-coded relative command names

Functions in the useradd execution module had hard-coded command names for useradd, usermod, and userdel which relied on the PATH environment variable to be properly set in order to run correctly. This PR added support for searching for those binaries using the salt.utils.path.which function instead of solely relying on the PATH environment variable to find them.

#62043 Fix Jinja template cache loading

This PR fixed a long-standing edge case where Jinja templates could be loaded from the cache despite being removed from the Salt filesystem. After a Jinja file is removed from the Salt filesystem, the state should fail to run with a TemplateNotFound error. Loading cached files could result in user surprise when complex map loading decisions were made. With the new behavior, the template will not be loaded if the file is cached but no longer present on the Salt filesystem.

#61986 Fix Debian repo with trailing slash always shows changes

Fixed an issue where Salt showed a change every time a pkgrepo.managed state was run with a trailing slash added to an apt repository’s name.

#61896 Fix archive.extracted top level directory ownership handling

This PR fixes a long-standing issue with archive.extracted where it would not properly enforce ownership on the extracted directories and files in certain scenarios.

#61847 Fix reporting of errors for file.directory in test mode

Fixed a scenario where the file.directory state module reported an error and shows the state as failed in test mode when a user account has not yet been created and is specified as the owner of a directory. Going forward, a warning will be thrown about the inability to find the user/group in test mode. However, the state will not return as failed.

#61744 Fix file.replace mtime update on no changes

This PR fixed an issue where a file’s modification date was changed by the file.replace state module even if there were no content changes.

#61739 Fix master caching for dynamic environments

This PR fixed an issue where performing cp.list_states, cp.list_master, or similar function calls between multiple dynamic environments often resulted in incorrect file listings depending on cache settings. The code responsible for returning file lists cached the listing by environment. In a dynamic configuration, the environment used for the cache file was the literal __env__ string. This resulted in the cache for a previously run environment being returned for other environments within the cache retention period.

#61488 Fix setting state_aggregate option on the minion

This PR addressed an issue where the state_aggregate minion option was not respected and always favored the master option whether or not it was actively set on the master.

#61201 Fix state includes in a dynamic environment

Fixed an issue where including another state when running with a dynamic salt environment (__env__) in file_roots throws a "Nonexistent saltenv found in include" error.

Features Added

#62227 Add random shuffle and sample functions

This PR adds execution module and Jinja filter access to random.sample and a variation of random.shuffle.

#62045 Add file.comment ignore_missing

This PR added the ability to ignore missing regex matches in a file when using the file.comment state. This now allows for a “comment only if a match is found” workflow and prevents unnecessary state failures or requisite use. This PR also fixes a small bug where changes would be reported in test mode when the line has already been commented.

#61992 Add dig.cname function

This PR adds a function to the dig execution module which supports returning CNAME records for a given host.

#61833 Update salt.utils.stringutils.human_to_bytes for broader use

This PR moved the _handle_unit function from the virt module to the stringutils.human_to_bytes utility in order to replace the narrow functionality there. It also tags the function as a Jinja filter.

#61824 Add file.tidied flexibility

This PR extends the file.tidied state module to include the following functionality:

• Add exclusion regular expressions to allow for “all these things, but not these few” logic.

• Allow match and exclusion regular expressions to match on the whole path instead of just the file/directory name.

• Allow file.tidied to follow links instead of treating them as items to tidy.

• Add the ability to compare the change and modified times of the file to the current date instead of just access time.

#61764 Add Etag support to archive.extracted

This PR added use_etag as a new parameter to the archive.extracted state module. With this new functionality, a quick check with the remote web server is always performed for web (http or https) sources. The last Etag response header value is stored with the cached file, and is provided as the value of the If-None-Match request header with every “check-in”. If the stored Etag value matches the value calculated by the web server, the remote file is not retrieved. A 304 HTTP status code (Not Modified) is sent back to the minion instead. This allows new file versions to be retrieved as soon as they’re available on the web server without needing to download the whole file and provide a hash for comparison. Especially for large files, this represents a huge gain in efficiency and time to run states.

#61709 Add grains_refresh_pre_exec minion option

This PR adds the grains_refresh_pre_exec minion option, which allows for a minion to check its grains prior to the execution of any operation to see if they have changed and, if so, to inform the master of the new grains.

#61682 Add compare function to freezer module

This PR adds a compare function to the freezer module, which can display the diff between two named frozen states. The freezer.compare function shows output similar to that found in the changes dictionary for state output. This can be used to quickly determine changes between packages and repos in two frozen states.

#61564 Allow cp functions to derive saltenv from config

This PR allows the cp execution module functions to derive saltenv from config if not explicitly set as a keyword argument to the function.

#61550 Allow roll-up of duplicate IDs with different names in highstate output

This PR allows for rolling IDs with multiple names into a single state block for reporting purposes. Enabling the new state_compress_ids option consolidates the state data by ID and result (e.g. success or failure). The earliest start time is chosen for display, duration is aggregated, and the total number of names if shown in parentheses to the right of the ID. This functionality is most useful for any “_id” state_output options, such as terse_id.

#61531 Add __env__ substitution inside file and pillar root paths

This PR takes dynamic environments one step further than the previous implementation so __env__ can now be used in the file_roots and pillar_roots paths. It will be replaced with the actual saltenv or pillarenv and searched for states and/or data to provide to the minion.

#61503 Add new Jinja filters

Added some new and useful Jinja filters:

• dict_to_sls_yaml_params: Renders a formatted multi-line YAML string from a Python dictionary. Each key/value pair in the dictionary will be added as a single-key dictionary to a list that will then be sent to the YAML formatter.

• combinations: Allows the use of the combinations function from the itertools library as a Jinja filter.

• combinations_with_replacement: Allows the use of the combinations_with_replacement function from the itertools library as a Jinja filter.

• compress: Allows the use of the compress function from the itertools library as a Jinja filter.

• permutations: Allows the use of the permutations function from the itertools library as a Jinja filter.

• product: Allows the use of the product function from the itertools library as a Jinja filter.

• zip: Allows the use of the zip function as a Jinja filter.

• zip_longest: Allows the use of the zip_longest function from the itertools library as a Jinja filter.

#61391 Add Etag support for file.managed web sources

This PR added use_etag as a new parameter to the file.managed state module. With this new functionality, a quick check with the remote web server is always performed for web (http or https) sources. The last Etag response header value is stored with the cached file, and is provided as the value of the If-None-Match request header with every “check-in”. If the stored Etag value matches the value calculated by the web server, the remote file is not retrieved. A 304 HTTP status code (Not Modified) is sent back to the minion instead. This allows new file versions to be retrieved as soon as they’re available on the web server without needing to download the whole file and provide a hash for comparison. Especially for large files, this represents a huge gain in efficiency and time to run states.

#60991 Add percent success/failure of state runs in highstate summary output

Added state_output_pct as a new configuration option which adds success and failure percentages to the highstate output when enabled.

Announcing Salt Extension Modules for Prometheus!

As part of the collaboration between EITR and HRT, a returner module was created to take data from salt states runs and “return” it into a file formatted for Prometheus using the Text Exposition Format which rolls up state success and failure data. The intended use case for this module is to have distributed success/failure reporting from minions for unattended state or highstate runs.

This code was open-sourced as part of the Salt Extension Modules for Prometheus, which will become the new home for any Salt modules related to Prometheus. Some proposed upcoming functionality is an engine which provides metrics about the running Salt processes on a system. Stay tuned for more updates from the community!

Additional EITR Updates in Salt 3005

#62184 Deprecate all Azure cloud modules

This PR starts the process of deprecating all Azure cloud modules in the core of Salt Project. The old “Classic” API is going away in March 2023, and the Resource Manager modules are moving to a new Salt Extension.

#62069 Fixed backslash literal bytestring for file.replace

Fixed an issue where file.replace would throw a TypeError any time the backslash_literal parameter was used.

#62056 Fix salt-cloud sync_after_install ImmutableDict error

Fixed an issue where any attempts to run salt-cloud operations with the sync_after_install option would fail due to attempting to update the immutable DEFAULT_MASTER_OPTS dictionary.

#61938 Tiamat binary configurable pypath

This PR introduces the ability for the Tiamat-built Salt binary to install/use pip installed modules from a configurable location set via the TIAMAT_PIP_PYPATH environment variable.

#61910 Fix salt-ssh support for sudo with a password

This PR fixes an issue initially reported in 2013 by porting over some previous work by another community member to the master branch from the old develop branch and adding some tests. It allows salt-ssh to be used with accounts which need to provide a password for sudo escalation. Previously, salt-ssh could not be used without having root ssh or passwordless sudo.

Seth Cooper is a Technical Writer, Teacher, Builder of People and Teams, and Tinkerer. He’s spent close to 30 years trying to remove roadblocks and put out fires, so that other people can work smarter… not harder. Whether spending time changing the color of a header, rewriting a 300-page, six-year-out-of-date user manual, or creating hundreds of pages of new documentation for an application which isn’t complete yet, Seth has one goal in mind; creating a document that makes the users’ job easier to complete. After spending ten years in direct mail, writing commercials, and helping produce spam… He has spent the last 15 years trying to keep America and her allies safe. In his spare time, he is a Father, a Husband, a Lego-Guy, a gamer, and a lover of reptiles and aquariums.