BaseEnrtRecipe class

class lnst.Recipes.ENRT.BaseEnrtRecipe.BaseEnrtRecipe(**kwargs)

Bases: BaseSubConfigMixin, BaseMeasurementGenerator, PingTestAndEvaluate, Recipe

Base Recipe class for the ENRT recipe package

This class defines the shared test method defining the common test procedure in a very generic way. This common test procedure involves a single main test_wide configuration that is different for every specific scenario. After the main configuration there is usually a loop of several minor sub configrations types that can take different values to slightly change the tested use cases.

Finally, for each combination of a test_wide + sub configuration we do a several ping connection test and several performance measurement tests.

test_wide and sub configurations are implemented with context manager methods which ensure that if any exceptions are raised (for example because of a bug in the recipe) that deconfiguration is called.

Both test_wide and sub configurations are to be implemented in different classes, the BaseEnrtRecipe class only defines the common API and the base versions of the relevant methods.

Test wide configuration is implemented via the following methods:

Sub configurations are mixed into classes defining the specific scenario that is being tested. Various sub configurations are implemented as individual Python Mixin classes in the ConfigMixins package. These make use of Pythons collaborative inheritance by calling the super function in a specific way. The “machinery” for that is defined in the BaseSubConfigMixin class. It is then used in this class from the test method loop.

Parameters:

  • driver (StrParam (default “ixgbe”)) – The driver parameter is used to modify the hw network requirements, specifically to request Devices using the specified driver. This is common enough in the Enrt recipes that it can be part of the Base class.

  • ip_versions (Tuple[Str] (default ("ipv4", "ipv6"))) – Parameter that determines which IP protocol versions will be tested.

  • ping_parallel (BoolParam (default False)) – Parameter used by the generate_ping_configurations generator. Tells the generator method to create PingConf objects that will be run in parallel.

  • ping_bidirect (BoolParam (default False)) – Parameter used by the generate_ping_configurations generator. Tells the generator method to create PingConf objects for both directions between the ping endpoints.

  • ping_count (IntParam (default 100)) – Parameter used by the generate_ping_configurations generator. Tells the generator how many pings should be sent for each ping test.

  • ping_interval (FloatParam (default 0.2)) – Parameter used by the generate_ping_configurations generator. Tells the generator how fast should the pings be sent in each ping test.

  • ping_psize (IntParam (default None)) – Parameter used by the generate_ping_configurations generator. Tells the generator how big should the pings packets be in each ping test.

  • net_perf_tool (BaseFlowMeasurement (default IperfFlowMeasurement)) – Parameter used by the generate_perf_configurations generator to create a PerfRecipeConf object. Specifies a network flow measurement class that accepts PerfFlow objects and can be used to measure those specified flows

  • cpu_perf_tool (BaseCPUMeasurement (default StatCPUMeasurement)) – Parameter used by the generate_perf_configurations generator to create a PerfRecipeConf object. Specifies a cpu measurement class that can be used to measure CPU utilization on specified hosts.

  • perf_iterations (IntParam (default 5)) – Parameter used by the generate_perf_configurations generator. To specify how many times should each performance measurement be repeated to generate cumulative results which can be statistically analyzed.

  • perf_test_simulation (BoolParam (default False)) – Parameter that will switch the performance testing into a simulation mode only - no measurements will actually be started and they’ll simply generate 0 value measurement results as if they ran

test()

Main test loop shared by all the Enrt recipes

The test loop involves a single application of a test_wide configuration, then a loop over multiple sub configurations that involves:

  • creating the combined sub configuration of all available SubConfig Mixin classes via generate_sub_configurations

  • applying the generated sub configuration via the _sub_context context manager method

  • running tests

  • removing the current sub configuration via the _sub_context context manager method

test_wide_configuration()

Creates an empty EnrtConfiguration object

This is again used in potential collaborative inheritance design that may potentially be useful for Enrt recipes. Derived classes will each individually add their own values to the instance created here. This way the complete test wide configuration is tracked in a single object.

Returns:

returns a config object that tracks the applied configuration that can be used during testing to inspect the current state and make test decisions based on it.

Return type:

EnrtConfiguration

Example:

class Derived:
    def test_wide_configuration():
        config = super().test_wide_configuration()

        # ... configure something
        config.something = what_was_configured

        return config
test_wide_deconfiguration(config)

Base deconfiguration method.

In the base class this should maybe only check if there’s any leftover configuration and warn about it. In derived classes this can be overriden to take care of deconfiguring what was configured in the respective test_wide_configuration method.

Example:

class Derived:
    def test_wide_deconfiguration(config):
        # ... deconfigure something
        del config.something #cleanup tracking

        return super().test_wide_deconfiguration()
describe_test_wide_configuration(config)

Describes the current test wide configuration

Creates a new result object that contains the description of the full test wide configuration applied by all the test_wide_configuration methods in the class hierarchy.

The description needs to be generated by the generate_test_wide_description method. Additionally the description contains the state of all the parameters and their values passed to the recipe class instance during initialization.

generate_test_wide_description(config: EnrtConfiguration)

Generates the test wide configuration description

Another class inteded to be used with the collaborative version of the super method to cumulatively desribe the full test wide configuration that was applied through multiple classes.

The base class version of this method creates the initial list of strings containing just the header line. Each string added to this list will later be printed on its own line.

Returns:

list of strings, each representing a single line

Return type:

List[str]

Example:

class Derived:
    def generate_sub_configuration_description(config):
        desc = super().generate_sub_configuration_description(config)
        desc.append("Configured something: {}".format(config.something))
        return desc
do_tests(recipe_config)

Entry point for actual tests

The common scenario is to do ping and performance tests, however the method can be overriden to add more tests if needed.

do_ping_tests(recipe_config)

Ping testing loop

Loops over all various ping configurations generated by the generate_ping_configurations method, then uses the PingRecipe methods to execute, report and evaluate the results.

do_perf_tests(recipe_config)

Performance testing loop

Loops over all various perf configurations generated by the generate_perf_configurations method, then uses the PerfRecipe methods to execute, report and evaluate the results.

generate_ping_configurations(config)

Base ping test configuration generator

The generator loops over all endpoint pairs to test ping between (generated by the generate_ping_endpoints method) then over all the selected ip_versions and finally over all the IP addresses that fit those criteria.

Returns:

list of Ping configurations to test in parallel

Return type:

List[PingConf]

generate_ping_endpoints(config)

Generator for ping endpoints

To be overriden by a derived class.

Returns:

list of device pairs

Return type:

List[Tuple[Device, Device]]

generate_perf_configurations(config)

Base perf test configuration generator

The generator loops over all flow combinations to measure performance for (generated by the generate_flow_combinations method). In addition to that during each flow combination measurement we add CPU utilization measurement to run on the background.

Finally for each generated perf test configuration we register measurement evaluators based on the cpu_perf_evaluators and net_perf_evaluators properties.

Returns:

list of Perf test configurations

Return type:

List[PerfRecipeConf]

register_perf_evaluators(perf_conf)

Registrator for perf evaluators

The registrator loops over all measurements collected by the perf tests to pick evaluator based on the measurements using the evaluator_by_measurement method.

Once appropriate evaluator is picked, it is registered to the PerfRecipeConf.

evaluator_by_measurement(measurement)

Selector for the evaluators based on measurements

The selector looks at the input measurement to pick appropriate evaluator.

Returns:

list of Result evaluators

Return type:

List[BaseResultEvaluator]

property cpu_perf_evaluators

CPU measurement evaluators

To be overriden by a derived class. Returns the list of evaluators to use for CPU utilization measurement evaluation.

Returns:

a list of cpu evaluator objects

Return type:

List[BaseEvaluator]

property net_perf_evaluators

Network flow measurement evaluators

To be overriden bby a derived class. Returns the list of evaluators to use for Network flow measurement evaluation.

Returns:

a list of flow evaluator objects

Return type:

List[BaseEvaluator]

class lnst.Recipes.ENRT.BaseEnrtRecipe.EnrtConfiguration

Container object for configuration

Stores configured devices and IPs configured on them. Can also be used to store any values relevant to configuration being applied during the lifetime of the Recipe.