Matrix42 Worker Engine

Matrix42 Worker Workflow Engine architecture and worker management: default worker, installation process and update.

+ More

Table of Contents

Overview Architecture Security Authentication Token Trusted Worker How to use the ESMP with an external URL Worker and wrong DNS configuration Default Matrix42 Worker Installing Worker Updating Worker Manage Matrix42 Workers Using Matrix42 Workers as Data Gateway Data Gateways before 26.1 Configure Proxy for Remote Worker Disabling Worker Applications Versions before 26.1 Running 3d Party extensions in Workflow Worker Other information Legacy Information for versions before 12.1.1 Matrix42 Workers & Workflows Matrix42 Workers & Data Gateways Default Data Gateway Tasks Migrate remote Data Gateway to remote Worker

Overview

The Matrix42 Worker is a dedicated process of the Workspace Management application designed to scale the execution of asynchronous background tasks.

Prior to version 26.1, the Worker was responsible for a limited set of tasks, such as hosting the Workflow Engine for workflow execution, email notifications, and DataGateway tasks.

Starting with version 26.1, the Matrix42 Worker was completely redesigned as the Functional Worker. It is now responsible for the full range of background processing tasks within the ESM platform, including workflows, engines, PowerShell executions, compliance rules, and more. In addition, the Functional Worker now provides a unified hosting environment for the execution of all types of custom business logic and platform customizations.

Architecture

Matrix42 Worker is a Windows Service installed by default on the Application Server. Optionally, it can also be installed on additional machines within the organization. In container-based ESM deployments, the Worker is hosted in a dedicated container.

The Matrix42 Worker Engine is based on a message queue architecture. Commands such as workflow executions are first added to the message queue (the production database table QueueTask). Matrix42 Workers continuously monitor the queue and process messages one by one.

For more details about the Worker architecture introduced in version 26.1, see Functional Worker Architecture.

Such an approach solves the following challenges:

  • Scalability
    Add additional Worker instance to the Cluster if the Queue starts growing and the current number of active Workers are not coping with the Workflow commands processing. Using such solutions as Kubernetes allows automatic scaling when additional instances of Workers are created or disposed of automatically, depending on the workload.
  • Reliability
    The Queue has high availability, which guarantees the Workflow command is stored in the Queue. The messages stay in Queue until they are successfully processed. 

Security

Authentication Token

Matrix42 Workers uses the Token Authentication with API Token to set up connectivity with the Application Server. The API Token is provided on Worker installation or automatically generated for the Default Worker (Worker installed on Application Server). The provided Token is encrypted with the Machine Key and stored in the Matrix42.Worker.host.exe.config file. The API Token is sent to the Application Server for checking the validity on Matrix42 Worker process start. If the Token is valid and not expired the Server issues the short-time Access Token which is used for any other operation with the Server.

Trusted Worker

Trusted Worker mechanism was introduced to guarantee data security. It is critical for the Matrix42 Worker to have a direct connection to the SQL server, as it dramatically improves the performance.

Generally, the connection string is sent to the worker. This approach is acceptable only when the integrated security is used. 

For the cases when the SQL authentication is used (when connection string keeps password and username ) the data security cannot be guaranteed. Therefore the Trusted Worker mechanism is introduced, also Worker Capability “Server” automatically assigned to Worker

The worker can be installed anywhere: in the local network, behind a DMZ (a physical or logical subnetwork that contains and exposes an organization's external-facing services to an untrusted, usually larger, network such as the Internet). 

The administrator must explicitly enable the Trusted Worker attribute when SQL authentication (SA password) is used and the Matrix42 Worker service runs on remote machines within the local network or behind a DMZ, rather than on the Application Server itself.

A Worker polls tasks from the queue only when its supported capabilities match the capabilities required by the task. For example, the “Send Email” task requires the Server and Email capabilities, therefore only a Worker providing both capabilities can retrieve and execute this task.

Before 26.1, the following logic is used for Task dispatching to Workers:

  • Only Trusted Workers can execute both Workflow tasks and assigned DataGateway tasks.

All worker instances that run on the Application Server are marked as trusted by default.

 
  • Not Trusted Workers: the workers that are marked as not trusted can execute only the tasks of the manually assigned DataGateway. By default, the DataGateway is not assigned. Other tasks (like workflows or later on send e-mail tasks as well) are processed by the Trusted Worker. 

How to use the ESMP with an external URL

(discoveryHostName = external domain url)

The ESMP carries out its internal communication via HTTP(S). So it can happen if the following configuration is not observed, that e.g. the internal communication via a proxy server goes outwards and comes back inwards.

This can happen even without a proxy server if the DNS is not configured correctly for the ESMP system.

For example, the following configuration of the ESMP:

  • my.company.com (with external IP address 77.77.77.77)
  • host.mylocaldomain.com (192.168.1.111)

As discovery proxy name the external address was entered in the .\bin\sps.config, which is correct.

Now it is important, when an internal query of the system happens, that the DNS server does not resolve the external IP address, but the internal one.

For this purpose create a DNS entry on the internal DNS server for my.company.com, so that it is resolved with the IP address 192.168.1.111.

You can check this case as follows:

  1. Log on to the Application Server;
  2. Open a CMD box;
  3. Execute the following command:
ping my.company.com
  • ❌ External IP: if the external IP is resolved here, there is a configuration problem for the ESMP.
  • ✔️ Internal IP: if the system resolves the internal IP, it's fine.

Proxy

It is also important that the communication does not run over a proxy, because otherwise the communication may go to the outside and other errors in the ESMP may occur.

You can check this as follows:

  1. Log on to the Application Server under the ESMP User;
  2. Check the proxy configuration. For example, open Internet Explorer and check the connection settings (proxy) under Settings.
  3. If you use a ProxyAutoConfiguration File, make sure that the DiscoveryHostName is excluded for the proxy.

Also, check whether a generally set proxy is stored in the system. You can do this as follows:

  1. Press the Windows button and enter "cmd". 
  2. Open the search result. 
  3. Execute the following command:
netsh winhttp show proxy

If a proxy is displayed here, please configure the exclusion of the discoveryHostName.

Worker and wrong DNS configuration

As soon as the setup described above is not done and an external IP address is resolved, the Worker is not considered trustworthy and local and denies the service. 

So it is not possible to execute workflows with the worker or to send e-mails (partly) from the system.

Default Matrix42 Worker

The Product Setup automatically installs the Matrix42 Worker on the Application Server, or, in case of container-based ESM deployments, the Worker is hosted in a dedicated container. It guarantees the overall System is operable and able to handle all commands assigned for Workers right after the Setup is over and no other extra activities are required.

If the Matrix42 Worker is already installed, then the Setup validates the API Token assigned to Worker, and in case it is not valid anymore, or even expires within the next month, the Setup reissues a new API Token to avoid the cases when the connection of the Worker with Server is lost due to expired token.

Installing Worker

The System allows installing an unlimited number of Matrix42 Workers. Adding additional Worker to a Farm allows to scale the System and make it more reliable and adjustable for various infrastructure challenges like unexpected growth of the system resource usage.

To install worker, in Administration application, open the management area Services & Processes → Workflows → Workers:

  1. Run "Download Worker" action: the System automatically generates a new archive that already includes the basic configuration settings like Application Server URL and name of the Service Account.
    Due to CAS Policy, the downloaded Worker.zip archive can be flagged by Windows as a web application, even if it resides on the local computer. To change this:
    • right-click on the ZIP file
    • open properties
    • apply the unblock option and continue the installation: 
  2. Extract the received archive file on the computer where you want to install the Matrix42 Worker.
  3. Run "InstallWorker.cmd": the batch file runs the signed PowerShell script. Depending on the Domain Policies, the execution of the Powershell file can be rejected due to an unknown publisher. In this case, make sure the certificate used by Configure.ps1 is installed into the Trusted Publishers Certificate Store on the Local Machine.
  4. Proceed with the installation steps:
    1. Service Name
      Verify if the Worker service name is correct. Different Worker instances running on the same machine should have different service names.
    2. Server URL
      Verify if the Server URL is correct, otherwise, provide the URL of the Application Server.
    3. Valid API token
      The Worker requires the valid API token to authenticate with the Server. Please provide the valid token for the user with the Administrative rights. If you do not have a prepared token, it can be generated. See "Generate API Token" for more details. 
      The provided token is encoded with the machine-key and stored in the config file.
    4. Server connection
      The System checks the validity of the provided credentials and sets up the connection with the Server.
      If you have a connectivity problem and got a message the server cannot be contacted and the trusted SSL/TLS secure channel cannot be established, then please check that the Application Server certificate is installed on the local machine to Trusted Root Authorities.
    5. Set the Service Account
      Matrix42 Worker uses Integrated Security to work with the Product databases. Therefore the Service account should have the db_ownerpermissions to the databases. By default, the installation suggests the Service account which is used on Application Server, but if necessary, it can be changed. Pay attention, that this is not related to Workers that acts as Data gateway, as they don't operate with DataBase directly

If you want to install a Matrix42 Worker in an external domain and want to use this Worker as a DataGateway, please use for "Set Service Account" a user from that domain that has the rights to run power shell scripts. This user will be used to run the Matrix42 Worker windows service.

 
  1. The installation registers and starts "Matrix42 Worker" Windows service.

Multiple Matrix42 Workers can be installed on the same computer. For that provide the unique name for Windows Service during the installation or specified it in the Worker config file before running the installation script

 

Updating Worker

Matrix42 Worker, in general, does not require any manual operation for updates, as it is automatically updated any time the relevant resource on Application Server changes. It means that as soon as the Application Server is updated with a new version or the patch, the next moment all related Matrix42 Workers will be automatically updated.

Nevertheless, there are a few cases that require manual update operations.

Updating API Token

On installing the Matrix42 Worker the API Token is provided, which is used for communication with the Application Server. The token is encrypted with the machine key and stored in the Config file. When the API token becomes invalid (either removed from Server or expires) the Matrix42 Worker is not able to communicate with the Server. To troubleshoot the issue, the new API Token needs to be provided for the Matrix42 Worker.

  1. Generate a new Token. See "Generate API Token" for more details.
  2. In the Matrix42 Worker Application folder start "UpdateToken.cmd" and follow instructions.
  3. Windows service "Matrix42 Worker" will be restarted to apply configuration changes.

Manage Matrix42 Workers

Once the Matrix42 Worker is installed it automatically registers itself in the Application. All registered Matrix42 Workers, as well as their current statuses, can be found in Administration application in the management area Services & Processes → Workflows → Workers:

Using Matrix42 Workers as DataGateway

To use Matrix42 Workers as DataGateway, in Administration application, open Services & Processes → Workflows → Workers → choose Worker Capabilities.

Create a new Worker capability that represents the specific environment or network location of the machine where the Worker is running. For example, if the Worker runs inside the Imagoverum company DMZ, create a capability such as “Imagoverum VPN”

Then open the Workers management area, locate the corresponding Worker, and assign the created capability in the Worker edit dialog.

DataGateways before 26.1

The reference can be changed or removed in the edit mode. Click Edit action:

  1. In DataGateway, choose the necessary option or remove the DataGateway:

    DataGateway options:
    • Set: by default, the App Server DataGateway is assigned for the Workers running on App Server. This reference defines the relation between the DataGateway and the Worker which is currently executing the tasks that were previously assigned to the DataGateway. 
    • Not set: if the relation is not set the Worker executes only the workflow tasks.
  2. Click Done;
  3. Restart the Matrix42 Worker service to apply the configuration changes. DataGateway does not need any extra configuration and stops executing the tasks that were reassigned to the Matrix42 Workers.

The edited Matrix24 Worker will execute the tasks that were previously handled by the specified DataGateway.

You can assign more than one Matrix42 Worker to the same DataGateway.

 

 

Configure Proxy for Remote Worker

In case the Remote Worker is installed in a Client Private Network (DMZ) with communication routed through a proxy server it is recommended to bypass the Proxy server for the Worker requests to the Application Server hosted in the Cloud. To configure the direct Worker connection with App Sever, bypassing the Proxy Server, add the <defaultProxy> section to the Matrix42.Worker.Host.exe.config:

<system.net>
   <defaultProxy>
      <proxy
         usesystemdefault="True"
         proxyaddress="http://<Proxy FQDN/IP>:<Proxy Port>"
         bypassonlocal="True"   />
         <bypasslist>
            <add address="[a-z]+\.<Domain>\.<TLD>$" />
            </bypasslist>
         </defaultProxy>
 </system.net>

Proxy tag has the information of the proxy configuration in it. The bypasslist lists all exceptions where the proxy must no be used. (In the example it bypasses all subdomains of a given domain.)
For more information you can check out the documentation from Microsoft: https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/network/defaultproxy-element-network-settings

Disabling Worker Applications

Starting with version 26.1, background task dispatching to a specific Worker is based entirely on the set of Worker Capabilities assigned to that Worker. If the Worker does not provide at least one of the required capabilities, the task will not be dispatched to it.

Versions before 26.1   

By default, the Worker defines which tasks to run according to the Worker configuration. Optionally, starting from ESMP v.11.0.1 it is possible to limit the type of tasks that are executed on the Worker. Use this option if you need a Worker that runs only specific tasks from the allowed applications. 

Matrix42.Worker.Host.exe.config file allows disabling the following applications run by Worker:

  • Email Engine
  • DataGateway
  • Workflow Engine

To disable the applications modify the configuration file according to the example:

<add key="DisableWorkerApps" value="Email Engine,DataGateway,Workflow Engine" />

In this configuration, the Worker will not run the tasks from the enumerated applications.

The following example disables running tasks from the E-mail Engine application but keeps processing DataGateway (if configured) and Workflow Engine tasks:

<add key="DisableWorkerApps" value="Email Engine" />

Running 3d Party extensions in Workflow Worker

Sometimes using the 3d part components running in Workflow on Remote Worker could lead to the error that one of the related assembly cannot be resolved on Worker like "Microsoft.Practices.Unity.ResolutionFailedException: Resolution of the dependency failed, type =XXX"

To workaround such issue the missing Assemblies need to be placed in the folder:

<Application Folder>\Worker\Apps\Matrix42.Workflows.WorkerApp

Other information

The following sections of the preview show additional worker information:

  • Worker Applications status of the Data Providers and Workflows 
  • Database connections are necessary for workflows processing:
Databases Possible Status
Production
  • Direct Access
  • Web API
  • None
Instance Store
  • Direct Access
  • None
Monitoring
  • Direct Access
  • None

 

 

 

 

 

 

 

Legacy Information for versions before 12.1.1

Matrix42 Workers & Workflows

In contrast to AppFabric execution, the Matrix42 Worker is an external Windows process and all Business Components and other resources hosted on Application Server are not accessible on Worker. That limitation puts additional requirements to Workflow Activities running on Worker. For example, if the Activity requires executing Business Logic, it must be triggered over the Web Service. 

Delivered with the Product standard Workflow Activities that consider Matrix42 Worker specifics and can be executed on Worker are marked as Worker Compatible. The Workflow can be executed on Matrix42 Worker only when all the Workflow Activities of the Workflow are Worker Compatible.

Matrix42 Workers & DataGateways

Since 10.0.1 release, Matrix42 Worker becomes the universal approach for executing asynchronous background tasks in the Enterprise Service Management Platform. It also can play the role of the data gateway, or in other words, can be installed remotely and carry out the tasks of collecting data for Data Providers (as Matrix42 DataGateway does).

Default DataGateway Tasks

Previously, all tasks were executed by the DataGateway. Since 10.0.1 release the Matrix42 Workers that are run on the Application Server are assigned with App Server default DataGateway by default. Thus the following tasks are executed by the Matrix42 Workers instead of the App Server DataGateway by default:

Further, this list will be completed and all available tasks that are still executed by App Server default DataGateway will be run by Matrix42 Workers while the Matrix42 DataGateway service will be discontinued.

 

Migrate remote DataGateway to remote Worker

To make a remote DataGateway running on Worker:

  1. Install Worker on the machine where the DataGateway is installed
  2. In the Administration application, open Services & Processes → Workflows → Workers → choose installed Worker and assign the remote DataGateway
  3. Open a remote desktop connection to your Application Server, on which the Enterprise Service Management Platform is installed. Open Windows Explorer and navigate to the following directory:
<Application Folder>\DataGateway\Host

The default directory for a ESMP installation would be:

C:\Program Files (x86)\Matrix42\Matrix42 Workplace Management\DataGateway\Host

It might differ in individual cases so adjust the path accordingly.

 
  1. Click into the path field at the top of the window and copy the path to the clipboard.
  2. Open a CMD window as Administrator and run the following command: cd/
  3. Then run the following command if you have a default directory for a ESMP installation or adjust the path accordingly: cd C:\Program Files (x86)\Matrix42\Matrix42 Workplace Management\
  4. Once you have navigated to this directory, run the following command: Matrix42.DataGateway -uninstall
  5. When the uninstall has been confirmed, open a Services window and check that the Windows service "Matrix42 DataGateway" is no longer present. 
  6. A reboot of the Application Server is recommended and should be conducted during the next suitable maintenance window.

Related Articles