| October 23, 2020
Passing Variables from an Event Rule to a Job Stream
Event Rules are an extension of Workload Automation (WA) capabilities that enable events occurring external to the scheduling environment to trigger actions on scheduling objects within WA. An ideal use of this capability is to detect the arrival of a file and then trigger an action to submit a Job Stream containing jobs to process the data contained in that file. This capability has been available for a while and is widely used. In this article, a hidden feature is explored where the name of the file and other properties related to the file are passed as variables to a Variable Table associated to the Job Stream being submitted as ac action where any Job within that Job Stream can retrieve those variables and process the data in the file.
Create an Event Rule with the Event, File Created, and Action, Submit Job Stream
Follow the steps below to create an Event Rule
- Login to the Dynamic Workload Console
- Navigate to Design à Create Event Rules
- Click on Create new à Event Rules
- Click on the field, <not_named> and the General Info panel is displayed on the right
- Click on the Name field, and enter the name, IF_FILE_EXISTS_PROCESS_FILE
- Click on Save as draft to disable it
- Click on Add Events à File Monitor à File created (click on + sign) à Close
- Click on the event just added, the Properties panel opens on the right
- Fill in the values in the fields as shown below:
a. File name: /tmp/datafile.*.txt An existing file with a matching name will not satisfy the condition, a new file with matching name must be created.
b. Sample interval: 300 Indicates how often in seconds to check for the file
c. Workstation: EEL
10. Click on Add Actions à IBM Workload Scheduler actions à Submit job stream
11. Click on the Action just added, the properties panel opens on the right
12. Fill in the values in the fields as shown below:
a. Job stream name: PROCESS_FILEThe name of the Job Stream to be submitted that contains jobs to process the
contents of the file.
b. Workstation name: EEL
11. Click on the Action just added, the properties panel opens on the right
12. Fill in the values in the fields as shown below:
a. Job stream name: PROCESS_FILEThe name of the Job Stream to be submitted that contains jobs to process the
contents of the file.
b. Workstation name: EEL
c. Custom parameter 1:
Click on the Variables icon on the right and choose fileCrtEvt1 à File name
FILE_NAME=%{fileCrtEvt1.FileName} will be entered
The value of File name will be passed to the Variable, FILE_NAME
FILE_NAME=%{fileCrtEvt1.FileName} will be entered
The value of File name will be passed to the Variable, FILE_NAME
d. Custom parameter 2: Similarly, click on Variables icon on the right and choose any other variable that needs to be
passed as Variables, e.g.
e. Custom parameter 3: WORKSTATION=%{fileCrtEvt1.Workstation}
f. Custom parameter 4: TIME_STAMP=%{fileCrtEvt1.TimeStamp}
g. Custom parameter 5: HOSTNAME=%{fileCrtEvt1.Hostname}
h. Custom parameter 6: IP_ADDRESS=%{fileCrtEvt1.IPAddress}
i. Custom parameter 7: EVENT_RULE_ID=%{fileCrtEvt1.EventRuleId}
passed as Variables, e.g.
e. Custom parameter 3: WORKSTATION=%{fileCrtEvt1.Workstation}
f. Custom parameter 4: TIME_STAMP=%{fileCrtEvt1.TimeStamp}
g. Custom parameter 5: HOSTNAME=%{fileCrtEvt1.Hostname}
h. Custom parameter 6: IP_ADDRESS=%{fileCrtEvt1.IPAddress}
i. Custom parameter 7: EVENT_RULE_ID=%{fileCrtEvt1.EventRuleId}
13. Click on the selection box next to the event Rule name and click on Save on the menu bar.
14. Click on the X to close the successful saved message.
15. The following is the Event Rule definition is composer format.
14. Click on the X to close the successful saved message.
15. The following is the Event Rule definition is composer format.
<?xml version=”1.0″?>
<eventRuleSet xmlns:xsi=”https://www.w3.org/2001/XMLSchema-instance”
xmlns=”https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules”
xsi:schemaLocation=”https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules/EventRules.xsd”>
<eventRule name=”IF_FILE_EXISTS_PROCESS_FILE” ruleType=”filter” isDraft=”no”>
<eventCondition name=”fileCrtEvt1″ eventProvider=”FileMonitor” eventType=”FileCreated”>
<scope>
/TMP/DATAFILE.*.TXT ON EEL
</scope>
<filteringPredicate>
<attributeFilter name=”FileName” operator=”eq”>
<value>/tmp/datafile.*.txt</value>
</attributeFilter>
<attributeFilter name=”SampleInterval” operator=”eq”>
<value>300</value>
</attributeFilter>
<attributeFilter name=”Workstation” operator=”eq”>
<value>EEL</value>
</attributeFilter>
</filteringPredicate>
</eventCondition>
<action actionProvider=”TWSAction” actionType=”sbs” responseType=”onDetection”>
<scope>
SBS EEL#PROCESS_FILE
</scope>
<parameter name=”JobStreamWorkstationName”>
<value>EEL</value>
</parameter>
<parameter name=”Parm_1″>
<value>FILE_NAME=%{fileCrtEvt1.FileName}</value>
</parameter>
<parameter name=”Parm_2″>
<value>SAMPLE_INTERVAL=%{fileCrtEvt1.SampleInterval}</value>
</parameter>
<parameter name=”Parm_7″>
<value>EVENT_RULE_ID=%{fileCrtEvt1.EventRuleId}</value>
</parameter>
<parameter name=”Parm_5″>
<value>HOSTNAME=%{fileCrtEvt1.Hostname}</value>
</parameter>
<parameter name=”Parm_6″>
<value>IP_ADDRESS=%{fileCrtEvt1.IPAddress}</value>
</parameter>
<parameter name=”Parm_3″>
<value>WORKSTATION=%{fileCrtEvt1.Workstation}</value>
</parameter>
<parameter name=”JobStreamName”>
<value>PROCESS_FILE</value>
</parameter>
<parameter name=”Parm_4″>
<value>TIME_STAMP=%{fileCrtEvt1.TimeStamp}</value>
</parameter>
</action>
</eventRule>
</eventRuleSet>
<eventRuleSet xmlns:xsi=”https://www.w3.org/2001/XMLSchema-instance”
xmlns=”https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules”
xsi:schemaLocation=”https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules https://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules/EventRules.xsd”>
<eventRule name=”IF_FILE_EXISTS_PROCESS_FILE” ruleType=”filter” isDraft=”no”>
<eventCondition name=”fileCrtEvt1″ eventProvider=”FileMonitor” eventType=”FileCreated”>
<scope>
/TMP/DATAFILE.*.TXT ON EEL
</scope>
<filteringPredicate>
<attributeFilter name=”FileName” operator=”eq”>
<value>/tmp/datafile.*.txt</value>
</attributeFilter>
<attributeFilter name=”SampleInterval” operator=”eq”>
<value>300</value>
</attributeFilter>
<attributeFilter name=”Workstation” operator=”eq”>
<value>EEL</value>
</attributeFilter>
</filteringPredicate>
</eventCondition>
<action actionProvider=”TWSAction” actionType=”sbs” responseType=”onDetection”>
<scope>
SBS EEL#PROCESS_FILE
</scope>
<parameter name=”JobStreamWorkstationName”>
<value>EEL</value>
</parameter>
<parameter name=”Parm_1″>
<value>FILE_NAME=%{fileCrtEvt1.FileName}</value>
</parameter>
<parameter name=”Parm_2″>
<value>SAMPLE_INTERVAL=%{fileCrtEvt1.SampleInterval}</value>
</parameter>
<parameter name=”Parm_7″>
<value>EVENT_RULE_ID=%{fileCrtEvt1.EventRuleId}</value>
</parameter>
<parameter name=”Parm_5″>
<value>HOSTNAME=%{fileCrtEvt1.Hostname}</value>
</parameter>
<parameter name=”Parm_6″>
<value>IP_ADDRESS=%{fileCrtEvt1.IPAddress}</value>
</parameter>
<parameter name=”Parm_3″>
<value>WORKSTATION=%{fileCrtEvt1.Workstation}</value>
</parameter>
<parameter name=”JobStreamName”>
<value>PROCESS_FILE</value>
</parameter>
<parameter name=”Parm_4″>
<value>TIME_STAMP=%{fileCrtEvt1.TimeStamp}</value>
</parameter>
</action>
</eventRule>
</eventRuleSet>
16. Select the Job Stream Name and choose Select an Action à Submit Job Stream into Current Plan
Create a Variable Table, Job, and Job Stream with to process the file
Follow the steps below to create the objects
- Login to the Dynamic Workload Console
- Navigate to Design à Manage Workload Definitions
- Click on Create New à Variable Table
- In the General tab, enter in the Name field, PASS_VAR
- The variable to hold the properties of the file monitored are created and referenced internally, as such, no Variables need to be created.
- Click on the Save icon to save the Variable table.
- The following is the Variable Table definition is composer format.
VARTABLE PASS_VAR
MEMBERS
END
8. Click on Create New à Job Definition à UNIX
9. In General tab, enter in the Name filed, FILE_RECEIVED_N_PROCESSING
10. In the Task tab, Select Script and enter in the Script name field:
/bin/echo -e “\nThe properties of the file received are: \n\nFile Name=^FILE_NAME^ \nSample Interval=^SAMPLE_INTERVAL^ \nWorkstation=^WORKSTATION^ \nTime stamp=^TIME_STAMP^ \nHostname=^HOSTNAME^ \nIP Address=^IP_ADDRESS^ \nEvent Rule ID=^EVENT_RULE_ID^ \n\nThe next job in the Job Stream can now start processing the file.\n”
11. Notice that the same variable names are used to retrieve the values as they are assigned to the respective properties of the file monitored in the Event Rule, IF_FILE_EXISTS_PROCESS_FILE.
12. Click on the save icon to save the Job
13. The following is the Variable Table definition is composer format.
EEL#FILE_RECEIVED_N_PROCESSING
SCRIPTNAME “/bin/echo -e \”\nThe properties of the file received are: \n\nFile Name=^FILE_NAME^ \nSample Interval=^SAMPLE_INTERVAL^ \nWorkstation=^WORKSTATION^ \nTime stamp=^TIME_STAMP^ \nHostname=^HOSTNAME^ \nIP Address=^IP_ADDRESS^ \nEvent Rule ID=^EVENT_RULE_ID^ \n\nThe next job in the Job Stream can now start processing the file.\n\””
STREAMLOGON iwadmin
DESCRIPTION “A Job to demo how to retrieve variables passed from an Event Rule”
TASKTYPE UNIX
RECOVERY STOP
SCRIPTNAME “/bin/echo -e \”\nThe properties of the file received are: \n\nFile Name=^FILE_NAME^ \nSample Interval=^SAMPLE_INTERVAL^ \nWorkstation=^WORKSTATION^ \nTime stamp=^TIME_STAMP^ \nHostname=^HOSTNAME^ \nIP Address=^IP_ADDRESS^ \nEvent Rule ID=^EVENT_RULE_ID^ \n\nThe next job in the Job Stream can now start processing the file.\n\””
STREAMLOGON iwadmin
DESCRIPTION “A Job to demo how to retrieve variables passed from an Event Rule”
TASKTYPE UNIX
RECOVERY STOP
14. Click on Create New à Job Stream
15. In General tab, enter in the Name filed, PROCESS_ FILE
16. Scroll down to the Variable table field and enter PASS_VAR to associate the variable table to the Job Stream.
17. In the Details view, right click on the Jobs and choose Add Jobs
18. Enter in the Job definition field, FILE_RECEIVED_N_PROCESSING, and click on Add
19. Click on the save icon to save the Job Stream
15. In General tab, enter in the Name filed, PROCESS_ FILE
16. Scroll down to the Variable table field and enter PASS_VAR to associate the variable table to the Job Stream.
17. In the Details view, right click on the Jobs and choose Add Jobs
18. Enter in the Job definition field, FILE_RECEIVED_N_PROCESSING, and click on Add
19. Click on the save icon to save the Job Stream
20. The following is the Variable Table definition is composer format.
SCHEDULE EEL#PROCESS_FILE
DESCRIPTION “A Job Stream to demo how to retrieve a variable passed from an Event Rule”
VARTABLE PASS_VAR
:
EEL#FILE_RECEIVED_N_PROCESSING
END
DESCRIPTION “A Job Stream to demo how to retrieve a variable passed from an Event Rule”
VARTABLE PASS_VAR
:
EEL#FILE_RECEIVED_N_PROCESSING
END
Monitor the execution of the Event Rule
Follow the steps below to create the file, monitor the execution of the Event Rule, submission of the Job Stream, and the Job log to view the variable passed from the Event Rule to the Job Stream.
1. Login to server where the file is monitored and create a file to simulate the arrival of the file cd /tmp
touch datafile.100120201202.txt
2. On the MDM, monitor the contents of the messages.log cd /opt/ibm/iwa/iws/TWSDATA/stdlist/appserver/engineServer/logs
tail -f messages.log
3. Once messages are displayed about the Event Rule being executed, login to the Dynamic Workload Console and navigate to monitoring the Job stream.
4. The Job Stream, PROCESS_FILE, is submitted.
5. Go into Jobs view and monitor the status of the Job, FILE_RECEIVED_N_PROCESSING
6. Once it is successful, view the Job Log.
7. Notice that the Variables are successfully passed from the Event Rule to the Job Stream and the Job is able to retrieve them from the Variable Table.
8. Also notice that the Variable Table has not been populated with the variables but were mapped to it internally.
8. Also notice that the Variable Table has not been populated with the variables but were mapped to it internally.
VARTABLE PASS_VAR
MEMBERS
END
MEMBERS
END
The same process can be used to create an Event Rule with Modification completed file monitor event.
Want to learn more? Write to me at Sajjad.Kabir@hcl.com or schedule a demo.
Want to learn more? Write to me at Sajjad.Kabir@hcl.com or schedule a demo.
Comment wrap
