User Tools

Site Tools


pub:workwithmergeengine

[ Up To Index ]

Work With Merge Engines

OverView

Merge engines are processes that watch outqueues. When a spool file arrives in the outqueue or is released in it the engine proceeds through the specified tests looking for a match with the spool file. If a match is found the engine processes the spool file with the specified form, check or form set and moves the original spool file to the archive outqueue.

The engine is a Data Queue Engine if a data queue is given. Otherwise the engine is a Polling Engine. Data Queue Engines block until a spool file arrives or is released in the outqueue and therefore are more economical with system resources than Polling Engines, which periodically poll the outqueue.

Engines are batch jobs that run in batch subsystems. The iDocs Utility Menu has an option to create the FMGBATCH subsystem if the user does not wish to run engines in QBATCH.

Engine List Display

Option 4 from menu IDOCS displays the list of all defined engines:

  4/29/08                     iDocs Form Suite                        FM0450C1  
 11:27:42               Engine Definition Maintenance                 QSECOFR   
                                                                                
 Logging Status: *ON                                  Hold Archive Status: *ON  
 1=Select  3=Copy  4=Delete  5=View  7=Rename  8=Start 9=End  10=Thread 11=Group
 18=Start (FMC0459)                                                                                
             Opt Engine       Description                 Status                
                 DCYTEST      DCY Test                    *NONE                 
                 POLLING      Test Engine                 *NONE                 
                 QUICKTEST    QuickTest                   *NONE                 
                 TESTCDA      Test Engine cda             *NONE                 
                 TESTENG      Test Engine                 *NONE                 
                 MYENGINE     My Engine                   *NONE               
                                                                                
                                                                         Bottom 
 F3=Exit   F6=Add   F7=Toggle Logging   F8=Toggle Hold Status      F21=Command  
                                                                                

Logging and Archive Status

The status display shows the current status of the engine.

Logging Status:*ON - Engine logging is active
Hold Archive Status:*ON - Archived entries will be placed on hold status

Note:
Logging is a debugging tool. *ON enables engine logging and each engine process will record an entry in a data base file named FRMLOGF. Normally this value can be set to *OFF.
Hold Archive *ON places the resulting FMGLASER entries on hold (HLD) in the designated Archive output queue. An *OFF designation leaves the entries with a ready (RDY) status.

Engine Status

The status display shows the current status of the engine.

 *NONE - the engine is not running
 *ACTIVE - the engine is running in a subsystem
 *JOBQ - the engine is waiting in a jobqueue to enter a subsystem to run

Note: For the display to detect a running engine, the job name in the subsystem must be the engine name. This setting is automatically made when launching the engine from this display using option 8. If you are manually starting an engine with the Submit Job command (SBMJOB) you must set this parameter:

SBMJOB CMD(CALL PGM(FMC0462) PARM('MYENGINE')) JOB(MYENGINE)

The above command starts a data queue engine named MYENGINE, specifying MYENGINE as the job name. MYENGINE will now show active in the engine display screen.

There is the same issue with automatic start/stop of engines.

Options

Option 8 Start Engine

Select this option to start the engine into a subsystem as a batch job. Specify the appropriate jobqueue at the prompt. Use QS36EVOKE if the subsystem FMGBATCH is not set up, otherwise use FMGJOBQ. Bear in mind that this subsystem and jobqueue should be multi-threaded. The QBATCH jobqueue ships from IBM as single threaded. The subsystem FMGBATCH and its jobqueue FMGJOBQ ship configured to support 2 concurrent engines. Any properly configured jobqueue that routes into any properly configured subsystem may be specified.

Option 9 End Engine

Select this option to end an engine. A datqueue engine will stop at the next point of inactivity. A polling engine will stop at the time of its next polling cycle, i.e. at the end of its wait period.

Option 10 Thread Engine

Select this option to start the data queue engine as a Thread Engine. A Thread engine is used when large print volumes are anticipated. It will prevent the 9999 spooled file limit for a data queue, where a job will error out when it has reached it max and need to be restarted. Each spool file job is batched instead of being run interactively. The batch jobs are sent to jobq QTXTSRCH as the default. The data area FMG/DSJOBQMON is used to store the default job queue used and can be changed if desired.

Note: If the Thread Engine is used, an archive output queue must be specified.

Option 18=Start (FMC0459)

This is fundamentally identical to selecting with Option 8, but provides the ability to retain the originating spool file user attributes for the resulting output.

Add / Maintain Merge Engine

To work with merge engines, go to menu IDOCS and take

4. Work with Engines 

Use F6 to create a new engine or select an existing engine (not running) with option 1.

  3/15/14                      iDocs Form Suite                       FM0450C3
 07:37:01                Engine Definition Maintenance                DYOKANA  

 Engine Name. . . . . . : IDOCSENG   Description. . : iDocs Engine             
 Archive Out Queue. . . : ARCHIVE    Archive Library: IDOCS661      F4=List     
 Data Queue . . . . . . : IDOCSDTAQ  Dataque Library: IDOCS661          
 Engine Status. . . . . :            Custom Command:                    
 Engine Type  . . . . . : *SERIAL    Test Type . . .: *FIRST *FIRST, *ALL 
        Originating        Spool     Spool      Form       Resulting  Resulting
 Prc Out Queue  Library    Attribute Attribute  App        SpoolFile  Out Queue
 Grp      F4=List          F4=List   Value      F4=List    Name        F4=List  
     IN         QUSRSYS    *FORM    INVOICE    INVOICE               OUT      
                                                                                
                                                                               
                                                                               
                                                                        More...
 F3=Exit F4=Prompt F7=Server View F8=Aggregation Settings F12=Previous F21=CMD
                                                                               
5250                   MW                                               075/012


Header Information

  • Engine Name: Name of the engine. Must be unique. Do not use embedded blanks.
  • Description: A description of the engine.
  • Archive Out Queue: Outqueue in which to place the input *SCS spool file after processing. If *NONE is specified the spool file will be deleted after processing. iDocs does not maintain the archive outqueue, so the administrator must periodically delete spool files from the outqueue.
  • Archive Library: Library containing archive outqueue.
  • Data Queue: Enter the name of a dataqueue to use with this engine or leave blank to run as a Polling Engine. The interval between pollings is specified in seconds in the data area DSWAIT. Engines should not share dataqueues.
  • Dataque Library: Library containing the dataqueue.
  • Engine Status: Current status of the engine.
  • Custom Command: Command to invoke a custom engine.
  • Engine Type: *SERIAL for normal processing and *AGGREGATE to perform aggregation.
  • Test Type:: *FIRST to process just the first target matched or *ALL to process every matching target.

Detail Information

These are tests applied to each spool file arriving in the outqueue. A spool file is processed once with the form, check or form set specified for the first test that returns true.

  • Prc Grp: For future use.
  • Originating Outqueue / Library: Specify the outqueue to monitor.

Originating Outqueue Considerations
Do not monitor outqueues attached to a printer or monitored by another engine as which entity will process the spool file is unpredictable. Use F4 to select the outqueue from a list.
Do not monitor an outqueue with the same name as any engine as the engine uses an outqueue by its own name in FMG as a work outqueue.

  • Spool Attribute: Attribute of the spool file to test. Use F4 to select the spool attribute from a list of supported values.
  • Spool Attribute Value: Value to use in the spool attribute test.
  • Form App: Form, check or form set to use in processing the spool file. Use F4 to select the form, check or form set from a list of these objects.
  • Resulting SpoolFile Name: The output PCL spooled file name from iDocs Engines is named “FMGLASER” reflective of its contents. You can specify a resulting spool file name here. A new enhancement in iDOCS 8 (09/19/23) allows administrators to configure the iDocs Engine to use the originating spool file name as the output PCL spool file name's name. To configure the iDocs Engine to use the originating spool file’s name as the PCL’s spool file name use *SPLFNAME or *FILE as values in the Resulting SpoolFile Name field on the iDocs Engine line. See the below example where the first line in the iDocs Engine configuration below will use FMGLASER as the PCL File Name in the output. The second line in the iDocs Engine configuration will use the originating spool file’s name as the PCL File Name in the output.

  • Resulting Out Queue: Outqueue to place the resulting *USERASCII spool file. This is the electronic document. Use F4 to specify the outqueue from a list.

Resulting Outqueue Considerations
Specify *FORMDEF to use the destination outqueues from the form definition.
Any destination outqueue specified in the form will override this.
DO NOT specify an outqueue in library FMG of the same name as the engine.

Server View Parameters

        Originating        Spool     Spool      Form       Resulting  Resulting 
 Prc Out Queue  Library    Attribute Attribute  App        SpoolFile  Out Queue 
 Grp      F4=List          F4=List   Value      F4=List    Name        F4=List  
     PRT02      QUSRSYS    *FORM    *STD       DEMOCHK               *FORMDEF   
 Archive Queue:            Library:                    Second File:             
                                                                                
 Archive Queue:            Library:                    Second File:             
                                                                                

From the main screen you may press F7 to specify these additional parameters:

  • Archive Queue: The archive outqueue specifically for this spool file test.
  • Library: The library containing the archive outqueue for this spool file test.
  • Second File: If a second file is specified, when the engine line is triggered through a successful test, the engine will append the contents of the named second spool file to the first spool file (which triggered the engine test) before performing the merge.

Aggregation options

In iDocs 6.5 the Aggregation Engine feature was added. This option allows you to combine multiple spool files into a single pdf to be emailed.

The aggregating engine operates by accumulating output until the trigger, as defined on the screen above, returns true. At that time it outputs the accumulated document ( as one document ) and clears the document cache. This feature has its most common use in creating one unified attachment for email delivery. If there are multiple documents that need be attached to an email, then this feature can amalgamate them into one attachment.

  4/21/20                      iDocs Form Suite                       FM0450C3  
 14:10:15                Engine Definition Maintenance                KKRAMER   
                                                                                
 Engine Name. . . . . . : SIGMAENG   Description. . : SIGMA Test Engine         
 Archive Out Queue. . . : SIGMASPLF  Archive Library: QGPL                      
 Data Queue . . . . . . : SIGMAENGDQ Dataque Library: QUSRSYS                   
 Engine Status. . . . . : *ACTIVE    Custom Command:                            
 Engine Type  . . . . . : *AGGREGATE Test Type . . .: *FIRST *FIRST, *ALL       
        Originating        Spool     Spool      Form       Resulting  Resulting 
 Prc Out Queue  Library    Attribute Attribute  App        SpoolFile  Out Queue 
 Grp                                 Value                 Name                 
     SIGMAIN    QUSRSYS    *FORM    ORDER_ACK  SGACKNW               IN         
     SIGMAIN    QUSRSYS    *USRDTA  PO610A     SGPOFORM              IN         
     SIGMAIN    QUSRSYS    *FORM    ORD_SHIPTX SGSO                  IN         
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                        More... 
 F3=Exit   F7=Server View   F8=Aggregation Settings   F12=Previous   F21=CMD    

By creating the engine as Engine Type Aggregate, you can utilize option F8 to enter the Aggregate Settings.

                                                                     FM0450W8 
                            Aggregate Setting                                 
                                                                              
   Select Form to Combine/Aggregate . . . . .   *ALL                          
   User program to check aggregate condition.   TTRAN       USERPGM           
                                                                              
   Sweep the outputs when                                                     
   AND/OR Field       Op  Value (Characters)                                  
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
                                                                     More...  
                                                                              
                                                                              
    F4=Prompt   F5=Refresh   F10=Save    F12=Previous    F13=PGM Template     
                                                              
                                                                              

In the Aggregate Setting screen shown above, the user can either specify a custom program (IDOCS 6.95) to handle the business logic of Aggregation or define “Sweep the outputs when”. The custom program will override the definition in “Sweep the outputs when”.

F13 will assist the user in creating a program along with the parameters that the iDocs engine will utilize when calling the the custom program. A sample CL program is shown below.

/*¹iDocs sample user program for Aggregate engine·                             +
  ¹Once this program is specified in iDocs aggregate engine,                   +
  ¹iDocs will call it to check if the aggregate criteria is matched or not,    +
  ¹if TRUE then &RETURN is 'Y' else &RETURN is blank.                          +
  ¹                                                                            +
  ¹Input parameters:                                                           +
  ¹  - &SPLNAME : SCS spool file name which is being processed by iDocs engine +
  ¹  - &SPLNBR  : SCS spool file number                                        +
  ¹  - &JOBNAM  : SCS spool file job name                                      +
  ¹  - &JOBUSR  : SCS spool file job user                                      +
  ¹  - &JOBNBR  : SCS spool file job number                                    +
  ¹                                                                            +
  ¹Output parameters:                                                          +
  ¹  - &RETURN  : If this parameter is Y then iDocs will complete the group    +
 ¹                and aggregate all PCLs in group into 1 PCL                   +
*/
             PGM        PARM(&SPLNAME &SPLNBR &JOBNAME &JOBUSER +
                          &JOBNBR &RETURN)

             DCL        VAR(&SPLNAME) TYPE(*CHAR) LEN(10)
             DCL        VAR(&SPLNBR) TYPE(*DEC) LEN(6 0)
             DCL        VAR(&JOBNAME) TYPE(*CHAR) LEN(10)
             DCL        VAR(&JOBUSER) TYPE(*CHAR) LEN(10)
             DCL        VAR(&JOBNBR) TYPE(*CHAR) LEN(6)
             DCL        VAR(&RETURN) TYPE(*CHAR) LEN(1)

/*¹If &RETURN is 'Y' then it's end of the spool files group, +
¹  and iDocs will aggregate the PCL files into 1 PCL·*/

             ENDPGM
             

Daisy Chaining Engines

One of the architectural advantages of the iDocs Suite is that a spool file can be processed by multiple engines serially. This enables functionality impossible within one form, formset or engine.

The key mechanism of daisy chained engines is the engine archive outqueue. The normal use of the archive outqueue is to store the original spool file for re-use. However, it can be used to hand off the original spool file to another engine by specifying as the archive outqueue an outqueue that is monitored as an originating outqueue by another engine.

  -------------        --------------             -------------           ------------        ---------------
 / engine 1  /  -->>>  / engine     /  -->>>>    / engine 1   /  -->>>>  / engine 2  / -->>> / engine 2     /
/ orig outq /         /    1       /            / archive \  /          /           /       / archive outq /
------------         --------------            /  engine 2  /           ------------       ----------------
                          |                   /   orig outq/                 |
                           \                 --------------                   \
                            \                                                  \
                         -----------                                        -----------
                        / normal   /                                       / normal   /
                       /   output /                                       /  output  /
                      ------------                                       ------------

The engine setups would look something like this:

Engine 1 Setup:

 6/08/10                      iDocs Form Suite                       FM0450C3  
11:27:27                Engine Definition Maintenance                DYOKANA   
                                                                               
Engine Name. . . . . . : ENGINE1    Description. . : First Engine              
Archive Out Queue. . . : ARCHIVE1   Archive Library: FMG           F4=List     
Data Queue . . . . . . : DTAQ1      Dataque Library: FMG                       
Engine Status. . . . . :            Custom Command:                            
                                    Remove Over Printing: N                    
       Originating        Spool     Spool      Form       Resulting  Resulting 
Prc Out Queue  Library    Attribute Attribute  App        SpoolFile  Out Queue 
Grp      F4=List          F4=List   Value      F4=List    Name        F4=List  
    ORIGOUTQ1  FMG        *FORM    *STD       FORMAPP1              NORMALOUT1 
                                                                               

Engine 2 Setup:

 6/08/10                      iDocs Form Suite                       FM0450C3  
11:27:27                Engine Definition Maintenance                DYOKANA   
                                                                               
Engine Name. . . . . . : ENGINE2    Description. . : Second Engine             
Archive Out Queue. . . : ARCHIVE2   Archive Library: FMG           F4=List     
Data Queue . . . . . . : DTAQ2      Dataque Library: FMG                       
Engine Status. . . . . :            Custom Command:                            
                                    Remove Over Printing: N                    
       Originating        Spool     Spool      Form       Resulting  Resulting 
Prc Out Queue  Library    Attribute Attribute  App        SpoolFile  Out Queue 
Grp      F4=List          F4=List   Value      F4=List    Name        F4=List  
    ARCHIVE1   FMG        *FORM    *STD       FORMAPP2              NORMALOUT2 
                                                                               

The normal use of daisy chaining is to automate complex functionality. Some possible uses include:

  • Process a document to print and then archive to networked directory using the iPDF Monitor.
  • Process a document to email, then to fax, then to print where users get more than one output type.

(i.e. a user can get both an email and a fax )

Configuring Automatic Engine Start / Stop

Engines can be configured to automatically start and stop two ways.

Starting Engines at IPL

Attention: This change should be done by a properly qualified programmer. A mistake in the startup program can render your system dysfunctional.

Presuming that the FMGBATCH subsystem has been installed and configured on your system, alter the system startup program ( system value QSTRUPPGM ) to execute the following:

ADDLIBLE FMG *FIRST
STRSBS FMGBATCH
SBMJOB CMD(CALL PGM(FMC0462) PARM(ENGNAME)) JOB(ENGNAME) JOBD(FMGJOBD) JOBQ(FMGJOBQ) USER(FMGUSER)

Discussion:

  • The FMGBATCH subsystem can be created from menu FMGUTILITY, option 1. The default configuration is limited and the configuration should be reviewed by a qualified system administrator.
  • Never call the iDocs engine from the startup program, instead submit it with SBMJOB. If you call it the startup program will never progress beyond the call statement and thus never complete. Additionally CALLing the engine will cause it to run under the user profile running the startup program (user QSYS) which causes problems. iDocs engines will run in any properly configured batch subsystem.
  • Program FMC0462 is a data queue engine; use FMC0460 for a polling engine. (FMR0471 ends either one).
  • The one parameter to the FMC0462 or FMC0460 call should be the name of the engine in upper case.
  • The job name should be the engine name in upper case.
  • You can control library list and execution environment by specifying the job description as FMGJOBD, or another job description of your creation. FMGJOBD ships with a default configuration and should be reviewed / changed by a qualified system administrator.
  • Always specify a user in the SBMJOB command so that user QSYS does not run the engine. The user needs *SPLCTL and *JOBCTL authorities, as well as an entry in the System Directory so that it has access to folder FORMFMG. It will also require access to library FMG as well as folder FORMFMG specifically.

Configuring the Engines to Start and Stop

You may create generic Job Schedule entries to start and stop engines from menu FMGUTILITY, option 2. See Schedule an Engine to Start/stop .

The easiest way to create entries is with the Utility menu. This option will generate JOBSCDE entries. You can them view the entries (WRKJOBSCDE) and modify them as needed.

Examples: To start and end a Data Queue engine named ENGINE1 in library IDOCS:
To start: CALL PGM(IDOCS/FMC0462) PARM('ENGINE1')
To end: CALL PGM(IDOCS/FMR0471) PARM('ENGINE1')

Proper Target Outqueue Configuration

Essentially, iDocs is a pcl 5e and pcl 5c printer driver for the IBM i. Its output is printer ready pcl. As such its output does not need translating, and in fact such translation may corrupt the output.

There are two considerations:

  • The destination outqueue on the IBM i should specify TRANSFORM(*NO) so that the output is not transformed as it leaves the IBM i.
  • The destination outqueue on the IBM i should specify as the target queue on the target printer a queue that does not transform the input so that the target printer does not transform the stream. For Hewlitt Packard printers, that queue is named RAW. So the IBM i outqueue should contain this configuration parameter: RMTPRTQ('RAW').

Engine User Profile Issues

Profile Settings

The profile running the engine ( IDOCS in this section ) should have the following attributes set in the user profile:

  • USRCLS(*SYSOPR):User class *SYSOPR
  • SPCAUT(*JOBCTL *SPLCTL): Special authorities: job control, spool control
  • MAXSTG(*NOMAX): No ( or very large ) maximum storage limit for profile
  • JOBD(FMGJOBD): Job description should be FMGJOBD, which has a limited and correct library list specified
  • OWNER(*USRPRF): New objects should be owned by the profile
  • HOMEDIR('/home/idocs'): Specify a home directory for the profile; make certain it exists and that the profile has full authority over it.

The user profile should have access to the iDocs library:
GRTOBJAUT OBJ(iDocs library) OBJTYPE(*LIB) USER(IDOCS) AUT(*ALL)
GRTOBJAUT OBJ(iDocs library /*ALL) OBJTYPE(*ALL) USER(IDOCS) AUT(*ALL)

You may use the below command to permanently grant the required permissions for iDocs:
CHGAUT OBJ('/qsys.lib/iDocs.library/*') USER(*PUBLIC) DTAAUT(*RWX) OBJAUT(*ALL)

If there are no object authority issues with iDocs you might want to disable adopted authority on the programs:
CHGPGM PGM(iDocs library /*ALL) USRPRF(*USER) USEADPAUT(*NO)

Job Description Settings

FMGJOBD Job Description has these settings:

  • JOBQ(QGPL/QS36EVOKE): Uses multithreaded jobqueue by default.
  • INLLIBL(FMG QGPL QTEMP): FMG at the top of a simple library list.
  • JOBMSGQFL(*WRAP): Wrap a full message queue.

Directory Entry

This user profile should also have an entry in the system directory; you can use the command ADDDIRE to add the entry:
ADDDIRE USRID(IDOCS LCLSYSTM) USRD('iDocs user profile') USER(IDOCS) SYSNAME(*LCL)


Engine Troubleshooting Q & A

Q. When I submit the engine it prompts me for the job queue. What is this?
A. A job queue is an IBM i object that contains entries for jobs that are waiting to be processed by the system.

Q. When I submit the engine it runs normally, but afterwards no one else can process their jobs submitted to batch?
A. In this case the engine has been submitted to a single thread job queue that can only process a single job at a time. The solution to this bottleneck is to submit the engine to a multi thread job queue. Create one or use a pre-configured multi thread job queue such as QS36EVOKE.

Q. Polling Engine starts for 60 seconds and ends normally without processing any entries?
A. Check each originating output queue and insure that they all still exist on the system.

Q.The engine runs and the original spool fie is successfully copied to the archive out queue, yet the merged spool file disappears.
A. The writer may not be responding correctly. Follow these steps to fix the writer: end the writer, vary off the device, vary on the device with a reset, restart the writer. If this doesn't resolve the issue, verify that the printer is HP4 compatible.

Q. After the engine is submitted, it ends several days later in error?
A. There could be numerous reasons why the engine has ended in error after successfully running without interruption. Most common is that the engine has been running for many weeks accumulating system resources and job structures. It is recommended that an engine be stopped and restarted weekly. Other specific causes:

  • A lock on the engine record has occurred from a backup routine. Its best to make a single backup of the iDocs library whenever the definitions have been added or altered and set it aside, then remove the iDocs library from the backup definition.
  • An engine will end abnormally if the job's message queue has filled up. This could be the case if the engine has been running for weeks. To remedy this type of abnormal end change the system value QJOBMSGQFL value to *WRAP. This will prevent the job queue from filling up. CHGSYSVAL SYSVAL(QJOBMSGQFL) VALUE(*WRAP).
  • If the engine has been running for weeks, it could end abnormally if it exceeds the maximum printer output files that can be created per job. Support was added to V5R1M0 to allow a job to create up to 999,999 spooled files. To remedy this type of abnormal end change the IBM i system value QMAXSPLF value from the default value of 9999 to the maximum of 999999. CHGSYSVAL SYSVAL(QMAXSPLF) VALUE(999999).
  • If the engine is processing a large volume of spool files there could be an abend due to 'QSPOOL job not closing'. Keeping spooled files with jobs allows job commands such as Work with Submitted Jobs (WRKSBMJOB) to work with the spooled files even after the job has ended. Detaching spooled files from jobs reduces the use of system resources by allowing job structures to be recycled when the job ends. The default for QSPLFACN is *KEEP. CHGSYSVAL SYSVAL(QSPLFACN) VALUE(*DETACH): When the job ends, the spooled files are detached from the job and the job is removed from the system.

Note: You must have *ALLOBJ and *SECADM special authorities to change system values. A change to the system value takes effect immediately; therefore, there is no need to IPL the system after making the change.



[ Up To Index ]

pub/workwithmergeengine.txt · Last modified: 2023/09/19 13:35 by kkramer