How to Setup Advanced Data Processing Rules in the Command Line ASCII Interface Transcript for Video:
Welcome to the video tutorial on how to configure advanced data processing rules with PageGate’s GetASCII module. In this tutorial, we’ll be going over how to configure the GetASCII module to process custom ASCII files like those output by security monitoring and CAD systems as well as XML, HTML and JSON formatted content.
Before we begin, it’s incredibly important that you understand the basics of the GetASCII module before attempting to configure the advanced polling. For more information on the module’s basic configuration and functions, have a look at our GetASCII overview video and check out the GetASCII section of our documentation.
So, let’s say that you’ve already configured the basics for GetASCII and now it’s time to start breaking down your data in to something usable. The first, and most important, thing is that the files must be in ASCII. The specific structure of the data doesn’t matter as much as whether or not the data is in ASCII. Examples of ASCII files are plain text files as well as “structured” files like HTML, XML and JSON.
Before we really get in to the minutae of configuring an advanced polling rule, let’s go over our special characters here in the Symbols section.
The Single Character Wildcard is a character that can be used in a search pattern to designate a location where any single character (or symbol) will be accepted as a match. For example if ? is the Single Character Wildcard, then the pattern ?use would be a match for words like fuse, muse, ruse, but would not match amuse or reuse.
The Multiple Character Wildcard is a character that can be used in a search pattern to designate a location where any group of characters (and symbols) will be accepted as a match. For example if * is the Multiple Character Wildcard, you could use the pattern *use to find muse, fuse, ruse, amuse, accuse, refuse or any other word or alphanumeric sequence that ends with the letters 'use'. When used by itself as a start pattern, the Multiple Character Wildcard would always match the first character of the input text.
The End of Record Symbol is a character that can be used in a search pattern to designate the last character of the input text, regardless of what that character is. The End of Record Symbol is most commonly used in End Pattern fields as a value that means "the last character of the input data". If this character appears within the body of a file scanned, the program will stop reading at the character and will not read in anything past it
The ASCII Value Delimiter is a character that means "the next three digits stand for an ASCII character value". This is normally used for expressing non-printable values like a carriage return, line feed, horizontal tab and other control characters. For example, if you use \ as the ASCII Value Delimiter, let’s say your data shows up like this:
SubscriberName=FD12
MessageLine1=
The line break between FD12 and MessageLine1= is a carriage return, line feed statement that can be expressed as \013\010. Any character, even normal alphanumeric characters, can be expressed in this ASCII format, if necessary.
Please note that if you want to include the actual ASCII Value Delimiter character in a search pattern, use its ASCII code. For example, if the ASCII Value Delimiter is \ and you need to use the character \ as part of a search pattern, you would need to use \092 to represent the \ as 092 is the ASCII character value for \.
The first step in configuring an advanced polling rule is to go to the Files section. This table is a collection of all of your advanced polling rules. To create a new one, click add.
If all messages processed by this rule should go to a single recipient or group, select it in the dropdown. Otherwise, leave it at * to indicate that any recipient or group may receive messages from files processed by this rule.
The From section allows you to specify a sender’s name value to be associated with messages processed from this rule.
The File(s) section specifies the name of the file or files to be processed. For example, to have this rule scan for all .txt files in a folder, you would use *.txt in the field. As another example, you could specify a specific file name like CADLog.XML in the field. You can even combine the two methods. For example, you could use CAD*.XML and the program will scan for any file whose name begins with the letters ‘CAD’ and whose extension is ‘.XML’.
The Path section specifies the directory in which the data can be found. This can be set to any local or network access/UNC path. It’s important to note, though, PageGate’s services run under the local system account by default. The local system account in Windows very rarely has any network privileges, which means if the GetASCII module is configured to reference a network share, the local system account won’t have permission to read, write and modify the contents of the folder or file. To grant it permission, you’ll need to apply a service account to the PageGate GetASCII Windows service.
Last we have the File Type. This section allows you to select from one of four potential file reading options.
First, we have Standard. PageGate’s standard file format is as follows:
<Recipient or Group>
<Sender>
<Message>
Since they’re non-printable characters, it’s important to note that each variable needs a carriage return, line feed after it as GetASCII looks for CRLF as the terminating character in each variable statement. So, if the files you need to process have the above structure, you don’t need to configure the Record, Recipient, Sender and Message advanced polling rules, you just need to specify the custom location and “Standard” as the file type.
The Custom file type allows you to tell PageGate how to pattern-match the data contained in processed files. This is the file type you should use for XML, JSON and HTML data processing as well as any ASCII file that doesn’t follow PageGate’s basic processing format.
The Adding file type tells PageGate that all data processed will be contained in a single file that has information appended to the end of it. So, you could have a single file called “alerts.txt” that PageGate monitors and any time new information is appended to the end of the file, the program will process it as a new message to be delivered.
The Changing file type is a bit like the Adding file type, where you tell PageGate to monitor a single file. However, the Changing file type scans to see if any of the existing document has been modified. If it has, PageGate re-processes the entire file, looking for message data.
Once you determine what type of file you’re going to be working with, there are four bits of information we’ll need to configure to tell PageGate how to understand the data:
- Record
- Recipient
- Sender
- Message
The Record settings designate what part of the new file contains the message data (i.e. recipient, sender, message text). For example, if * is your multiple character wildcard and ~ is your end of record symbol, then if you use a * as a start pattern and a ~ as an end pattern, that will tell PageGate that the entire contents of the file are to be scanned for relevant information. As another example, if you’re scanning XML data, you could use the XML header tag as the start pattern and the closing tag as the end pattern (Example: <XML> as the start pattern and </XML> as the end pattern).
Basically, we’re telling PageGate that everything in between the declared characters is relevant data. This also allows PageGate read in multiple messages from a single file, if multiple sections match the requirements of the Record settings.
You’ll also notice that there is a Start Offset and an End Offset.
The Start Offset represents the number of characters before (expressed as a negative value) or after (expressed as a positive value) the Start Pattern to begin reading. For example, if you were using <XML> as your start pattern, you could set a Start Offset of 5 to say that the text used as the start pattern should be skipped because it isn’t relevant for the Recipient, Sender or Message information. Using a value of -5 would say that 5 characters before the first character in the start pattern is the beginning of the record.
The End Offset, on the other hand, represents the number of characters before (expressed as a negative value) or after (expressed as a positive value) the End Pattern to not include in the record data.
As an important note, the value -999 has special meaning, and is typically used in conjunction with the use of the Multiple Character Wildcard in the search strings.
If -999 is used as a Start Offset, an offset of one character past the string that matches the start pattern, is used. This will cause the number of characters that were found to match the Start Pattern to be skipped, even though the actual number of characters in the Start Pattern matching string aren't known ahead of time.
If -999 is used as an End Offset, an offset of one character before the string that matches the end pattern, is used. This will cause the number of characters that were found to match the End Pattern to be skipped, even though the actual number of characters in the End Pattern matching string aren't known ahead of time.
The Recipient section determines where in the record data the recipient or group name will be found. If the data read in should always go to the same recipient or group, you can select Static Recipient and select the recipient or group from the dropdown list. Otherwise, you’ll need to tell PageGate where the recipient’s information appears in the data by configuring the Start and End Patterns. The Start and End Patterns tell PageGate that everything between the declared character or characters is the Recipient or Group name. For example, let’s say that your data is structured like this:
SubscriberName=FD12
[Message]
Line1=
Line2=
You would use SubscriberName= as the start pattern and \013\010 as the end pattern. You would also want to use a Start Offset of 15 as there are 15 characters in SubscriberName= and we don’t want that text treated as part of the recipient value. Likewise, you would want to set the End Offset to 0 as the end pattern is not included in the recipient information by default.
As another example, let’s say that you’re breaking down HTML, XML or any other format that uses tags and let’s say your data is structured like this:
<Name>Precinct 3</Name>
You would use <Name> as the start pattern with a start offset of 6 and </Name> as an end pattern with an end offset of 0. By default, the end pattern is not included in the recipient information.
As an important note, the value -999 has special meaning, and is typically used in conjunction with the use of the Multiple Character Wildcard in the search strings.
If -999 is used as a Start Offset, an offset of one character past the string that matches the start pattern, is used. This will cause the number of characters that were found to match the Start Pattern to be skipped, even though the actual number of characters in the Start Pattern matching string aren't known ahead of time.
If -999 is used as an End Offset, an offset of one character before the string that matches the end pattern, is used. This will cause the number of characters that were found to match the End Pattern to be skipped, even though the actual number of characters in the End Pattern matching string aren't known ahead of time.
You’ll also notice this option here: Use Lookup Table. This enables you to do basic search and replace strings on the data read in as the recipient value. To take our earlier example, let’s say that the recipient value read in is Precinct 3. Since recipient and group names in PageGate can’t have spaces in them, we’ve got to translate that name in to the recipient or group name in the program. To do that, check “Use Lookup Table” and click Edit Table.
Once here, click Add.
Assuming * is your multiple character wildcard, in “Data In”, we’ll use *Precinct 3* to indicate that there could be any characters on either side of the sequence we’re interested in but we’re really just interested in the expressed sequence.
In “Data Out”, we’ll use the actual name of the recipient or group as it appears in the PageGate Admin. In this case, precinct_13.
Click Apply, then click Apply again.
The “Sender’s Name” is the text that will appear before the message to denote who or what sent the message and is used to construct the “Mail From” value when delivering SMTP/email messages. The Sender section determines where in the data the sender’s value can be found. This section uses the same pattern matching as the recipient section and can be configured similarly. Typically speaking, most systems won’t pass a sender’s name value in the data written and you’ll want to select Static Sender and enter a sender’s name.
The final section is Message. The Message section determines where in the record the message is found and uses similar Start and End Patterns. To use our earlier example, let’s say that our data looks like this:
SubscriberName=FD12
[Message]
Line1=
Line2=
You would want to use [Message] as the Start Pattern and 9 as the Start Offset and ~ as the End Pattern with 0 as an offset. Using ~ as an End Pattern tells the program that the entire remaining contents of the data are part of the message to be delivered. That would grab this as the message:
Line1=
Line2=
For some systems, that line by line delineation is important because it helps them determine what information belongs in what field. However, if you’re sending these messages directly to your recipient’s cell phones, pagers and email addresses, you may want to remove those line declarations. To do that, check Use Lookup Table and click Edit Table.
Assuming * is your multiple character wildcard, you could use *Line1=* as the Data In and leave the Data Out field blank to remove the text Line1= from the message. You would want to add a second entry for Line2= and any others as well.
When finished, click Apply then click Apply again.
This concludes the video tutorial on configuring advanced data processing rules with PageGate’s GetASCII module.
|
