TN 7669: ServiceNow Integration
Sassafras AllSight can be used as a data discovery source for ServiceNow
Overview
The ServiceNow ITSM Platform can utilize a number of data sources for asset discovery. Sassafras AllSight offers one such option by way of our free Application Plugin available in the ServiceNow Store. While the use of this module is plug and play, this document details how the data is mapped in the complex tables of the ServiceNow Platform.
Use Case
The most common question is why use our plugin for a data discovery source. ServiceNow has their own discovery module as well as leveraging SCCM, so this is a valid question. The answer is detail. For example, our computer records have 24 additional data points that we import to our extended computer table compared to the built in fields. In addition, we import the usage information with relations between user, computer, and software. While the amount of data and robustness of reporting that results from this are not as complete as when used in AllSight itself, it is data that can not be gathered with the built in discovery methods. Integration of this data into your ITSM platform can also be preferable to using a separate dashboard for AllSight. Note however that AllSight provides for rich embed options that could be used in a ServiceNow Dashboard.
Because one of the data points imported is the SCCM ID our agent discovers, a customer can leverage the AllSight data to make decisions on software management and then take actions using the SCCM integration with ServiceNow. Sassafras Software has a vast Product catalogue so the software data imported to ServiceNow is pre-normalized. This is an added benefit for customers using CMDB. Those using SAM Pro still benefit from our normalization, and simply need to link the effective product from the ServiceNow normalization catalogue.
Development
Our near term plan is to simplify the options in the plugin to a single choice that covers most items. This choice quite simply indicates what you plan to use for Asset Management. If you choose to use AllSight for asset management, we will import more data and automatically create entries in linked tables as needed. The assumption in this use case is you will be updating Asset fields in AllSight and using our platform for lifecycle management, and using the imported CMDB records in ServiceNow to write Incident and Request tickets.
Choice two will be to use ServiceNow for asset management. In this case we'll import minimal information because you'll be using the tools in ServiceNow for asset and lifecycle management and simply using our data as the most basic hardware discovery tool.
Prerequisites
Version 1.5 of the plugin is required for compatibility with the San Diego and later editions of ServiceNow. In order to obtain proper data through the integration version 7.9.0.2 or later of AllSight is required. While the framework was in place as far back as 7.7, many changes were made since then. Sassafras Software always recommends using the latest release of the platform to provide the latest functionality. It also requires the following ServiceNow modules:
- Base ServiceNow
- Content Management
- System Import Sets
- Configuration Management (CMDB)
Support for SAM Pro is also included with no additional configuration needed. The plugin will import data to the proper table based on detected table structures.
How it Works
The plugin contains Import Sets for each of the data types we bring in: Computers, Printers, Displays, Products, Audits, Software Usage, and Computer Sessions. Note that several of these are optional, see Configuration below for details. Each Import runs on a Schedule. By default this schedule is initiated at 5am local time which follows the 4am default Product Audit run in AllSight. These schedules can be adjusted as needed. The other 5 imports trigger in succession after the primary. The Audit data in particular is dependent on the Computer and Product data being imported first so references exist properly and records are not dropped.
Because the data is pulled by ServiceNow from Allsight (by means of dynamically generated CSV files in AllSight over HTTPS), you must ensure that any firewalls in the environment allow this communication. Keep in mind that ServiceNow is a cloud hosted solution and AllSight is most likely hosted on-prem. Any edge firewall therefore must allow HTTPS from your ServiceNow instance to your AllSight server.
This integration is one way only. No data is written back to AllSight, and any imported data will replace manual changes. For example, if you change the lifecycle stage of a Computer in ServiceNow, it will revert to the actual setting in place in that record in AllSight. To assist with this management, each computer record has a link to the corresponding record in AllSight. This allows edits to be made in the source record easily which will then synchronize on the next import. Remember this is a Discovery source, so the source is taken as the record of truth.
From the Configuration page of the plugin:
This application allows you to import Computer and Software information including installs and usage from a Sassafras AllSight Server into ServiceNow. This is done by way of the Web Service in the AllSight Server on a nightly basis. It is compatible with both CMDB and SAM Pro configurations. Computer data is stored in KS Computers which extends Computers. We add many custom fields to the defaults in ServiceNow. Duplicates by computer name and serial are noted for reference/resolution in a report on the Dashboard. Display and Printer data is also stored in extended tables of those CMDB types. For "vanilla" CMDB, we extend the Software and Instances tables for our Product and Audit data respectively. For SAM Pro we bring our Audits directly into Software Installations which populates the Discovery information for reconciliation. Usage for software and Session data is also imported into custom tables. User records can optionally be create in sys_users, which is required to have any linking of the above data/records to a user.
Click here for additional details
Integration Details Herein we detail more of the behind the scenes information about this integration including tables, menus, permissions, etc. The intention is to use your Sassafras Server as a primary source of inventory data for computer and installed software asset records. Because the default tables are extended or directly written to, all the imported data is available in the normal default forms (e.g. Incidents). In the event of duplicates to the internal CI tables, you will note the table source in selection menus in forms to assist with which record you wish to reference. It is recommended that you remediate duplicates as you see fit. We provide a report for this in the Dashboard to assist that process. SAM Pro users should note they may need to perform more manual remediation of references (i.e. Normalized Product) than with some data sources. This is because Sassafras Software already normalizes Programs (executables) into Products (suites). This is integral to the operation of our software, so some additional manual correlation may be needed to equate our Products with those in the ServiceNow catalogue, should that be part of your desired workflow. Note at this time no data is exported to KeyServer. Changes made to records in Servicenow can and will be overwritten by the next import from KeyServer. Care should be taken to either enter data updates in KeyServer which then sync to Servicenow, or ensure you are not editing imported fields that contain values on the KeyServer side. Permissions As per normal, an Admin will have full access to the application. There is an admin role in the application (search keyserver) that will grant full access to all components. Additionally there is an itil role that will grant only read access to the KS Computers table and Dashboard. Note that due to limitations of the platform, the reports in the Dashboard will not be visible until each one is shared with a Group that your "itil" users are members of. Because the data is imported, write access is not granted to the itil role and it is discouraged to do so in most cases. You can however create a role as desired to grant this access, being aware that only fields that are not replaced by the data import will not be overwritten by the next import. File Structure All Tables, Data Sources, and Reports, as well as most other modules start with KS in the name. Data Sources: KS Audit Products - pulls /ext/servicenow/audprod.csv KS Computers Source - pulls /ext/servicenow/computers.csv KS Products Source - pulls /ext/servicenow/products.csv KS Sessions Source - pulls /ext/servicenow/sessions.csv KS Usage Source - pulls /ext/servicenow/usage.csv KS Printers Source - pulls /ext/servicenow/printers.csv Transform Maps: (puts data from import tables into destination tables) and Scripts KS Audits Transform - ks_installs OnStart - set install date to bogus date so corrected on import if still valid OnBefore - only run if sam table does not exist OnComplete - find bogus date records and delete them (clean up uninstalls) KS Audits SAM Transform - cmdb_sam_sw_install OnStart - sets all records inactive so import marks active if still valid OnBefore - only run if sam table exists KS Comp Transform - ks_computers KS Prod Transform - ks_products KS Prod SAM Transform - not active, should delete. Products in SAM are made by way of the data imported to the sw_install table by Audits. KS Sessions Transform - ks_sessions KS Usage Transform- ks_usage KS Printers Transform - ks_printers Modules: (menu items) About - points to about content page sassafras/About.do Dashboard - sassafras/Dashboard.do Contact - link out to Sassafras Software website Tables - links to the named tables Import tables - not shown Tables: KS Computers - Extends cmdb_computers to add fields Custom Default View to show our fields KS Installs - extends "Software Instance" KS Sessions - custom table KS Software - ks_products extends "Software". KS Usage - extends "Software Usage" KS Printers - extends "Printers" KS Settings - Custom table. Used for one record for the settings page Content: Site: Sassafras and related blocks to create the About and Dashboard pages. Roles: keyserver.itil - Read access to the Application menu and KS Computers table Business Rules: Update KS Data Sources: Takes input from the about page form (KS Settings) and updates data sources and other settings. Data Import To see full details of the fields imported and how they are imported you can review our transform maps. Where possible data is brought in straight from the KeyServer values. In many cases a script is needed to convert values into readable data or perform a simple units conversion. Some notable items include: Computer Lifecycle: We translate our terminology into Servicenow standard choices and input to hardware_status and install_status to appease Computer and ALM level forms. Duplicate Computers: We check for a match in all other computer tables on serial and name to flag possible duplicate records. These are summarized in a report in the Dashboard.
Configuration
Once the plugin has been activated in your instance, navigate to the Sassafras Discovery module. You will see several navigation items which will reflect your instance configuration (CMDB vs SAM). Click on the About/Config to enter the information for your AllSight instance, as well as choose various data import options. Mousing over each option will display a tooltip.
Options
Scheduling
You should be aware of the recommendations of ServiceNow to not overlap data imports. Our default schedule for importing from your AllSight Server is 5am instance time. This is for two reasons. First, the default import time for the SCCM integration is 3am. Second, the default time for KeyServer to perform the Audit Products operation is 4am. As such, we want to ensure our import happens after these operations complete.
There are 7 data imports in our integration: Computers, Products, Audits, Sessions, Usage, Printers, and Displays. These occur in that order in a dependent configuration. If you want to modify the starting import time, all 7 imports should be re-scheduled to the same time and the dependency will handle the execution order.
If you need to change the schedule in ServiceNow, consider how this might impact your schedules in KeyServer. There are two main concerns here: PRS, and Audit Products. These are both scheduled in KeyConfigure -> Config -> General Settings. You want to ensure that the PRS check is first, and allow about an hour before the Audit Products. Both of these should complete before you import into ServiceNow, to ensure the most up to date information is being pulled in each night. Allow at least an hour for the Audit to complete, perhaps longer for very large sites.
Removal
In the event you wish to remove our plugin and the related data, this should be easily accomplished. The majority of our data is put into extended tables so removal of the plugin should remove those tables. For data that is put into normal tables (e.g. Users), the records are all noted with a source of "KeyServer". This makes it easy to find and delete those records manually, or use a Background Script to automate this.
Navigation
The main navigation items provide access to the above detailed About/Configuration page, an example Dashboard with a number of Reports using the imported AllSight Data, a link to Sassafras Software Support, a list of the imported Computer records, and a list of the Scheduled Data Imports. Included with the KS_Computers custom table is a custom form view to allow easy viewing of our extensive data fields. The Additional Tables sub menu includes the Printers table, filtered views of import data for troubleshooting, Usage and Session tables, and a filtered list of imported Users.
Duplicates
A typical concern, especially if using multiple Discovery sources, is duplicate CI records. Because we store Computer records in an extended table, there is no concern about data overwrite from another source like SCCM which writes to the default computers table. In addition, we perform a duplicate search based on Name and Serial and flag any duplicates in our table with a reference to the match in the default table. These are then viewable in our example dashboard in a widget, so you can remediate if desired. Because we use an extended table, referencing a CI in an Issue will display in the drop down if the record is from the KS_Computers table as well, so you know which source you are referencing in the ticket.
Users as mentioned above are put in the default table, but as inactive and flagged with a data source of KeyServer. If a userid matches an existing entry, no new entry is made. This means there is no need to remediate duplicates per se. However, if you would like Usage to line up with existing records, the proper approach would be to ensure the existing SN User record uses the same userid or name that is used for computer logins. This can be complex in some environments which is why this is optional. The Imported Users navigation item will show all accounts created by the import.
For those using SAM Pro, there can be concern about the ServiceNow product vs the Sassafras Software Product. This is best handled by using the Normalized Product reference in SAM to correlate their product to our product. Once this has been done, you should be able to use the advanced features of SAM Pro normally while leveraging the Sassafras Software data. We recommend contacting ServiceNow for question related to using those features.
Troubleshooting
Several items are presented in the modules list to help with potential troubleshooting:
One important tip is to ensure you do not use commas in any manual data in AllSight. For example, adding a comma to the Building field of a computer record will cause a cascade failure of the import. This is due to a limitation of the platform, so if you see only a fraction of your computer records imported you should check for this condition.
Considerations and Limitations
There are several challenges with importing data to ServiceNow due to the platform's complex structure and limited flexibility. There are also several known issues with the platform that create problems. This section has been written to discuss these should there be questions about the behavior of our plugin design. This is only a few of the hurdles we have encountered in working with ServiceNow.
Terminology
As can often be the case, one fundamental issue is simply that of language. For example, computers are organized in AllSight under Divisions which can in turn be nested in Sections. These are used for administrative access rights and logical grouping of the computers for reporting purposes. ServiceNow really has no concept of grouping computer objects, there are simply data fields in a spreadsheet. We have Asset fields that include Department and Location. ServiceNow documentation says "Business units typically comprise departments and are associated with a company." which leads us to equate Sections to Business Units. However our notion of Division and Department are such that you can have any number of divisions in a section and computers could have any department designated regardless of division. For example, 50 computers could be in one department, but organized into 3 labs (Divisions) under a Section that is likewise representative of the Department. Or perhaps there are several departments that fall under the Section, but it's a meta structure. The simple choice in this case was to map Section to Business Unit, Department to Department, and store Division in a custom KS_Division field. The latter may very well equate to the Active Directory OU the computer lives in if you're using our AD mapping for computers. There is no field in SN for AD OU, so while a meaningful data point, we had to make our own place to store it. This is all just one example of various choices that had to be made regarding "where to put data".
User Handling
User data is very important. Who is the Owner of the computer? Last User? Who used what software how long? In AllSight, all user identification comes from the workstation OS telling us who is logged in performing the events. You can then use this to link to the Owner field, or put in arbitrary information there. None of this requires a unique identifier, and two computers could report the same person as two different users due to long or short names in a network account. This is all separate from Authentication to the AllSight platform which can tie directly to a domain system.
In ServiceNow, all user records are in one table and the purpose of this table is for system access only. In other words where we have two distinct notions of authenticating users vs user event data, SN has only one. ServiceNow Users do not have to be unique or in any way linked to domain accounts (though usually are). Because we have no way to correlate these perfectly, we are left with two options. First, we can simply not bring in any user related data. The problem here is that you lose any reference at the asset level to a person (more on this below). Second choice, create user records that are "noise". In the latter case we take care to make the records programmatically and mark them as created by AllSight for easy manipulation later, as well as making them disabled. However, this does mean it's likely a user can exist in the database twice. Once with their real account used for SN access, and a second time under some other name which is tied to a computer and usage data.
Hard Links Only
In AllSight we have many data tables that reference each other. All links are "soft" in that corresponding data may or may not exist in the referenced table for that field. We handle this gracefully. ServiceNow only has "hard" links between tables. That is, if I want to put a name in a field for say the computer owner, that name MUST exist in the user table. There is no option to simply put in a value that may not have a full record on the other end. This limitation of the platform creates many challenges which includes Users, Manufacturers, computer Models, and many more. Your choice on data import is to either Create an entry in the linked table, leave the field blank, or reject the entire object being imported because this one field has no linked reference. Creating entries is problematic as it can create a lot of orphaned noise, and it is not recommended by ServiceNow. Leaving data blank means we are leaving out important asset information.
The solution to this problem is to write an import script for each of these fields that is linked which contains the best logic possible to handle the gap. If we find no match for a User, we make a record but ensure it is disabled and marked as made by AllSight. If a Company does not exist for a computer Manufacturer, we again make the record but mark it as made by AllSight. This way we can create records and ensure good data import, but the objects we make can easily be identified later for any desired remediation efforts.
Rules must be Broken
ServiceNow has a variety of built in scripts called Business Rules. These are woven through the platform with the intent of automating certain actions and ensuring data integrity. Unfortunately while their intention is to prevent issues, they invariably cause issues. As can be notoriously found in the SN forums, you basically need to disable Business Rules when importing data, otherwise your data import will invariably fail to some degree. However, by disabling these rules in the import, you also remove the automations that the platform relies on in its complexities. One specific example is that because we must disable these rules to have data actually come in reliably, we only have the CMDB records for the computers. In AllSight, there is one computer database, which we believe is the proper clean way to have a single unique asset record linked to all other platform functions. ServiceNow has at least two records for a computer, one in CMDB, the other in Asset. Allegedly the idea is you might manage the computer differently for asset management vs configuration management, though we're not clear on this. Because of the platform limitations, our data import does not create Asset records at this time. This is an issue if you intend to use ServiceNow for your asset management, but we believe AllSight provides for a much more straight forward and very adequate asset management solution by comparison. If you choose to do asset management in SN instead of AllSight, you will need to add your own process for creating the linked Asset records. Presumably this could be a scheduled script in the global context that runs shortly after the import sets in our plugin. Unfortunately we can not trigger such a thing in our plugin due to application context limitations.
While untested, documentation at https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0749624 seems to indicate all you may need to do is create a Model Category (Product Catalog -> Product Models -> Model Categories) for KS Computers. In theory per https://docs.servicenow.com/bundle/washingtondc-it-service-management/page/product/product-catalog/task/t_CreatingModelCategories.html this would simply use KS Computers for the CI class, Hardware Model for the Product model class, and Hardware for the Asset class. Once made their documentation seems to indicate that Asset records should be auto created on the creation of new CI records in our computers table. It is unclear if updates to existing records will also do this, or if you'd need to delete all records and re-populate, or if another post population method is available. Again, we can only offer some possible hints as the intent of this integration is to use ServiceNow as your ITSM and AllSight as your ITAM, so we do not explore the ITAM features of ServiceNow.
Another example of this is the Software install counts. We extend the Software table in "vanilla" ServiceNow to add fields in KS_Products. Business rules are supposed to total up the Instal Count for each Software item based on the links to the computer records. However once again these can not run automatically lest they stop the import entirely, and they can not be triggered from the application context. As a result, while you can view the records and see the linked installs, the count is always 0. Manually adding a schedule to run the needed script to calculate and add these totals is unfortunately therefore left to the customer.
Bad Date
There is a known bug in the platform that was introduced around Quebec and has yet to be addressed. When setting date/time format in a transform map, the displayed pattern is always some default rather than the actual format set in the map. This makes it impossible to verify the current configuration, so one must simply verify by testing that the results are as expected.
There is also a known limitation (counter to some documentation) that time zone is not supported. As such our date/time import values may differ from what you expect based on what you see in AllSight. This is because we store information in Z time and use localization to display in local time. Unfortunately despite the z syntax ServiceNow does not actually support specifying that these values are Z time. As such it's probably you will see the Z times shown as local times and therefore have an offset discrepancy from the original data.
Comma Issue
In attempting to find the best way to import our data, we found that CSV was the only "reliable" and frankly functional option. The one big issue with this is our open platform for data input. If you add values to any of the fields that are imported via this plugin that contains a comma, you have introduced a parsing issue. ServiceNow will silently stop an import when it hits a row where there are too many "columns". There is no log of this issue occurring, and it will not continue with additional rows in the import set. As mentioned in Troubleshooting above, if you see only a portion of your computer records this may be the issue.
Version History
Initial Release
Version 1.0 was the first published release to the ServiceNow Store.
1.1.0
1.5
Several versions were skipped due to timing and significant overhauls.
As always please contact support if you need assistance.