Null values make conditional splits, derived column calculations, script tasks and other components cumbersome to deal with. This in-place column replacement component makes short work of:

• Replacing NULL values with empty strings.
• Replacing empty strings with NULL values.
• Replacing NULL values with a substitute string (e.g. “Unknown”).

Replacing NULL values can also improve the user-acceptance of your reporting environments as well!

This tool is available now for only $9.95 per computer. TacTools for SSIS are provided as individual components so unlike other SSIS task packages, you only buy what you need. And you’ll find our prices to be much lower, per component, than virtually all competitors’ products.

Don’t let NULL field values give you any more headaches in your SSIS packages. Try TacTools for SSIS: Null Manager.

The Method at a Glance

The beauty of the technique is simplicity in itself. We will load the data via a quick, three-step process.

1. Create a staging table with the same structure of the destination table.
2. Load the data into the staging table.
3. Drop the destination table and rename the staging table to the destination table name.

The Basic SSIS Project

Launch Microsoft SQL Server Business Intelligence Development Studio (BIDS) and create your SSIS project. Create your package, and configure your source and destination Connection Managers.

Our Control Flow will consist of three steps as outlined above. We’ll first need an Execute SQL Task followed by a Data Flow Task and finally another Execute SQL Task.

The sources for both Execute SQL Tasks will be your destination Connection Manager.

Create the Staging Table

You can create the required staging script relatively quickly using SQL Server Management Studio (SSMS) as follows:

1. Connect to the database where the destination table is stored.

2. Locate the destination table in the Object Explorer view.

3. Right-click the destination table and select Script table as: DROP and CREATE to: New Query Editor Window. Be careful not to execute your script at this point! It is best to be connected as a user with read-only permission to the database to ensure that this script is not accidently executed.

4. Select Edit: Find and Replace: Quick Replace.

5. In the Find what text box, type [dbo].[<table name>], and in the Replace with box, type [dbo].[STAGE_<tablename>]. For example, if the table is called History, enter [dbo].[History] and [dbo].[STAGE_History] respectively. This will become our load table.

6. Click the Replace All button.

7. If you have any indexes or primary key constraints, you will need to rename these as well in the creation script. A SQL Server database may not contain two indexes with the same logical name. In reality, you should keep only the primary key constraint. Other indexes can be removed from the script altogether to potentially improve load performance.

The script that now exists in your Query Editor Window is the T-SQL script for your first Execute SQL Task. It checks for the existence of the STAGE table, performs a DROP if it exists, and then CREATEs a new STAGE table.

Execute the task and ensure that the STAGE table is created in the destination database. It will need to exist for the next step in our process.

Populating the Staging Table

Next, open the Data Flow Task control. This is where you will load data from the source table to the destination STAGE table. Using the appropriate Data Flow Source, Transformations and Data Flow Destination, build the ETL that you would normally use to populate the destination table. For this example, a very direct load with no transformations is used. You can use any necessary transformation controls for your particular situation.

When the script is executed in sequence, your STAGE table is guaranteed to be empty due to the first Execute SQL Task. In the absence of indexing, this should be a relatively fast load if the data is reasonable. Note that this method may not be the best method for the full load of a table with very wide rows (lots of TEXT or IMAGE type fields).

Return to the Control Flow and execute this task to ensure that the data can load into the STAGE table as expected.

In order to prevent Validation errors prior to the creation of the STAGE table, set the DelayValidation property of the Data Flow Task to True.

The Quick Change

The last step in the process is the quick change technique. This can be scripted into a single T-SQL script and saved into the final Execute SQL Task.

First, script any indexes from the destination table (with the exception of the primary key destination which was created along with the STAGE table above) as CREATE statements via SQL Server Management Studio (SSMS). Script these to new Query Editor windows.

Second, script the destination table (not the STAGE table) as DROP to a new Query Editor window. The DROP statement includes an existing table check.

Now compose a single T-SQL script as follows:

1. In the DROP statement’s Query Editor window, after the GO following the IF…DROP… statement, enter EXEC sp_rename ‘STAGE_<tablename>’, ‘<tablename>’ (include the single quotes). Follow this by a GO statement. This calls the sp_rename stored procedure which allows you to rename a table or other database object. Example: EXEC sp_rename ‘STAGE_History’, ‘History’

2. Next, enter EXEC sp_rename ‘<name you gave the STAGE primary key>’, ‘<name of the real primary key>’ and a GO command. Example: EXEC sp_rename ‘PK_STAGE_History’, ‘PK_History’

3. After this, paste the CREATE INDEX scripts from above in sequence.

Basically, you are writing a script that recreates the entire destination table including all indexes. Below is an example script created from the ReportServer.History table from a default Microsoft SQL Server Reporting Services (SSRS) database. This example assumes that the STAGE table is named STAGE_History and that the primary key on the STAGE table is called PK_STAGE_History.

USE [ReportServer]
GO

/****** Object:  Table [dbo].[History]    Script Date: 02/28/2011 23:34:29 ******/
IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N’[dbo].[History]‘) AND type in (N’U'))
DROP TABLE [dbo].[History]
GO

EXEC sp_rename ‘STAGE_History’, ‘History’
GO

EXEC sp_rename ‘PK_STAGE_History’, ‘PK_History’
GO

CREATE UNIQUE CLUSTERED INDEX [IX_History] ON [dbo].[History]
(
[ReportID] ASC,
[SnapshotDate] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
GO

CREATE NONCLUSTERED INDEX [IX_SnapshotDataID] ON [dbo].[History]
(
[SnapshotDataID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
GO

Since we have tested population of the STAGE table step, it should be relatively safe to execute this script for testing purposes. Please ensure that you have appropriate backups and are using a development environment until you are certain that the script will work properly!

If the test was successful, paste this script into the final Execute SQL Task.

Executing the Package

When you execute the package, you are building a STAGE table separate from the active reporting/production table and populating it. Only during the final Execute SQL Task – which should execute very quickly – is there any table locking or potential for a failed query. Since the scripts check the existence of tables prior to drops and the STAGE table validation in the Data Flow Task is delayed, the package execution should be relatively bullet-proof.

Conclusion

Provided you have the data storage to support it, this is a vastly superior alternative to a TRUNCATE and INSERT method. I have multiple tables that utilize this method in various production environments and have not yet experienced any significant issues from using it. If you decide to load multiple tables in the same script in this way, I encourage you to use Sequence Containers if for no other reason than to group each flow together and gain some control over memory utilization.

Before using any full load method, you need to decide how best to load data for your particular table. Do you require periodic refreshes of table data while applications and/or users are executing reports? This method may be a very reliable (and fast) option. This method is very good for master data-type tables but is not well-suited for large, transactional tables that may be very wide. In most production environments, a variety of load methods may be required.

What is a Reporting Database?

Quite simply, a reporting database is a database that is separate from your production environment but contains data from your production environment. It exists for the purpose of ad hoc queries and reporting without the risk of direct production connection and is optimized with reporting needs rather than operational needs in mind. Additionally, a reporting database may contain data from multiple production applications allowing you to create cross-application dashboards and queries that might not otherwise be possible.

Report Database Architecture

Why do I need a Reporting Database?

There are numerous benefits to using a reporting database. Let’s explore a few of the largest benefits.

1. Report queries do not execute against the production environment.

While it is sometimes convenient to report data in real time, in almost all cases near real time is more than adequate. Most data simply does not change as rapidly as we’d like to think and due to the cyclic nature of reports, having data driven by the second is not very beneficial. It is quite possible to update a reporting database from production on frequent intervals – even every few minutes.

The production environment is also quite busy. It has transactional processing to do, after all. Considering how easy it is to write a bad query (e.g. a full-text search on a non-indexed field with tens of thousands to millions of data rows), and the security considerations needed to prevent damaging queries such as TRUNCATE, DROP, and ALTER statements, executing queries directly against production environments simply isn’t safe.

2. Reporting database tables can be optimized for reporting.

Your production system is optimized via indexing and constraint mechanisms to support your production application. These indexes may not necessarily be adequate for your reporting needs. Consider a help desk ticketing application, for example. Your production application may frequently search for tickets that are open and the assigned technician is a specific person. Your reports, on the other hand, might be more concerned about how long tickets are open or tickets that are closed within a particular date range. While the source data is the same, the WHERE clauses of these queries are vastly different which reason that the indexing for the tables may also need to be vastly different.

Reporting databases can also contain data stored in a different format than production. Your production application may use proprietary data types or may contain high orders of normalization. If you find yourself constantly running the same data parsing expressions or making the same JOIN expressions over and over again for your reporting needs, then you may find performance improvement in storing that data in a denormalized form.

3. Reporting databases can be clean.

While not quite reaching the cleanliness of a full-fledged data warehouse, the nature of the extract-transform-load (ETL) processes that exist between your production database and reporting database allow you to clean data for reporting purposes. Suppose your application contains a field that contains a logical flag – typically “true” or “false” – as a character expression. If the flag isn’t set, you may experience a NULL value as well. The ETL can be written to take this into consideration by substituting NULL values for its appropriate default (likely “false”). Corrupt data records can be discarded, calculated fields can be inserted, and any other benefit of an ETL intermediary can be realized.

4. Reporting databases can contain data from multiple production applications.

This may well be one of the largest benefits to using a reporting database. Data from multiple sources can be combined into a “one-stop-shop” for reporting. This allows some of the benefits of application integration – such as multi-application dashboards – without the need for full application integration. Example: Data loaded from our help desk ticketing application is stored in the same database as data loaded from our call accounting system (CAS). Associate John was on the phone for 4 hours yesterday and opened 27 service tickets for an average of 9 minutes per call. Associate Jane was on the phone for 3 hours and opened 12 service tickets for an average of 15 minutes per call. Perhaps Jane dealt with more demanding customers or could use some additional training.

Designing the Reporting Database

Too often, a developer’s first inclination when configuring a reporting database is to simply copy the production database to another location and start pointing reports to it. Unless the production database is very small, this is generally a very poor approach to a reporting database environment.

What data do you really need?

First, consider the source data. If none of your reports consider transactional data over a year or two years old, there is no need to waste time, resources, and storage moving three year old transactions into the reporting database. Performance is key. If you don’t need to query a million rows to execute your reports adequately, then don’t store a million rows.

Another consideration along this same line is field selection. Most production applications, especially packaged applications, contain some level of legacy data structure. There are fields that exist purely for backwards compatibility to previous versions. There may also be fields necessary to the production application that you may never have a need to report on, as well.

With careful analysis, you may find that reports only require a small percentage of production data to be kept “fresh” in your reporting database and still ensure adequate, meaningful reporting.

Should data be denormalized?

If you are constantly joining two or more tables to generate reports, denormalization might be in order. An example might be one in which you join an orders table to a customer table just to get the customer’s name or zip code. Denormalizing this data would allow you to store the name or zip code within the orders table in your reporting database so that the join can be eliminated. While the benefit may appear negligible, this is only a single example and, in the case of millions of orders, may in itself be significant. While you may not want to store denormalized or redundant data within your production database, these should be considered just fine in a reporting database.

Should you pre-calculate?

Like denormalization, storing calculated values is traditionally frowned upon… for production databases. While you wouldn’t typically store simple, quick calculated fields in a reporting database, tables that could be generated via ETL or stored procedures that allow particular reports to retrieve their results quickly can benefit greatly from pre-calculation. A pivot query result, for example, could be loaded into a table and referenced directly into a report without the need for complicated report logic or long rendering time.

Will data be identifiable?

Production databases, particularly packaged applications or legacy home-grown applications, often contain unrecognizable data. A database field used for one purpose in a past version is now used for a different purpose. You’re looking for field called “lastName” and find the data in a field called “userSur.” Guess what? Your ETL process can rename the field to something more logical in your reporting database. Your reporting database can be structured so that it can be understood by real people and not just by application procedures. Ad hoc reporting has never been easier.

What are your indexing needs?

Reports have different indexing requirements than production applications. Your application requires indexing such that transactional data – usually single records – can be retrieved quickly while reports tend to need to aggregate data together. The queries that service the production application and the reports are seldom even similar. In our earlier example, the production help desk application needed to be able to identify open tickets assigned to particular technicians. The report, on the other hand, needed to identify tickets closed within a particular date range. The index on the status and technician fields turns out to have no use to the report query while an index added to the status and closeDate fields might be extremely useful and improve report performance.

Explore the queries that drive your reports and determine where real indexing benefit will be. Don’t be tempted to simply mirror your production indexes into your report environment.

Will data be sourced from multiple systems?

If so, then you may need to derive a naming convention for the tables within your reporting database. A prefix or suffix added to the table name is a good place to start. This reduces risk of table name collisions, and it facilitates fast identification of the source data. You’ll know just by looking what production application provided the data for a particular table.

How frequently does data need to be updated?

Most report databases do not need data to be as “fresh” as production data. Data that is minutes, hours, and sometimes even days old is sufficient enough to derive meaningful reports. Some data simply doesn’t change very often. Use multiple ETL scenarios if tables and applications have different needs. It might be of great interest for an orders table to be kept fresh for counting purposes but our employees table doesn’t really change very often. We might refresh orders every 30 minutes while we only refresh employees daily.

Note that you may need to derive a “delta” update mechanism for data that refreshes rapidly. If a table needs to be refreshed quickly and frequently, you must be able to identify and query only the recently updated records from your production system. Otherwise, querying the entire production table every time that the refresh scenario executes could have a negative impact on the performance of the production database. There are many methods to doing this kind of action. One common approach is to examine a modification timestamp field in the production source data. Only query records where that value is greater than the last refresh time.

For some data, it may simply be easier and almost as fast to simply snapshot the production table. Truncate the reporting database table and reload it from scratch from production. This method is not recommended for rapid refresh data and should not be used during peak usage of the production application, however. An overnight snapshot of associate records from the payroll system or other relatively static data can benefit from this mechanism.

Conclusion

This article was written to advocate for and demonstrate the use of a reporting database. While we only scratch the surface in this article, the concepts presented here should show you some of the greater benefits and thoughts that need to be considered prior to implementation. Your environment is unique to your organization, as are your reporting needs. There is no “one-size-fits-all” reporting solution, and a dedicated report environment allows you to tailor your application data to best fit your organization. I encourage you to consider exploring a dedicated reporting environment simply for the fact that it’s safer and more robust than reporting directly from production data, and it allows greater flexibility and creativity for report designers as well. Plus, it’s simply more “fun” when you can play with and manipulate the data without the risk of impacting a production application.

In future posts, we will explore various reporting environment solutions and examples that will hopefully open entirely new doors to your organization in terms of valuable data analysis opportunities.

Happy reporting!

Our Example

First, let’s take a look at the data that we’re going to split. Our CUSTOMERS table contains two arrays – FIRST_NAMES and LAST_NAMES – that are “position specific.” Basically, the first FIRST_NAME value needs to be matched to the first LAST_NAME value and so forth.

Looking at the table, we see that Vanessa’s last name is missing. We will need to be able to manage “missing” values as indexes. Neither index-positioning nor missing values can be appropriately handled with the function that we explored yesterday, so we’ll make some modifications.

The Original fnSplit() Function

Let’s look at the original fnSplit() function.

CREATE FUNCTION [dbo].[fnSplit](
    @sInputList VARCHAR(MAX) -- List of delimited items
  , @sDelimiter VARCHAR(MAX) = ',' -- delimiter that separates items
) RETURNS @List TABLE (item VARCHAR(MAX))
 
BEGIN
	DECLARE @sItem VARCHAR(MAX)
	WHILE CHARINDEX(@sDelimiter,@sInputList,0) <> 0
	BEGIN
		SELECT
		@sItem=RTRIM(LTRIM(SUBSTRING(@sInputList,1,
			CHARINDEX(@sDelimiter,@sInputList,0)-1))),
		@sInputList=RTRIM(LTRIM(SUBSTRING(@sInputList,
			CHARINDEX(@sDelimiter,@sInputList,0)+LEN(@sDelimiter),
			LEN(@sInputList))))
 
		IF LEN(@sItem) > 0
		INSERT INTO @List SELECT @sItem
	END
 
	IF LEN(@sInputList) > 0
	INSERT INTO @List SELECT @sInputList -- Put the last item in
	RETURN
END

We can then CROSS APPLY our fnSplit() function within a SELECT query of our CUSTOMERS table, but we can only reference one field at a time this way.

SELECT C.ID, F.ITEM
FROM CUSTOMERS C
CROSS APPLY dbo.fnSplit(FIRST_NAMES, ',') F

Our result looks like the following:

If we run the same query and substitute the LAST_NAMES fields, the problem becomes apparent.

Whereas seven rows were returned in the FIRST_NAMES query, only six are returned in the LAST_NAMES query. We can surmise that the “missing” name is from the second row since only three rows were returned from that row. But which first name is missing a last name? With our existing function, there is no systematic way to tell, and there is no way to join the two queries reliably.

The Enhanced fnSplit() Function

To fix this problem, we need only make some revision to our fnSplit() function. We need to ensure that null entries are retained, and we need a position index number that we can use in joins.

ALTER FUNCTION [dbo].[fnSplit](
    @sInputList NVARCHAR(MAX) -- List of delimited items
  , @sDelimiter NVARCHAR(MAX) = ',' -- delimiter that separates items
) RETURNS @List TABLE (ARRAY_IDX INT, ITEM NVARCHAR(MAX))
 
BEGIN
	DECLARE @sItem NVARCHAR(MAX)
	DECLARE @sIndex INT
	SET @sIndex = 1
	IF CHARINDEX(@sDelimiter,@sInputList,0) > 0
	BEGIN
		WHILE CHARINDEX(@sDelimiter,@sInputList,0) > 0
		BEGIN
			SELECT
			@sItem=RTRIM(LTRIM(SUBSTRING(@sInputList,1,
				CHARINDEX(@sDelimiter,@sInputList,0)-1))),
			@sInputList=RTRIM(LTRIM(SUBSTRING(@sInputList,
				CHARINDEX(@sDelimiter,@sInputList,0)+LEN(@sDelimiter),
				LEN(@sInputList))))
 
			IF LEN(@sItem) > 0
				INSERT INTO @List SELECT @sIndex, @sItem
			ELSE
				INSERT INTO @List SELECT @sIndex, NULL
 
			SET @sIndex = @sIndex + 1
		END
 
		IF LEN(@sInputList) > 0
			INSERT INTO @List SELECT @sIndex, @sInputList
	END
	ELSE
		INSERT INTO @List SELECT 1, NULL
 
	RETURN
END

Here, we have made three changes. First, we modified the IF condition that checks for string length and added an ELSE clause to inject a NULL value if the string length is equal to zero. Second, we added an integer counter variable called @sIndex that increases with each iteration of the WHILE loop block. This index will be returned as a column called ARRAY_IDX. Finally, if there are no valid entries at all, we create a NULL table with a single row. This ensures that we will always get an index of at least 1, even if the value is NULL.

Now, we execute the following query:

SELECT C.ID, F.ARRAY_IDX, F.ITEM
FROM CUSTOMERS C
CROSS APPLY dbo.fnSplit(FIRST_NAMES, ',') F

And our result should look like this:

And, when we substitute the LAST_NAMES field, the result appears like this:

We now have a retained NULL value in the third position of the second array. We also have an array index field (position identifier) that we can use to build a join.

SELECT C.ID, F.ITEM AS FIRST_NAME, L.ITEM AS LAST_NAME
FROM CUSTOMERS C
CROSS APPLY dbo.fnSplit(C.FIRST_NAMES, ',') F
CROSS APPLY dbo.fnSplit(C.LAST_NAMES, ',') L
WHERE F.ARRAY_IDX = L.ARRAY_IDX

And achieve the following result:

It is now apparent that someone (or some process) neglected to record Vanessa’s last name. Additionally, we have joined two array fields based on their index positions.

Some Thoughts and Challenges

This technique works well if both arrays are ensured to contain the same number of elements. If you are faced with joining arrays that contain different numbers of elements but still wish to retain correlation between indexes, you could contain each CROSS APPLY clause into a sub-SELECT and use a LEFT OUTER JOIN to accomplish this. It may also be beneficial to place common CROSS APPLY queries into a VIEW so you aren’t burdened with remembering the syntax and wish to avoid a potential for multiple, nested sub-SELECT statements.

Conclusion

In adding position indexing to the fnSplit() function, you have a powerful function that can be used to manage all sorts of delimited data types. I have replaced fnSplit() on virtually every SQL Server database that I manage primarily for the added functionality. The additional field can always be left out of the SELECT statement when it’s not needed. I hope that you will experience similar benefits to this function as I have.

Fortunately for us, this data can easily be relocated out of large data fields and moved into a real, relational table in a form that can be indexed for rapid query. This article explains how.

Fixing this issue is relatively easy but does make changes to the underlying database. This is something important to take into consideration when planning upgrades. While converting your data as this article describes does create some upgrade conflict, the differences are not difficult to work through.

Updating the Database Dictionary Record

First, make a backup of your existing database tables as well as the database dictionary (dbdict) records that you are planning to change. Preferably, backup the entire database just in case something goes wrong. Of course, we’re not planning that it will, but it’s better to be safe than sorry!

For our conversion, we’ll be updating the “approvers” field of the “ocmgroups” file.

1.       Open the “ocmgroups” dbdict record via the Tailoring->Database Dictionary menu option.

2.       Select the “SQL Tables” tab.

3.       Look for an alias that begins with the letter “a” (e.g. a1). There may be zero or more of these listed. We want to know the “next available” alias number. If “a1” exists, then the next available is “a2” and so forth. On a fresh installation of HPSM 9.20, “a2” is the next available alias for the “ocmgroups” file.

4.       Select the “Fields” tab.

5.       Locate the field that we want to change. For our example, this is the “approvers” field.

6.       Array fields get two line entries in the dbdict record – one for the array and one for the actual database field. Open the first of the two (the array).

7.       In the “SQL Table” field, enter the next available alias from Step #3. We will enter “a2” (minus the quotes).

8.       Select “OK” and open the second array line (the database field).

9.       Change the “SQL Type” to the appropriate type for your RDBMS. I’m using Microsoft SQL Server, and this field stores operator names. A look at the operator table shows me that the names are stored as VARCHAR(60) in the database. I change “TEXT” to “VARCHAR(60).”

10.   Change the “SQL Table” field to the same alias as the array. We enter “a2” into this field as well.

11.   Click “OK” and allow HP Service Manager to update the database tables.

Conclusion

That’s all there is to it. You now have a new table in your database called OCMGROUPSA2 that stores the array values as VARCHAR(60) fields instead of in a TEXT field. These can be indexed by the RDBMS. You can add keys to improve query performance, and simply making this change will improve the performance of queries that use the field whether or not indexing is used. HPSM does not have to store large fields in memory and parse them apart to find values.

I believe that this technique works in all versions of HPSM that use an RDBMS. I’ve had success implementing such relationships on 6.14, 6.2, 7.01, 7.11, and 9.20.

Although the example in this article applies to HP Service Manager, the method can be used whenever delimited data needs to be split and joined to another table in Microsoft SQL Server.

In HP Service Manager, lists of data associated to a particular record are often stored in a proprietary data type called an “array” rather than a relational, foreign-key table. If you’re using Microsoft SQL Server as your backend database, you will find array data stored in TEXT or NTEXT type fields as a list (Oracle databases use CLOB types). It would be much more beneficial if this data could be represented as relational data so that it could be reported on or joined to another table for more information.

Our Example

Bill, our CFO, would like to send an e-mail to everyone listed as a Request Management approver in HP Service Manager reminding them of a budget freeze. He asks us for a list of approvers and their e-mail addresses so that he can target the e-mail rather than send a company-wide communication.

Special Note: Our example uses an out-of-box installation of HP Service Manager 9.20 and sample data.

The Data

Request Management groups (both members and approvers) are located in the ocmgroups file in HP Service Manager. A quick look at a Request Management group record in HPSM shows us members and approvers as lists. Approvers are displayed as operator names. Additionally, we need e-mail addresses. Both contact names that Bill would recognize and e-mail addresses can be found in the operator file.

HPSM Request Mgmt Group

A glance at the dbdict record for the ocmgroups and operator files shows us that the fields we want are stored in the OCMGROUPSM1 and OPERATORM1 tables within SQL Server.

We should be able to simply query the OCMGROUPSM1, join to OPERATORM1 and get the names and e-mail addresses, right? Unfortunately, it’s not quite that easy.

The Problem

The OCMGROUPSM1 table contains a TEXT type field called APPROVERS that can’t be directly joined to the OPERATORM1 table. The entire list of approvers is stored in a single field with a non-printable character delimiter. The character (both SQL Server and Oracle) is a new-line character (ASCII decimal value of 10). A simple join query will not work for our situation.

OCMGROUPSM1

We need to be able to represent the array data in a way that can be joined to another table.

SQL Server Array Splitting Function – fnSplit

For the first part of our solution, we will build a function that splits array data into something more understandable. We will actually go one step further and build a function that can split data on any delimiter, not just the new-line character.

In tackling this for the first time, I started with a simple SQL Server function that I found via an Internet search. The source code is as shown below:

CREATE FUNCTION [dbo].[fnSplit](
    @sInputList VARCHAR(MAX) -- List of delimited items
  , @sDelimiter VARCHAR(MAX) = ',' -- delimiter that separates items
) RETURNS @List TABLE (item VARCHAR(MAX))

BEGIN
	DECLARE @sItem VARCHAR(MAX)
	WHILE CHARINDEX(@sDelimiter,@sInputList,0) <> 0
	BEGIN
		SELECT
		@sItem=RTRIM(LTRIM(SUBSTRING(@sInputList,1,
			CHARINDEX(@sDelimiter,@sInputList,0)-1))),
		@sInputList=RTRIM(LTRIM(SUBSTRING(@sInputList,
			CHARINDEX(@sDelimiter,@sInputList,0)+LEN(@sDelimiter),
			LEN(@sInputList))))

		IF LEN(@sItem) > 0
		INSERT INTO @List SELECT @sItem
	END

	IF LEN(@sInputList) > 0
	INSERT INTO @List SELECT @sInputList -- Put the last item in
	RETURN
END

To see how this function works, create it and execute the following query:

SELECT * FROM dbo.fnSplit('Bill,Janet,Sarah,Tiffany,Vanessa,William,Xander', ',')

The data is split by comma-separated values and returned as a list.

In our example, the character delimiter is a non-printable new-line character. We can use SQL Server’s CHAR() function to manage this. CHAR() accepts a decimal value and returns the corresponding ASCII character. The new-line character has a decimal value of 10 and can be represented in a SQL Server query as CHAR(10). You will see how this is used in the next section.

Using fnSplit in a SQL Server Query using CROSS APPLY

Our goal is to get the data into a relational format that we can join to another table. We can use SQL Server’s CROSS APPLY syntax to accomplish this. CROSS APPLY allows us to join a table to a table-value function (TVF) result set within SQL Server.

SELECT M1.NAME, S.ITEM
FROM OCMGROUPSM1 M1
CROSS APPLY fnSplit(M1.APPROVERS, CHAR(10)) S

Our result should now be something that we can much more easily join to another table as shown in the screen shot.

SELECT CROSS APPLY

We need only join this result to the OPERATORM1 table to get our extra information – the CONTACT_NAME and EMAIL fields.

SELECT M1.NAME, S.ITEM, O.CONTACT_NAME, O.EMAIL
FROM OCMGROUPSM1 M1
CROSS APPLY fnSplit(M1.APPROVERS, CHAR(10)) S
JOIN OPERATORM1 O ON O.NAME = S.ITEM

Cleaning up the Result

Just for polish, we can clean this data up to provide our CFO, Bill, with the result of his specific request. We currently show duplicate names and e-mail addresses when the same person might approve on behalf of multiple Request Management groups.

We’ll just limit the result set to the columns we need, use a SELECT DISTINCT and an ORDER BY clause to get Bill the answer in a format that he wants.

SELECT DISTINCT O.CONTACT_NAME, O.EMAIL
FROM OCMGROUPSM1 M1
CROSS APPLY fnSplit(M1.APPROVERS, CHAR(10)) S
JOIN OPERATORM1 O ON O.NAME = S.ITEM
ORDER BY CONTACT_NAME

Conclusion

Array data can be difficult to work with especially when attempting to access it through database queries. Breaking components out of the model file, generating pending approver lists from the Approval file, and many other challenges can be solved by adopting the technique laid out in this article. The best part is that the fnSplit function is added to your SQL Server database and used whenever you need it.

While the example that was used in this article is specific to the HP Service Manager application, I have encountered numerous other databases where data is stored as delimited lists rather than as relational data. The method of using the fnSplit function and CROSS APPLY is universal.

In a future post, we will modify the fnSplit function to include NULL values (an empty value between delimiters) as well as an index column identifying the position of the item in the list.

Pre-Installation Requirements

Before setting up your local HPSM installation, you will need to download some software packages.

• Microsoft SQL Server 2008 Express with Tools – The SQL Server 2005 edition bundled with the HP E-Media is not fully compatible with Windows 7 and may not even install properly. Microsoft offers SQL Server Express both with and without the client tools. You will need the edition with the client tools in order to configure your database. Select either the 32-bit or 64-bit edition as required by your operating system.
• HP Service Manager 9.20 English SW E-Media – The core application is available from HP’s Software Support Website. You will need your Support Authorization ID (SAID) number from your HP order.

Installing and Configuring the Database

The database should be installed before the other components. The HPSM configuration process will require an configured, empty database to be available. This section assumes that you do not currently have a SQL Server instance installed on your local system.

1. Launch the Microsoft SQL Server 2008 Express with Tools installer. You may be prompted with a compatibility warning which is safe to ignore at this time. We will reconcile that later.

SQL Server Installation Center

2. From the SQL Server Installation Center, select the Installation menu.

3. Select the New SQL Server stand-alone installation… option.

4. On the Feature Selection screen, click the Select All button.

SQL Server Feature Selection

5. On the Instance Configuration screen, select Default Instance.

6. On the Server Configuration screen, click the button labeled Use the same account for all SQL Server services.

7. In the Account Name drop-down, select NT AUTHORITY\NETWORK SERVICE.

SQL Server Configuration

8. Leave the Password field blank and click OK.

9. On the Database Engine Configuration screen, select Mixed Mode authentication and select a password for your system administrator (sa) user. Also, click the Add Current User button at the bottom to ensure that you will be able to configure the SQL Server via your Windows account later.

SQL Server Database Engine

10. Complete the installation wizard.

11. Once the installation wizard completes, run your Windows Update service to download the latest SQL Server 2008 Service Pack. This fixes the compatibility warning you may have been presented with in Step #1.

Note: At the time of this writing, SQL Server 2008 Service Pack 1 is available directly from Microsoft’s website and can be installed alone without Windows Update.

12. Launch the SQL Server Configuration Manager (Start->All Programs->Microsoft SQL Server 2008->Configuration Tools).

13. Under SQL Server Network Configuration, select Protocols for SQLEXPRESS.

14. Right-click the TCP/IP protocol option and select Enable.

SQL Server Configuration Manager

15. Select SQL Server Services.

16. Right-click the SQL Server (SQLEXPRESS) service and select Restart.

17. Launch the SQL Server Management Studio (Start->All Programs->Microsoft SQL Server 2008).

SQL Server MS Connect

18. In the Server Name field, enter localhost\SQLEXPRESS.

19. Select SQL Server Authentication.

20. In the Login field, enter sa, and enter the password specified in Step #9.

21. Once connected, right-click the Databases folder in the Object Explorer and select New Database…

SQL Server New Database

22. In the Database Name field, enter a name for the Service Manager 9.20 database (e.g. sm920).

Configuring ODBC Data Source

HP Service Manager connects to the database via an ODBC Data Source Name (DSN). Next, we will configure a DSN that will be used to connect your HPSM application to the database you just created.

Note: Instructions for this are different for 32-bit and 64-bit Windows. You will need a 32-bit DSN for HP Service Manager.

32-bit Windows

Launch the ODBC Manager by selecting Start->Control Panel. Then Administrative Tools->Data Sources (ODBC).

64-bit Windows

Click the Start button and, in the Search programs and files box, enter c:\windows\sysWOW64\odbcad32.exe. You should see odbcad32 in the program list. Click the icon.

From this point on, directions are the same.

23. Select the System DSN tab and click the Add… button.

Create ODBC Data Source

24. Select the SQL Server driver and click Finish.

25. Enter a name for the data source (e.g. sm920) and localhost\SQLEXPRESS as the Server and click Next.

26. Select the With SQL Server authentication… option. Enter sa as the Login ID and the password specified in Step #9 and click Next.

27. Check the Change the default database to check box and select the database you created in Step #22.

28. Click Next then Finish.

ODBC Test

29. Make sure that your DSN is going to work correctly by clicking the Test Data Source… button.

Installing the HP Service Manager 9.20 Server

Once the database and data sources are both set up properly and you’ve been able to establish an ODBC connection via the test, you are ready to install and configure the HP Service Manager server.

30. Open the HP Service Manager 9.20 English SW E-Media and browse to the Server\Windows folder.

31. Right-click the setupserver application and click Properties.

HPSM Server Compatibility

32. On the Compatibility tab, check the Run this program in compatibility mode for… check box and select Windows Vista (Service Pack 2). Also, check the Run this program as an administrator check box.

33. Click Apply and OK.

34. Double-click the setupserver application to launch the HPSM server installation wizard.

HPSM Server Install

35. Select the default options through the wizard, and on the final screen, select the Run the configuration program after install check box and click the Finish button.

Configure the HP Service Manager 9.20 Server

The HP Service Manager Configuration appears, and we are ready to configure the server settings as well as install the core application structure and data into the database.

HPSM Configuration

36. Select the default HTTP Port (13080). You may select a different port if necessary for your installation (i.e. if you already have another service or application running on port 13080).

HPSM Config DB

37. On the Database Type and Connection Information screen, enter the name of the DSN that you specified in Step #25. The User is sa and the Password is the password that you specified in Step #9.

38. On the next screen, click the Verify Connection button to ensure that the database link is working properly. If the connection cannot be made and you are sure that the service is running, ensure that you used the correct method (32-bit or 64-bit) for creating the ODBC DSN as specified previously.

39. The next screen is the HP Service Manager Application and Demo data upload screen. Click the Upload Data button to populate the database. This process may take several minutes to complete.

40. Complete the configuration wizard, and your HP Service Manager 9.20 server is ready to launch.

41. Start the HP Service Manager 9.20 Server service via the Windows Service console.

Install the HP Service Manager 9.20 Windows Client

Now that your HP Service Manager 9.20 Server is ready to use, you’ll need a client to connect to it.

42. Open the HP Service Manager 9.20 English SW E-Media and browse to the Client folder.

43. Right-click the setupclient application and select Properties.

44. On the Compatibility tab, select to run the program in compatibility mode for Windows Vista (Service Pack 2). Otherwise, visual problems with Windows 7 Aero and the installer will prevent controls like buttons and check boxes to display properly.

45. Launch the setupclient application and select the default values for the wizard.

46. Once the installation is complete, launch the HP Service Manager client.

47. In the Connections dialog, click the New launch configuration button.

48. Enter the information to connect to your local installation. The Server host name is localhost, and the port is 13080 (unless you specified a different value in Step #36). Initially, use the user name falcon with a blank password to make your initial connection.

49. Click Connect.

Conclusion

You now have a functional, local installation of HP Service Manager 9.20 server and client on Windows 7. These instructions should work for both 32-bit and 64-bit editions of Windows. I ran through this tutorial on Windows 7 Professional, but the instructions should be the same for other Windows 7 editions as well.

In a future post, I will discuss setting up the web tier on Apache Tomcat locally as well so that complete development can be managed on the local PC.