Showing posts with label UNPIVOT. Show all posts
Showing posts with label UNPIVOT. Show all posts

Tuesday, April 12, 2011

T-SQL Tuesday #017: APPLY: It Slices! It Dices! It Does It All!

T-SQL TuesdayThis blog entry is participating in T-SQL Tuesday #017, hosted this month by Matt Velic.

You are invited to visit his blog and join the party and read more blogs participating in this month’s theme: APPLY Knowledge.

I’ve been a fan of the APPLY operator since the beginning. That’s why it’s incorporated into the name of my blog. And I've blogged about its power about a dozen times.

I don’t know how anyone lived without it before SQL2005.

Practically everyone knows that you can use it to invoke table-valued functions. That is its most obvious usage and it’s about the only way you’ll see APPLY demonstrated in 97% of the books on SQL Server (if they even mention it at all).

APPLY is a cool cat, baby!But, as you can see by the twinkle in APPLY’s eyes at the left (even behind the cool shades), he’s got other things up his sleeve.

(Okay, the goofy pointed hat and the goatee are a little much, but hey, if you want great artwork, you’re in the wrong place… Michael J. Swart or Kendra Little are the masters of illustration).

APPLY is capable of soooooo much more than just invoking TVF’s. It is incredibly versatile. It seems like it can do anything!

Let’s take a look at what you can do!



Call Table-Valued Functions!

Yawn. Okay, so this is the most common way people use APPLY. Here’s a quick demo… For each Vista credit card expiring in Jun2008, let’s get the Contact information for that card using a built-in function in AdventureWorks.

select f.FirstName
,f.LastName
,f.JobTitle
,f.ContactType
,cc.CardNumber
from Sales.CreditCard cc
join Sales.ContactCreditCard ccc on cc.CreditCardID=ccc.CreditCardID
cross apply dbo.ufnGetContactInformation(ccc.ContactID) f
where cc.ExpYear=2008
and cc.ExpMonth=6
and cc.CardType='Vista'
/*
FirstName LastName JobTitle ContactType CardNumber
--------- -------- ---------------- ------------- --------------
Peggy Justice Owner Store Contact 11119759315644
John McClane Purchasing Agent Store Contact 11119490672347
Laura Cai NULL Consumer 11112813884091
Natalie Gonzales NULL Consumer 11114369564985
Jarrod Sara NULL Consumer 11116045498593
Katherine Bailey NULL Consumer 11119100149656
Stephanie Gray NULL Consumer 11112324154556
Shawna Sharma NULL Consumer 11116413893110
Mindy Rai NULL Consumer 11115163407997
Jackson Jai NULL Consumer 11112011871602
And so on... (74 rows total)
*/
That’s very convenient, but kind of boring in the grand scheme of things. So let’s move on.

Execute SubQueries!

Why even bother to create a function when you can just create a table on the fly via a correlated subquery with APPLY?

You are only limited by your imagination.

Here’s an example…

For each store with a main office in Wisconsin, let’s look at the top 3 products (and their dollar amounts) that they bought in terms of dollars.

select c.CustomerID
,s.Name
,f.ProductID
,ProductName=p.Name
,f.PurchaseAmt
from Sales.Customer c
join Sales.Store s on c.CustomerID=s.CustomerID
join Sales.CustomerAddress ca on c.CustomerID=ca.CustomerID
join Person.Address a on ca.AddressID=a.AddressID
join Person.StateProvince sp on a.StateProvinceID=sp.StateProvinceID
cross apply (select top 3 ProductID
,PurchaseAmt=sum(LineTotal)
from Sales.SalesOrderHeader soh
join Sales.SalesOrderDetail sod on soh.SalesOrderID=sod.SalesOrderID
where CustomerID=c.CustomerID
group by ProductID
order by sum(LineTotal) desc) f
join Production.Product p on f.ProductID=p.ProductID
where ca.AddressTypeID=3 --MainOffice
and sp.StateProvinceCode='WI'
/*
CustomerID Name ProductID ProductName PurchaseAmt
---------- --------------------------- --------- -------------------------- ------------
418 Good Bike Shop 795 Road-250 Black, 52 30367.350000
418 Good Bike Shop 794 Road-250 Black, 48 24136.807500
418 Good Bike Shop 792 Road-250 Red, 58 23508.517500
453 Unique Bikes 773 Mountain-100 Silver, 44 16319.952000
453 Unique Bikes 771 Mountain-100 Silver, 38 12239.964000
453 Unique Bikes 772 Mountain-100 Silver, 42 12239.964000
543 Friendly Neighborhood Bikes 782 Mountain-200 Black, 38 9638.958000
543 Friendly Neighborhood Bikes 868 Women's Mountain Shorts, M 671.904000
543 Friendly Neighborhood Bikes 869 Women's Mountain Shorts, L 335.952000
606 Little Bicycle Supply Shop 717 HL Road Frame - Red, 62 1717.800000
606 Little Bicycle Supply Shop 838 HL Road Frame - Black, 44 858.900000
606 Little Bicycle Supply Shop 738 LL Road Frame - Black, 52 809.328000
*/
That’s pretty slick, huh?

Shred XML!

Using the .nodes() function, coupled with the .value() and .query() functions, we can use APPLY to do some cool tricks with XML.

For the first 10 JobCandidates, let’s pull information out of the Resume column, which is of type XML. We’ll get their Name and the schools (there might be more than one) that they attended, listing them in order of their graduation date.:

with xmlnamespaces 
(
'http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/Resume' as ns
)
select JobCandidateID
,Name
,Education=stuff(EduList,1,2,'')
from HumanResources.JobCandidate
cross apply
Resume.nodes('/ns:Resume') F_ResumeNode(ResumeNode)
cross apply

ResumeNode
.nodes('(./ns:Name)') F_NameNode(NameNode)
cross apply

(select Name=NameNode.value('(./ns:Name.First[1])','nvarchar(50)')
+' '
+NameNode.value('(./ns:Name.Last[1])','nvarchar(50)')
) F_Name
cross apply
(select EduList=ResumeNode.query('for $p in (./ns:Education)
order by $p/ns:Edu.EndDate
return concat("; ",string($p/ns:Edu.School))'
).value('.','nvarchar(200)')
) F_Edu
where JobCandidateID<=10
/*
JobCandidateID Name Education
-------------- -------------------- ----------------------------------------------------
1 Shai Bassli Midwest State University
2 Max Benson Evergreen High School ; Everglades State College
3 Krishna Sunkammurali Western University
4 Stephen Jiang Louisiana Business College of New Orleans
5 Thierry D'Hers Université d'Aix-Marseille
6 Christian Kleinerman Lycée technique Émile Zola ; Université de Perpignan
7 Lionel Penuchot Université de Lyon
8 Peng Wu Western University
9 Shengda Yang Evergreen High School ; Soutern State College
10 Tai Yee Midwest State University
*/
As Miley Cyrus would say: That’s really cool.

Introduce New Columns!

This is probably the best use of APPLY because it makes code so much more clear.

Consider the following query, which groups the 2002 Sales by Month. That’s done by the DATEADD/DATEDIFF logic, but it has to be repeated in the GROUP BY and the SELECT and the ORDER BY:

select Mth=datename(month
,dateadd(month
,datediff(month,'19000101',OrderDate)
,'19000101'))
,Total=sum(TotalDue)
from Sales.SalesOrderHeader
where OrderDate>='20020101'
and OrderDate<'20030101'
group by dateadd(month
,datediff(month,'19000101',OrderDate)
,'19000101')
order by dateadd(month
,datediff(month,'19000101',OrderDate)
,'19000101')
/*
Mth Total
--------- ------------
January 1605782.1915
February 3130823.0378
March 2643081.0798
April 1905833.9088
May 3758329.2949
June 2546121.9618
July 3781879.0708
August 5433609.3426
September 4242717.7166
October 2854206.7518
November 4427598.0006
December 3545522.738
*/
I don’t know about you, but I hate all that repetition, and it looks a little busy. So APPLY to the rescue:

select Mth=datename(month,FirstDayOfMth)
,Total=sum(TotalDue)
from Sales.SalesOrderHeader
cross apply
(
select FirstDayOfMth=dateadd(month
,datediff(month,'19000101',OrderDate)
,'19000101')
) F_Mth
where OrderDate>='20020101'
and OrderDate<'20030101'
group by FirstDayOfMth
order by FirstDayOfMth
/*
Mth Total
--------- ------------
January 1605782.1915
February 3130823.0378
March 2643081.0798
April 1905833.9088
May 3758329.2949
June 2546121.9618
July 3781879.0708
August 5433609.3426
September 4242717.7166
October 2854206.7518
November 4427598.0006
December 3545522.738
*/
Now isn’t that much clearer as to what’s going on? And it costs nothing at all! The query plans of both of the queries above are exactly the same!

Perform Complicated Calculations!

This is the part of APPLY that I really love. Let’s look at an example.

Let’s say that you have a table of comma-delimited lists of one or more integers:

create table #t
(
ID int identity(1,1)
,ListOfNums varchar(50)
)
insert #t
values ('279,37,972,15,175')
,('17,72')
,('672,52,19,23')
,('153,798,266,52,29')
,('77,349,14')
select * from #t
/*
ID ListOfNums
-- -----------------
1 279,37,972,15,175
2 17,72
3 672,52,19,23
4 153,798,266,52,29
5 77,349,14
*/
Your job: Pull out only the rows that have the 4th number in the list less than 50 and sort the output by the 3rd number in the list.

Easy, right? Ha ha ha ha ha ha ha hee hee hee hee hee hee ho ho ho ho haw haw giggle chuckle guffaw!

Before the APPLY operator, SQL2000 folks would have to resort to something ludicrous like this in order to accomplish this task:

select ID
,ListOfNums
from #t
where substring(ListOfNums+',,,,',charindex(',',ListOfNums+',,,,',
charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,')+1)+1)+1,
(charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,',
charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,')+1)+1)+1)-
charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,',
charindex(',',ListOfNums+',,,,')+1)+1))-1)
< 50
order by substring(ListOfNums+',,,,',charindex(',',ListOfNums+',,,,',
charindex(',',ListOfNums+',,,,')+1)+1,(charindex(',',ListOfNums+',,,,',
charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,')+1)+1)-
charindex(',',ListOfNums+',,,,',charindex(',',ListOfNums+',,,,')+1))-1)
/*
ID ListOfNums
-- -------------
2 17,72
5 77,349,14
3 672,52,19,23
1 279,37,972,15
*/
But now, through the magic of APPLY, you can have a much clearer query:

select ID
,ListOfNums
from #t
cross apply (select WorkString=ListOfNums+',,,,') F_Str
cross apply (select p1=charindex(',',WorkString)) F_P1
cross apply (select p2=charindex(',',WorkString,p1+1)) F_P2
cross apply (select p3=charindex(',',WorkString,p2+1)) F_P3
cross apply (select p4=charindex(',',WorkString,p3+1)) F_P4
cross apply (select Num3=convert(int,substring(WorkString,p2+1,p3-p2-1))
,Num4=convert(int,substring(WorkString,p3+1,p4-p3-1))) F_Nums
where Num4<50
order by Num3
/*
ID ListOfNums
-- -------------
2 17,72
5 77,349,14
3 672,52,19,23
1 279,37,972,15
*/
See how I used APPLY to write a little program of a sort? First, I added commas to the end of the column to account for possibly missing numbers in the list. Then I calculated the position of the first comma in that string (p1). Then I calculated the position of the second comma (p2), and that can only be done by using the p1 position I calculated in the previous step. I continue on getting the position of the third and fourth comma. And now that I have those, I can pull out Num3 (from between the second and third comma) and Num4 (from between the third and fourth comma). And I can now use those values in my WHERE and ORDER BY clause.

And the best part? NO COST! The above two queries are exactly the same as far as the optimizer is concerned. All those CROSS APPLYs are glommed together into a Compute Scalar operator, essentially coming up with really complicated expressions like you see in the first query. Take a look at the query plan yourself and you’ll see.

Replace the UNPIVOT operator!

Throw the UNPIVOT operator out the window… The optimizer really translates it into an APPLY operator under the hood anyway… and you can have control over NULLs and differing datatypes.

Look at the following example, which is a function that accepts a CustomerID and spits out information on the customer in a vertical fashion (note that there can be multiple contacts for a customer… this just spits out the first one… thus the ROW_NUMBER() logic):

create function VerticalMainOfficeData
(
@CustomerID int
)
returns table
as
return
with
BaseData as
(
select SeqNo=row_number() over (order by ContactType)
,Name,AddressLine1,AddressLine2,City,StateProvinceName
,PostalCode,CountryRegionName
,ContactType,ContactName,Phone,EmailAddress
,YearOpened,NumberEmployees,Specialty
,SquareFeet,Brands,AnnualSales
from AdventureWorks.Sales.vStoreWithDemographics
cross apply (select ContactName=isnull(Title+' ','')
+FirstName+' '
+isnull(MiddleName+' ','')
+LastName
+isnull(' '+Suffix,'')) F_Name
where CustomerID=@CustomerID
and AddressType='Main Office'
)
select Property,Value
from BaseData
cross apply
(values ('NAME AND ADDRESS:','')
,(' Name',Name)
,(' Address',AddressLine1)
,(' ',AddressLine2)
,(' City',City)
,(' State/Province',StateProvinceName)
,(' Postal Code',PostalCode)
,(' Country/Region',CountryRegionName)
,('','')
,('CONTACT:','')
,(' Type',ContactType)
,(' Name',ContactName)
,(' Phone',Phone)
,(' EmailAddress',EmailAddress)
,('','')
,('DEMOGRAPHIC INFO:','')
,(' Year Opened',str(YearOpened,4))
,(' Number of Employees',convert(varchar(10),NumberEmployees))
,(' Specialty',Specialty)
,(' Square Feet',convert(varchar(10),SquareFeet))
,(' Brands',Brands)
,(' Annual Sales','$'+convert(varchar(20),AnnualSales,1))) P(Property,Value)
where SeqNo=1
and Value is not null
Watch it in action:

select * from VerticalMainOfficeData(34)
/*
Property Value
--------------------- -----------------------------
NAME AND ADDRESS:
Name Cycles Wholesaler & Mfg.
Address Science Park South, Birchwood
Stanford House
City Warrington
State/Province England
Postal Code WA3 7BH
Country/Region United Kingdom

CONTACT:
Type Owner
Name Ms. Barbara J. German
Phone 1 (11) 500 555-0181
EmailAddress barbara4@adventure-works.com

DEMOGRAPHIC INFO:
Year Opened 1999
Number of Employees 15
Specialty Touring
Square Feet 21000
Brands 4+
Annual Sales $800,000.00
*/
Can UNPIVOT do that? Not in a million years. It’s toast!

Make JOINs Extinct!
`
Okay, this is really kind of a joke, but really, think about it… What is a JOIN? For each row in the first table, I want to JOIN it somehow with a row or rows in the second table. That sounds like an APPLY type of thing, doesn’t it? Well it is!

Look at the following traditional JOIN query, which finds all the Accessories that are Yellow, Blue, White:

select SubCategoryName=s.Name 
,p.ProductID
,ProductName=p.Name
,p.Color
from Production.ProductSubCategory s
join Production.Product p on s.ProductSubcategoryID=p.ProductSubcategoryID
where s.ProductCategoryID=3 --Accessories
and p.Color in ('Yellow','Blue','White')
order by SubCategoryName
,p.ProductID
/*
SubCategoryName ProductID ProductName Color
--------------- --------- ------------------------------- ------
Jerseys 881 Short-Sleeve Classic Jersey, S Yellow
Jerseys 882 Short-Sleeve Classic Jersey, M Yellow
Jerseys 883 Short-Sleeve Classic Jersey, L Yellow
Jerseys 884 Short-Sleeve Classic Jersey, XL Yellow
Socks 709 Mountain Bike Socks, M White
Socks 710 Mountain Bike Socks, L White
Socks 874 Racing Socks, M White
Socks 875 Racing Socks, L White
Vests 864 Classic Vest, S Blue
Vests 865 Classic Vest, M Blue
Vests 866 Classic Vest, L Blue
*/
You can replace that JOIN with a CROSS APPLY:

select SubCategoryName=s.Name 
,p.ProductID
,ProductName=p.Name
,p.Color
from Production.ProductSubCategory s
cross apply (select *
from Production.Product
where ProductSubcategoryID=s.ProductSubcategoryID) p
where s.ProductCategoryID=3 --Accessories
and p.Color in ('Yellow','Blue','White')
order by SubCategoryName
,p.ProductID
/*
SubCategoryName ProductID ProductName Color
--------------- --------- ------------------------------- ------
Jerseys 881 Short-Sleeve Classic Jersey, S Yellow
Jerseys 882 Short-Sleeve Classic Jersey, M Yellow
Jerseys 883 Short-Sleeve Classic Jersey, L Yellow
Jerseys 884 Short-Sleeve Classic Jersey, XL Yellow
Socks 709 Mountain Bike Socks, M White
Socks 710 Mountain Bike Socks, L White
Socks 874 Racing Socks, M White
Socks 875 Racing Socks, L White
Vests 864 Classic Vest, S Blue
Vests 865 Classic Vest, M Blue
Vests 866 Classic Vest, L Blue
*/
The query plans for both of those are exactly the same!

And LEFT JOINs can be replaced by OUTER APPLYs!

Today UNPIVOTs and JOINs… Tomorrow the world! Bwu hu hu ha ha ha ha haaaaaaa (Diabolical laughter).

Do it ALL!

For my final example, I’ll do ALL of the above (except for the JOIN replacement, which was just kind of a joke/trick anyway).

In doing the examples above, my query cache got populated with the text and plans of the queries I executed. We will look in the cache for the CROSS APPLY(TOP 3) query that was in the Execute SubQueries! section above, shred its query plan, looking for the operators, figure out their percentage cost, and list them in descending order of that cost. For Scans and Seeks and Joins, we will show the table, column and/or index used. And it will be presented in a vertical manner.

Note that the challenge here is finding the cost of each operator… it is not stored in the plan. Each operator has a Total Subtree Cost, but that is the cost of the operator itself PLUS the Subtree Costs of each of its immediate children operators. So for each operator, I had to find its children, total up their Subtree Costs and subtract that from the Subtree Cost of the operator to get the Individual Cost of the operator. This is done in the CROSS APPLY of the OperatorCosts CTE.

Hopefully the comments are self-explanatory:

with xmlnamespaces 
(
default 'http://schemas.microsoft.com/sqlserver/2004/07/showplan'
)
,
OperatorData as
(
select ParentNodeID
,NodeID
,OperatorDesc
,ScanSchema,ScanTable,ScanIndex
,LoopSchema,LoopTable,LoopColumn
,HashSchema,HashTable,HashColumn
,SubTreeCost
from sys.dm_exec_query_stats qs
cross apply
--Get the Query Text
sys.dm_exec_sql_text(qs.sql_handle) qt
cross apply
--Get the Query Plan
sys.dm_exec_query_plan(qs.plan_handle) qp
cross apply
--Get the RelOp nodes from the Plan
qp.query_plan.nodes('//RelOp') F_RelNodes(RelNode)
cross apply
--Pull out the various attributes from the RelOp Node
--And also extract the ParentNodeID of the operator
(select ParentNodeID=RelNode.value('(../../@NodeId)','int')
,NodeID=RelNode.value('(./@NodeId)','int')
,LogicalOp=RelNode.value('(./@LogicalOp)','varchar(50)')
,PhysicalOp=RelNode.value('(./@PhysicalOp)','varchar(50)')
,SubTreeCost=RelNode.value('(./@EstimatedTotalSubtreeCost)','float')
) F_OpInfo
cross apply
--Make a nice description out of the Operator
(select OperatorDesc=case
when LogicalOp=PhysicalOp
then PhysicalOp
else PhysicalOp+' ('+LogicalOp+')'
end
) F_OpDesc
outer apply
--Get child nodes having to do with a Scan/Seek
--Note that OUTER APPLY is used since there may not
--be any child nodes of this type
RelNode.nodes('(./IndexScan[1]/Object[1])')
F_ScanNode
(ScanNode)
outer apply
--And pull out their Table/Index information
(select ScanSchema=ScanNode.value('(./@Schema)','varchar(50)')
,ScanTable=ScanNode.value('(./@Table)','varchar(50)')
,ScanIndex=ScanNode.value('(./@Index)','varchar(100)')
) F_ScanInfo
outer apply
--Get child nodes having to do with Nested Loops
--Note that OUTER APPLY is used since there may not
--be any child nodes of this type
RelNode.nodes('(./NestedLoops[1]/OuterReferences[1]/ColumnReference[1])')
F_LoopNode
(LoopNode)
outer apply
--And pull out their Table/Column information
(select LoopSchema=LoopNode.value('(./@Schema)','varchar(50)')
,LoopTable=LoopNode.value('(./@Table)','varchar(50)')
,LoopColumn=LoopNode.value('(./@Column)','varchar(50)')
) F_LoopInfo
outer apply
--Get child nodes having to do with Hash Joins
--Note that OUTER APPLY is used since there may not
--be any child nodes of this type
RelNode.nodes('(./Hash[1]/HashKeysBuild[1]/ColumnReference[1])')
F_HashNode
(HashNode)
outer apply
--And pull out their Table/Column information
(select HashSchema=HashNode.value('(./@Schema)','varchar(50)')
,HashTable=HashNode.value('(./@Table)','varchar(50)')
,HashColumn=HashNode.value('(./@Column)','varchar(50)')
) F_HashInfo
where qt.text like '%select top 3 ProductID%'
and qt.text not like '%with xmlnamespaces%' --Exclude this query
)
,
OperatorCosts as
(
--Calculate the Individual Costs by subtracting each Operator's
--SubTreeCost minus its immediate children's SubTreeCosts
select NodeID
,OperatorDesc
,ScanSchema,ScanTable,ScanIndex
,LoopSchema,LoopTable,LoopColumn
,HashSchema,HashTable,HashColumn
,OperatorCost=convert(numeric(16,8),SubTreeCost-ChildrenSubTreeCost)
from OperatorData o
cross apply
--Calculate the sum of the SubTreeCosts of the immediate children
(select ChildrenSubTreeCost=isnull(sum(SubTreeCost),0)
from OperatorData
where ParentNodeID=o.NodeID) F_ChildCost
)
,
CostPercents as
(
--Calculate the CostPercent using a window function
select NodeID
,OperatorDesc
,ScanSchema,ScanTable,ScanIndex
,LoopSchema,LoopTable,LoopColumn
,HashSchema,HashTable,HashColumn
,CostPercent=convert(numeric(5,1),100*OperatorCost/sum(OperatorCost) over ())
from OperatorCosts
)
select Information
from CostPercents
cross apply
--UNPIVOT the information into a vertical presentation
(values ('NodeID '+convert(varchar(5),NodeID)
+' ('+convert(varchar(10),CostPercent)+'%):')
,(' '+OperatorDesc)
,(' Table: '+ScanSchema+'.'+ScanTable)
,(' Index: '+ScanIndex)
,(' Table: '+LoopSchema+'.'+LoopTable)
,(' Column: '+LoopColumn)
,(' Table: '+HashSchema+'.'+HashTable)
,(' Column: '+HashColumn)) P(Information)
where Information is not null --Eliminate NULL rows
order by CostPercent desc
/*
Information
----------------------------------------------------------------
NodeID 13 (19.6%):
Sort (TopN Sort)
NodeID 15 (19.6%):
Sort
NodeID 9 (18.2%):
Clustered Index Scan
Table: [Sales].[CustomerAddress]
Index: [PK_CustomerAddress_CustomerID_AddressID]
NodeID 11 (14.5%):
Clustered Index Seek
Table: [Sales].[Store]
Index: [PK_Store_CustomerID]
NodeID 20 (8.7%):
Clustered Index Seek
Table: [Sales].[SalesOrderDetail]
Index: [PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID]
NodeID 5 (5.8%):
Hash Match (Inner Join)
Table: [Person].[Address]
Column: AddressID
NodeID 12 (4.9%):
Clustered Index Seek
Table: [Sales].[Customer]
Index: [PK_Customer_CustomerID]
NodeID 18 (4.7%):
Index Seek
Table: [Sales].[SalesOrderHeader]
Index: [IX_SalesOrderHeader_CustomerID]
NodeID 30 (2.3%):
Clustered Index Seek
Table: [Production].[Product]
Index: [PK_Product_ProductID]
NodeID 7 (0.6%):
Index Seek
Table: [Person].[StateProvince]
Index: [AK_StateProvince_StateProvinceCode_CountryRegionCode]
NodeID 8 (0.6%):
Index Seek
Table: [Person].[Address]
Index: [IX_Address_StateProvinceID]
NodeID 6 (0.2%):
Nested Loops (Inner Join)
Table: [Person].[StateProvince]
Column: StateProvinceID
NodeID 3 (0.2%):
Nested Loops (Inner Join)
Table: [Sales].[CustomerAddress]
Column: CustomerID
NodeID 0 (0.0%):
Nested Loops (Inner Join)
Table: [Sales].[SalesOrderDetail]
Column: ProductID
NodeID 1 (0.0%):
Nested Loops (Inner Join)
Table: [Sales].[Customer]
Column: CustomerID
NodeID 2 (0.0%):
Nested Loops (Inner Join)
Table: [Sales].[CustomerAddress]
Column: CustomerID
NodeID 19 (0.0%):
Compute Scalar
NodeID 16 (0.0%):
Compute Scalar
NodeID 17 (0.0%):
Nested Loops (Inner Join)
Table: [Sales].[SalesOrderHeader]
Column: SalesOrderID
NodeID 14 (0.0%):
Stream Aggregate (Aggregate)
*/
Is that, like, waaaaay cool, or what?!

I hope I’ve convinced you how powerful the APPLY operator can be. I couldn’t live without it.

Thursday, February 25, 2010

Spotlight on UNPIVOT, Part 2

UNPIVOT Those Columns!In my last blog entry, we looked at the UNPIVOT command and how it works in unpivoting columns into rows.

We learned that T-SQL simply translates the UNPIVOT operator into an OUTER APPLY, coupled with a filter to get rid of NULL values. With that knowledge, we were able to unpivot data ourselves using an APPLY operator directly. This gave us more flexibility… We could include NULLs if we wanted to and we could very easily unpivot multiple sets of columns.

In this entry, we’ll go over some scenarios for when unpivoting data is helpful (more likely essential), and we’ll present many examples.

Before we begin, though, I have to say one thing. Most of these examples employ unpivoting because the tables are horribly designed and not normalized. I urge you, in designing your tables, to not emulate some of these table structures I am going to present.

Aggregations Over Columns

Let’s say you’re a school teacher and you have given 12 tests (exams) to your students over the course of the semester. Each test score is a percentage value from 0 to 100. And here’s the interesting part… you gave your students the option to skip any test of their choice during the semester, so one of the 12 tests might have a NULL value. Some students might take advantage of that… others (the teacher’s pets no doubt) will take every exam.

Here’s the (poorly-designed) table:

if object_id('tempdb..#Scores','U') is not null drop table #Scores
go
create
table #Scores
(
StudentID int primary key
,Test1 int
,Test2 int
,Test3 int
,Test4 int
,Test5 int
,Test6 int
,Test7 int
,Test8 int
,Test9 int
,Test10 int
,Test11 int
,Test12 int
)
go
insert
#Scores select 1, 96, 87,null, 100, 99, 91, 73, 90, 90, 91, 93, 92
union all select 2, 83, 84, 89, 79, 88,null, 93, 82, 85, 86, 88, 82
union all select 3, 100, 100, 98, 91, 100, 95, 96, 96, 99, 100, 98, 100

select * from #Scores
/*
StudentID Test1 Test2 Test3 Test4 Test5 Test6 Test7 Test8 Test9 Test10 Test11 Test12
--------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ------ ------ ------
1 96 87 NULL 100 99 91 73 90 90 91 93 92
2 83 84 89 79 88 NULL 93 82 85 86 88 82
3 100 100 98 91 100 95 96 96 99 100 98 100
*/

Your job is to get an average test score for each student. Well, that seems easy enough. We just add the scores together and divide by 12, right?

Oh wait…

There’s that darn NULL value we have to deal with… some students might have it, and some may not. And we don’t know which of the 12 test scores have the NULL value.

So we have to come up with a really awful query like this:

select StudentID
,AverageScore=( coalesce(Test1,0)+coalesce(Test2,0)+coalesce(Test3,0)
+coalesce(Test4,0)+coalesce(Test5,0)+coalesce(Test6,0)
+coalesce(Test7,0)+coalesce(Test8,0)+coalesce(Test9,0)
+coalesce(Test10,0)+coalesce(Test11,0)+coalesce(Test12,0) )
* 1.0 /
( case when Test1 is not null then 1 else 0 end
+case when Test2 is not null then 1 else 0 end
+case when Test3 is not null then 1 else 0 end
+case when Test4 is not null then 1 else 0 end
+case when Test5 is not null then 1 else 0 end
+case when Test6 is not null then 1 else 0 end
+case when Test7 is not null then 1 else 0 end
+case when Test8 is not null then 1 else 0 end
+case when Test9 is not null then 1 else 0 end
+case when Test10 is not null then 1 else 0 end
+case when Test11 is not null then 1 else 0 end
+case when Test12 is not null then 1 else 0 end )
from #Scores
/*
StudentID AverageScore
--------- ---------------
1 91.090909090909
2 85.363636363636
3 97.750000000000
*/
You have to admit… That is really ugly.

But it gets uglier…

Even worse, you have to find the minimum and maximum test score for each student. Take a look at this monster (try not to laugh… I dare you), which finds the maximum score:

select StudentID
,MaxScore=case
when Test1>=coalesce(Test2,0) and Test1>=coalesce(Test3,0)
and Test1>=coalesce(Test4,0) and Test1>=coalesce(Test5,0)
and Test1>=coalesce(Test6,0) and Test1>=coalesce(Test7,0)
and Test1>=coalesce(Test8,0) and Test1>=coalesce(Test9,0)
and Test1>=coalesce(Test10,0) and Test1>=coalesce(Test11,0)
and Test1>=coalesce(Test12,0) then Test1
when Test2>=coalesce(Test1,0) and Test2>=coalesce(Test3,0)
and Test2>=coalesce(Test4,0) and Test2>=coalesce(Test5,0)
and Test2>=coalesce(Test6,0) and Test2>=coalesce(Test7,0)
and Test2>=coalesce(Test8,0) and Test2>=coalesce(Test9,0)
and Test2>=coalesce(Test10,0) and Test2>=coalesce(Test11,0)
and Test2>=coalesce(Test12,0) then Test2
when Test3>=coalesce(Test1,0) and Test3>=coalesce(Test2,0)
and Test3>=coalesce(Test4,0) and Test3>=coalesce(Test5,0)
and Test3>=coalesce(Test6,0) and Test3>=coalesce(Test7,0)
and Test3>=coalesce(Test8,0) and Test3>=coalesce(Test9,0)
and Test3>=coalesce(Test10,0) and Test3>=coalesce(Test11,0)
and Test3>=coalesce(Test12,0) then Test3
when Test4>=coalesce(Test1,0) and Test4>=coalesce(Test2,0)
and Test4>=coalesce(Test3,0) and Test4>=coalesce(Test5,0)
and Test4>=coalesce(Test6,0) and Test4>=coalesce(Test7,0)
and Test4>=coalesce(Test8,0) and Test4>=coalesce(Test9,0)
and Test4>=coalesce(Test10,0) and Test4>=coalesce(Test11,0)
and Test4>=coalesce(Test12,0) then Test4
when Test5>=coalesce(Test1,0) and Test5>=coalesce(Test2,0)
and Test5>=coalesce(Test3,0) and Test5>=coalesce(Test4,0)
and Test5>=coalesce(Test6,0) and Test5>=coalesce(Test7,0)
and Test5>=coalesce(Test8,0) and Test5>=coalesce(Test9,0)
and Test5>=coalesce(Test10,0) and Test5>=coalesce(Test11,0)
and Test5>=coalesce(Test12,0) then Test5
when Test6>=coalesce(Test1,0) and Test6>=coalesce(Test2,0)
and Test6>=coalesce(Test3,0) and Test6>=coalesce(Test4,0)
and Test6>=coalesce(Test5,0) and Test6>=coalesce(Test7,0)
and Test6>=coalesce(Test8,0) and Test6>=coalesce(Test9,0)
and Test6>=coalesce(Test10,0) and Test6>=coalesce(Test11,0)
and Test6>=coalesce(Test12,0) then Test6
when Test7>=coalesce(Test1,0) and Test7>=coalesce(Test2,0)
and Test7>=coalesce(Test3,0) and Test7>=coalesce(Test4,0)
and Test7>=coalesce(Test5,0) and Test7>=coalesce(Test6,0)
and Test7>=coalesce(Test8,0) and Test7>=coalesce(Test9,0)
and Test7>=coalesce(Test10,0) and Test7>=coalesce(Test11,0)
and Test7>=coalesce(Test12,0) then Test7
when Test8>=coalesce(Test1,0) and Test8>=coalesce(Test2,0)
and Test8>=coalesce(Test3,0) and Test8>=coalesce(Test4,0)
and Test8>=coalesce(Test5,0) and Test8>=coalesce(Test6,0)
and Test8>=coalesce(Test7,0) and Test8>=coalesce(Test9,0)
and Test8>=coalesce(Test10,0) and Test8>=coalesce(Test11,0)
and Test8>=coalesce(Test12,0) then Test8
when Test9>=coalesce(Test1,0) and Test9>=coalesce(Test2,0)
and Test9>=coalesce(Test3,0) and Test9>=coalesce(Test4,0)
and Test9>=coalesce(Test5,0) and Test9>=coalesce(Test6,0)
and Test9>=coalesce(Test7,0) and Test9>=coalesce(Test8,0)
and Test9>=coalesce(Test10,0) and Test9>=coalesce(Test11,0)
and Test9>=coalesce(Test12,0) then Test9
when Test10>=coalesce(Test1,0) and Test10>=coalesce(Test2,0)
and Test10>=coalesce(Test3,0) and Test10>=coalesce(Test4,0)
and Test10>=coalesce(Test5,0) and Test10>=coalesce(Test6,0)
and Test10>=coalesce(Test7,0) and Test10>=coalesce(Test8,0)
and Test10>=coalesce(Test9,0) and Test10>=coalesce(Test11,0)
and Test10>=coalesce(Test12,0) then Test10
when Test11>=coalesce(Test1,0) and Test11>=coalesce(Test2,0)
and Test11>=coalesce(Test3,0) and Test11>=coalesce(Test4,0)
and Test11>=coalesce(Test5,0) and Test11>=coalesce(Test6,0)
and Test11>=coalesce(Test7,0) and Test11>=coalesce(Test8,0)
and Test11>=coalesce(Test9,0) and Test11>=coalesce(Test10,0)
and Test11>=coalesce(Test12,0) then Test11
when Test12>=coalesce(Test1,0) and Test12>=coalesce(Test2,0)
and Test12>=coalesce(Test3,0) and Test12>=coalesce(Test4,0)
and Test12>=coalesce(Test5,0) and Test12>=coalesce(Test6,0)
and Test12>=coalesce(Test7,0) and Test12>=coalesce(Test8,0)
and Test12>=coalesce(Test9,0) and Test12>=coalesce(Test10,0)
and Test12>=coalesce(Test11,0) then Test12
end
from
#Scores
/*
StudentID MaxScore
--------- --------
1 100
2 93
3 100
*/

(Sheesh… The things I do to demonstrate stupidity… I can’t believe I actually typed that whole thing out… I must be insane).

Well, the UNPIVOT clause can come to our rescue. It can unpivot the columns into rows and we can use T-SQL’s ability to do aggregations. So we can easily calculate the Average, Minimum, Maximum, and even the Standard Deviation of those test scores:

select StudentID
,AverageScore=avg(TestScore*1.0)
,MinScore=min(TestScore)
,MaxScore=max(TestScore)
,StdDev=stdev(TestScore)
from #Scores
unpivot (TestScore for Descript in (Test1,Test2,Test3,Test4,Test5,Test6
,Test7,Test8,Test9,Test10,Test11,Test12)) P
group by StudentID
/*
StudentID AverageScore MinScore MaxScore StdDev
--------- ------------ -------- -------- ----------------
1 91.090909 73 100 7.18963900977711
2 85.363636 79 93 3.95658254741963
3 97.750000 91 100 2.80016233295663
*/
Now isn’t that a lot easier by a factor of about a thousand?

Transpositions

This section is just for fun, really… however it might give you some ideas for other things.

Let’s take our table of test scores and turn the columns into rows and rows into columns… in other words, we’ll unpivot and then re-pivot.

;with UnpivotedScores as
(
select StudentColumn='Student'+convert(varchar(10),StudentID)
,TestScore
,TestID=convert(int,substring(TestColumnName,5,len(TestColumnName)))
from #Scores
unpivot (TestScore for TestColumnName in (Test1,Test2,Test3,Test4,Test5,Test6
,Test7,Test8,Test9,Test10,Test11,Test12)) U
)
select *
from UnpivotedScores
pivot (max(TestScore) for StudentColumn in (Student1,Student2,Student3)) P
/*
TestID Student1 Student2 Student3
------ -------- -------- --------
1 96 83 100
2 87 84 100
3 NULL 89 98
4 100 79 91
5 99 88 100
6 91 NULL 95
7 73 93 96
8 90 82 96
9 90 85 99
10 91 86 100
11 93 88 98
12 92 82 100
*/
So in this query, after unpivoting the data, I took the StudentID’s of 1 and 2 and 3 and created a column called StudentColumn containing the strings ‘Student1’, ‘Student2’, and ‘Student3’. I also created a column called TestID, extracting its value out of TestColumnName, a column created by the UNPIVOT. Finally, I PIVOTed the data across the three students.

Of course, I was only able to do this because I knew that I had only 3 studentsin my data. If I had more, then I would have to list more student columns in the PIVOT.

Here’s another fun and crazy exercise.

Create a table of 20,000 rows, with each row containing 8 random values from 1 to 999999.

if object_id('tempdb..#BunchesOfNums','U') is not null drop table #BunchesOfNums
go
create
table #BunchesOfNums
(
Num1 int
,Num2 int
,Num3 int
,Num4 int
,Num5 int
,Num6 int
,Num7 int
,Num8 int
)
go
set
nocount on
insert
#BunchesOfNums
select abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
,abs(checksum(newid()))%999999+1
go 20000

select * from #BunchesOfNums
/*
Num1 Num2 Num3 Num4 Num5 Num6 Num7 Num8
------ ------ ------ ------ ------ ------ ------ ------
430583 396264 325292 396579 294463 540881 827428 332591
819493 634199 705003 711098 387911 146887 931888 705560
967582 19768 391719 243203 716007 378208 471666 908523
206660 382012 753810 289947 725329 309939 268837 770277
651409 353534 54753 799079 317000 589131 77782 136318
4796 481041 870133 282436 243489 890761 307494 510633
682449 686105 311957 626728 781477 902128 974246 256178
933380 499938 653796 33348 536858 359010 875253 414892
... And so on for 20,000 rows
*/
Let’s reorganize all of this data, sorting it across the columns (i.e. horizontally):

;with NumbersUnpivot as
(
select Num
,NewRow=(row_number() over (order by Num)-1)/8+1
,NewCol=(row_number() over (order by Num)-1)%8+1
from #BunchesOfNums
unpivot (Num for NumColumn in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
)
select Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8
from (select Num
,NewRow
,ColName='Num'+convert(varchar(10),NewCol)
from NumbersUnpivot) A
pivot (max(Num) for ColName in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
order by NewRow
/*
Num1 Num2 Num3 Num4 Num5 Num6 Num7 Num8
------ ------ ------ ------ ------ ------ ------ ------
17 40 63 67 68 71 79 90
94 97 103 109 120 122 134 140
166 173 174 177 183 192 194 195
204 206 207 208 216 220 222 229
238 247 263 273 277 278 290 300
. . .
999772 999772 999773 999780 999781 999783 999783 999787
999794 999804 999806 999808 999808 999822 999830 999831
999832 999840 999841 999855 999868 999875 999887 999905
999905 999910 999930 999938 999940 999949 999963 999966
999966 999972 999986 999988 999991 999991 999992 999996
*/
Again, this was an UNPIVOT followed by a PIVOT. The original table was unpivoted, creating a derived table of 1 column (called Num) containing all 160,000 of our numbers. I created new columns called NewRow and NewCol, which were calculated using the ROW_NUMBER() function based on the sorting of the Num column. NewRow was calculated by dividing by 8 and NewCol was calculated by using doing a modulo 8 (i.e. the remainder in dividing by 8). Then I re-PIVOTed the numbers based on these new Row/Column positions.

We could also reorganize the numbers down across the rows (i.e. vertically). This time, we have to use 20,000 (the number of rows) for our divisor. But let’s make it more general and assume we don’t know up front how many rows there are. Let’s use the COUNT() aggregate to figure that number out and use that as the divisor:

;with NumbersUnpivot as
(
select Num
,NewRow=(row_number() over (order by Num)-1)%(count(Num) over () / 8)+1
,NewCol=(row_number() over (order by Num)-1)/(count(Num) over () / 8)+1
from #BunchesOfNums
unpivot (Num for NumColumn in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
)
select Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8
from (select Num
,NewRow
,ColName='Num'+convert(varchar(10),NewCol)
from NumbersUnpivot) A
pivot (max(Num) for ColName in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
order by NewRow
/*
Num1 Num2 Num3 Num4 Num5 Num6 Num7 Num8
------ ------ ------ ------ ------ ------ ------ ------
17 125498 250005 374416 499463 623675 749265 875013
40 125501 250008 374421 499471 623680 749268 875025
63 125512 250010 374429 499472 623687 749269 875029
67 125517 250018 374431 499480 623691 749278 875031
68 125533 250028 374433 499485 623691 749289 875033
. . .
125479 249986 374402 499427 623659 749216 874991 999988
125480 249987 374409 499433 623661 749242 874994 999991
125484 249999 374411 499446 623662 749243 874997 999991
125488 250001 374413 499453 623663 749251 875003 999992
125493 250003 374416 499455 623675 749254 875004 999996
*/
This certainly works, but the query is not really that efficient. Because of the COUNT() aggregate and the way it’s used, the query employs a whopping 384159 reads and 738 writes, because it needs to spool information to tempdb to do its work.

The following is more efficient and only employs 650 reads and 0 writes… an enormous difference!

;with NumbersUnpivot as
(
select Num
,NewRow=(row_number() over (order by Num)-1)%NumRows+1
,NewCol=(row_number() over (order by Num)-1)/NumRows+1
from #BunchesOfNums
cross join (select NumRows=count(*) from #BunchesOfNums) C
unpivot (Num for NumColumn in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
)
select Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8
from (select Num
,NewRow
,ColName='Num'+convert(varchar(10),NewCol)
from NumbersUnpivot) A
pivot (max(Num) for ColName in (Num1,Num2,Num3,Num4,Num5,Num6,Num7,Num8)) P
order by NewRow
/*
Num1 Num2 Num3 Num4 Num5 Num6 Num7 Num8
------ ------ ------ ------ ------ ------ ------ ------
17 125498 250005 374416 499463 623675 749265 875013
40 125501 250008 374421 499471 623680 749268 875025
63 125512 250010 374429 499472 623687 749269 875029
67 125517 250018 374431 499480 623691 749278 875031
68 125533 250028 374433 499485 623691 749289 875033
. . .
125479 249986 374402 499427 623659 749216 874991 999988
125480 249987 374409 499433 623661 749242 874994 999991
125484 249999 374411 499446 623662 749243 874997 999991
125488 250001 374413 499453 623663 749251 875003 999992
125493 250003 374416 499455 623675 749254 875004 999996
*/
As you can see, I calculated the number of rows up front (into a column called NumRows) and I CROSS JOINed that with the table itself so that I could use it as the divisor in calculating NewRow and NewCol.

I encourage you to examine the plans of the two queries above to see the differences.

Column Data Repair

How often have you seen a scenario like this?: A table has 3 or 4 or 5 or more columns that contain the same kind of information. An address is a good example. A customer might have 1 or 2 or 3 lines of address and there are 3 columns for that information to be stored.

Another example is credit card information. A table has 5 columns to store up to 5 possible credit card numbers that we have on file for a person.

Again, this is very bad design… the credit card information should be put into a child table of its own, with one row for each credit card. Then many credit cards could be recorded and old ones can be deleted as much as desired.

But in the bad table design with 5 credit card columns, people will just erase outdated credit card numbers over time and not reorganize them and you end up with lots of possible “holes” as in the following example:

use TempDB
go
if
object_id('CredCardInfo','U') is not null drop table CredCardInfo
go
create
table CredCardInfo
(
ID int primary key
,Card1 varchar(16)
,Card2 varchar(16)
,Card3 varchar(16)
,Card4 varchar(16)
,Card5 varchar(16)
)
go

insert CredCardInfo values
(1,'','4898198371982987','5379110787298721','','')
,(2,'6798172988177792','','','5798116090379113','4798116444218977')
,(3,'','','','','4980926667212798')
,(4,'','','6877172983719871','','4079267155432178')
,(5,'4076372911736532','6888327238009837','5368200841237897','','')

select * from CredCardInfo
/*
ID Card1 Card2 Card3 Card4 Card5
-- ---------------- ---------------- ---------------- ---------------- ----------------
1 4898198371982987 5379110787298721
2 6798172988177792 5798116090379113 4798116444218977
3 4980926667212798
4 6877172983719871 4079267155432178
5 4076372911736532 6888327238009837 5368200841237897
*/
You can see that the first 4 rows have empty columns where credit card numbers used to be. We want to repair this information so that all the credit card numbers are like the 5th row, where all the card numbers are in the 1st and 2nd and 3rd slots and so on with no “holes”.

So here is a stored procedure that does just that:

use TempDb
go
if
object_id('RepairCardInfo','P') is not null drop procedure RepairCardInfo
go
create
procedure RepairCardInfo
as
with
CardsUnpivot as
(
select ID
,Card
,OrigColName
,NewColName='Card'
+convert(char(1),row_number() over (partition by ID
order by OrigColName))
from CredCardInfo
unpivot (Card for OrigColName in (Card1,Card2,Card3,Card4,Card5)) P
where Card<>''
)
,
IDsThatNeedChange as
(
select ID
from CardsUnpivot
where OrigColName<>NewColName
)
,
NewCardValues as
(
select ID,Card1,Card2,Card3,Card4,Card5
from (select ID,Card,NewColName
from CardsUnpivot
where ID in (select ID from IDsThatNeedChange)) A
pivot (max(Card) for NewColName in (Card1,Card2,Card3,Card4,Card5)) P
)
update C
set Card1=coalesce(n.Card1,'')
,Card2=coalesce(n.Card2,'')
,Card3=coalesce(n.Card3,'')
,Card4=coalesce(n.Card4,'')
,Card5=coalesce(n.Card5,'')
from CredCardInfo C
join NewCardValues N on C.ID=N.ID
Like other examples we’ve seen in this article, we UNPIVOT the data and then re-PIVOT it.

Take a closer look at the CardsUnpivot CTE… here’s what it produces:

select ID,Card,OrigColName
,NewColName='Card'
+convert(char(1),row_number() over (partition by ID
order by OrigColName))
from CredCardInfo
unpivot (Card for OrigColName in (Card1,Card2,Card3,Card4,Card5)) P
where Card<>''
/*
ID Card OrigColName NewColName
-- ---------------- ----------- ----------
1 4898198371982987 Card2 Card1
1 5379110787298721 Card3 Card2
2 6798172988177792 Card1 Card1
2 5798116090379113 Card4 Card2
2 4798116444218977 Card5 Card3
3 4980926667212798 Card5 Card1
4 6877172983719871 Card3 Card1
4 4079267155432178 Card5 Card2
5 4076372911736532 Card1 Card1
5 6888327238009837 Card2 Card2
5 5368200841237897 Card3 Card3
*/
Notice that after the UNPIVOTing, we create a NewColName, based on the ROW_NUMBER() ordered by OrigColName (within each ID). And that NewColName is sequenced as ‘Card1’, ‘Card2’, etc for each ID.

The next CTE, IDdsThatNeedChange, finds all IDs where OrigColName and NewColName are different. Note that ID Number 5 has them all matching because its data is already in the format we want, with all the numbers in slot 1 and 2 and 3.

The final NewCardValues CTE rePIVOTs the data back into the 5 credit card columns… but only for those IDs that need the change.

And that CTE is used to UPDATE the original table… updating only those rows that need change.

If we execute this procedure, and then look at the data again, we see that all is repaired:

set nocount off
exec
RepairCardInfo
/* (4 row(s) affected) */

select
* from CredCardInfo
/*
ID Card1 Card2 Card3 Card4 Card5
-- ---------------- ---------------- ---------------- ---------------- ----------------
1 4898198371982987 5379110787298721
2 6798172988177792 5798116090379113 4798116444218977
3 4980926667212798
4 6877172983719871 4079267155432178
5 4076372911736532 6888327238009837 5368200841237897
*/

Note that only 4 of 5 rows were actually UPDATEd. If we run the procedure again, nothing happens, because all of our data is already repaired:

set nocount off
exec
RepairCardInfo
/* (0 row(s) affected) */
Vertical Presentations of Data

Sometimes it’s nice to see data presented in a vertical format rather than in columns.

In AdventureWorks, there’s a view called Sales.vStoreWithDemographics, which JOINs many tables to come up with Contact information, Name and Address information, and Demographic information for Stores in the database.

Let’s say we want to create an inline table-valued function which accepts a @CustomerID parameter to present data for the Main Office of a Customer Store. Since there can potentially be more than one contact for a store, we will use ROW_NUMBER() to only look at the first instance.

If we attempt to use the UNPIVOT operator to do this, we run into trouble:

if object_id('VerticalMainOfficeData','IF') is not null 
drop function VerticalMainOfficeData
go
create
function VerticalMainOfficeData
(
@CustomerID int
)
returns table
as
return
with
BaseData as
(
select SeqNo=row_number() over (order by ContactType)
,Name,AddressLine1,AddressLine2,City,StateProvinceName
,PostalCode,CountryRegionName
,YearOpened,NumberEmployees,Specialty
,SquareFeet,Brands,AnnualSales
from AdventureWorks.Sales.vStoreWithDemographics
where CustomerID=@CustomerID
and AddressType='Main Office'
)
select Property,Value
from BaseData
unpivot (Value for Property in (Name
,AddressLine1
,AddressLine2
,City
,StateProvinceName
,PostalCode
,CountryRegionName
,YearOpened
,NumberEmployees
,Specialty
,SquareFeet
,Brands
,AnnualSales)) P
where SeqNo=1
/*
Msg 8167, Level 16, State 1, Procedure VerticalMainOfficeData, Line 22
The type of column "AddressLine1" conflicts with the type of other columns
specified in the UNPIVOT list.
*/
That’s yet another problem with the UNPIVOT operator. Every single column name specified in the list must be exactly the same datatype and have exactly the same length.

But, as we now know, we can just use an APPLY operator instead of UNPIVOT, and we get truckloads of flexibility, as you can see by this function definition, which returns our information in a nice fancy way:

if object_id('VerticalMainOfficeData','IF') is not null 
drop function VerticalMainOfficeData
go
create
function VerticalMainOfficeData
(
@CustomerID int
)
returns table
as
return
with
BaseData as
(
select SeqNo=row_number() over (order by ContactType)
,Name,AddressLine1,AddressLine2,City,StateProvinceName
,PostalCode,CountryRegionName
,YearOpened,NumberEmployees,Specialty
,SquareFeet,Brands,AnnualSales
from AdventureWorks.Sales.vStoreWithDemographics
where CustomerID=@CustomerID
and AddressType='Main Office'
)
select Property,Value
from BaseData
cross apply
(values ('NAME AND ADDRESS:','')
,(' Name',Name)
,(' Address',AddressLine1)
,(' ',AddressLine2)
,(' City',City)
,(' State/Province',StateProvinceName)
,(' Postal Code',PostalCode)
,(' Country/Region',CountryRegionName)
,('','')
,('DEMOGRAPHIC INFO:','')
,(' Year Opened',str(YearOpened,4))
,(' Number of Employees',convert(varchar(10),NumberEmployees))
,(' Specialty',Specialty)
,(' Square Feet',convert(varchar(10),SquareFeet))
,(' Brands',Brands)
,(' Annual Sales','$'+convert(varchar(20),AnnualSales,1))) P(Property,Value)
where SeqNo=1
and Value is not null
Since we’re effectively UNIONing the data in the APPLY, we of course had to CONVERT any numerical data to a VARCHAR value, but we didn’t have to care about the various lengths of the various VARCHAR values. Here are a couple of examples of the output:

select * from dbo.VerticalMainOfficeData(34)
/*
Property Value
--------------------- -----------------------------
NAME AND ADDRESS:
Name Cycles Wholesaler & Mfg.
Address Science Park South, Birchwood
Stanford House
City Warrington
State/Province England
Postal Code WA3 7BH
Country/Region United Kingdom

DEMOGRAPHIC INFO:
Year Opened 1999
Number of Employees 15
Specialty Touring
Square Feet 21000
Brands 4+
Annual Sales $800,000.00
*/

select * from dbo.VerticalMainOfficeData(388)
/*
Property Value
--------------------- ---------------------------
NAME AND ADDRESS:
Name Road Way Sales and Rental
Address 2900 - 25055 Dunning Street
City Vancouver
State/Province British Columbia
Postal Code V7L 4J4
Country/Region Canada

DEMOGRAPHIC INFO:
Year Opened 1979
Number of Employees 6
Specialty Mountain
Square Feet 10000
Brands 3
Annual Sales $300,000.00
*/

Notice in the first example how the second address line is displayed… with a blank string in the Property column… so it looks like it’s just a second line of the address like it should be. The second example had no second line of address (it was NULL in the original table)… it was eliminated by the WHERE clause that eliminated any NULL values.

Finally, I decided to go wild and create a more extensive function that included contact information (up to 2 contacts… no Main Office of any Store in AdventureWorks has more than 2 contacts anyway).

Here is the function I created to do that.

if object_id('VerticalMainOfficeDataWithContacts','IF') is not null 
drop function VerticalMainOfficeDataWithContacts
go
create
function VerticalMainOfficeDataWithContacts
(
@CustomerID int
)
returns table
as
return
with
BaseData as
(
select SeqNo=row_number() over (order by ContactType)
,Name,AddressLine1,AddressLine2,City,StateProvinceName
,PostalCode,CountryRegionName
,YearOpened,NumberEmployees,Specialty
,SquareFeet,Brands,AnnualSales
,ContactType,Title,FirstName,MiddleName
,LastName,Suffix,Phone,EmailAddress
from AdventureWorks.Sales.vStoreWithDemographics
where CustomerID=@CustomerID
and AddressType='Main Office'
)
,
ContactInfo as
(
select SeqNo
,ContactType
,ContactName=coalesce(Title+' ','')
+FirstName+' '
+coalesce(MiddleName+' ','')
+LastName
+coalesce(' '+Suffix,'')
,Phone
,EmailAddress
from BaseData
)
,
ContactUnpivot as
(
select Property=Property+str(SeqNo,1)
,Value
from ContactInfo
cross apply (values ('ContactType',ContactType)
,('ContactName',ContactName)
,('Phone',Phone)
,('EmailAddress',EmailAddress)) C(Property,Value)
)
,
ContactColumns as
(
select *
from ContactUnpivot
pivot (max(Value) for Property in (ContactType1,ContactName1,Phone1,EmailAddress1
,ContactType2,ContactName2,Phone2,EmailAddress2)) P
)
,
UnpivotInput as
(
select *
from BaseData
cross join ContactColumns
where SeqNo=1
)
select Property,Value
from UnpivotInput
cross apply
(values ('NAME AND ADDRESS:','')
,(' Name',Name)
,(' Address',AddressLine1)
,(' ',AddressLine2)
,(' City',City)
,(' State/Province',StateProvinceName)
,(' Postal Code',PostalCode)
,(' Country/Region',CountryRegionName)
,('','')
,('DEMOGRAPHIC INFO:','')
,(' Year Opened',str(YearOpened,4))
,(' Number of Employees',convert(varchar(10),NumberEmployees))
,(' Specialty',Specialty)
,(' Square Feet',convert(varchar(10),SquareFeet))
,(' Brands',Brands)
,(' Annual Sales','$'+convert(varchar(20),AnnualSales,1))
,('','')
,('CONTACT'+case when ContactType2 is not null then ' #1' else '' end+':','')
,(' Type',ContactType1)
,(' Name',ContactName1)
,(' Phone',Phone1)
,(' Email',EmailAddress1)
,('',case when ContactType2 is not null then '' end)
,('CONTACT #2:',case when ContactType2 is not null then '' end)
,(' Type',ContactType2)
,(' Name',ContactName2)
,(' Phone',Phone2)
,(' Email',EmailAddress2)) P(Property,Value)
where Value is not null
I’ll leave it up to you to examine how it works as an exercise. In a nutshell, it unpivots and re-pivots the contact information into a single-row derived table, and then that’s JOINed back to the original base data and then the whole thing is unpivoted.

Here are a couple examples of the output. Note how the first one has only 1 contact (with the heading ‘CONTACT’) and the second one has 2 contacts (with the headings ‘CONTACT #1’and ‘CONTACT #2’:

select * from dbo.VerticalMainOfficeDataWithContacts(34)
/*
Property Value
--------------------- -----------------------------
NAME AND ADDRESS:
Name Cycles Wholesaler & Mfg.
Address Science Park South, Birchwood
Stanford House
City Warrington
State/Province England
Postal Code WA3 7BH
Country/Region United Kingdom

DEMOGRAPHIC INFO:
Year Opened 1999
Number of Employees 15
Specialty Touring
Square Feet 21000
Brands 4+
Annual Sales $800,000.00

CONTACT:
Type Owner
Name Ms. Barbara J. German
Phone 1 (11) 500 555-0181
Email barbara4@adventure-works.com
*/


select * from dbo.VerticalMainOfficeDataWithContacts(50)
/*
Property Value
--------------------- ------------------------
NAME AND ADDRESS:
Name Hometown Riding Supplies
Address Bundesallee 9571
Rechnungsstelle - C 035
City Berlin
State/Province Brandenburg
Postal Code 14197
Country/Region Germany

DEMOGRAPHIC INFO:
Year Opened 1991
Number of Employees 46
Specialty Mountain
Square Feet 38000
Brands 4+
Annual Sales $1,500,000.00

CONTACT #1:
Type Owner
Name Mr. William Vong
Phone 809-555-0100
Email william6@adventure-works.com

CONTACT #2:
Type Purchasing Manager
Name Mr. Greg Harrison
Phone 1 (11) 500 555-0112
Email greg2@adventure-works.com
*/
I hope this article gave you some interesting examples of using UNPIVOT (and yes, PIVOT also). I also hope it stimulated your imagination on creative things you can do with T-SQL.