Category: Uncategorized

Dynamic Data CenterGuidance for Hosting Providers

Posted on by escapebusinesssolutions in Uncategorized

 

Dynamic Data Center

Guidance for Hosting Providers

 

 

 

 

 

 

© 2009 Microsoft Corporation. All rights reserved. The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication and is subject to change at any time without notice to you. This document and its contents are provided AS IS without warranty of any kind, and should not be interpreted as an offer or commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Microsoft, Active Directory, Hyper-V, Silverlight, SQL Server, Windows, Windows Powershell, and Windows Server are trademarks of the Microsoft group of companies.

All other trademarks are property of their respective owners.

The descriptions of other companies’ products in this document, if any, are provided only as a convenience to you. Any such references should not be considered an endorsement or support by Microsoft. Microsoft cannot guarantee their accuracy, and the products may change over time. Also, the descriptions are intended as brief highlights to aid understanding, rather than as thorough coverage. For authoritative descriptions of these products, please consult their respective manufacturers.

Microsoft will not knowingly provide advice that conflicts with local, regional, or international laws; however, it is your responsibility to confirm that your implementation of any advice is in accordance with all applicable laws.

 

 

Delivering Business-Critical Solutions with SharePoint 2010

Posted on by escapebusinesssolutions in Uncategorized


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Disclaimer

The information contained in this document represents the current plans of Microsoft Corporation on the issues presented at the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Schedules and features contained in this document are subject to change.

Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the expressed written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give any license or rights to these patents, trademarks, copyrights, or other intellectual property.

© 2011 Microsoft Corporation. All rights reserved.

Microsoft, the Microsoft logo, Access, Excel, Outlook, SharePoint, Visio, and other product names are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

All other trademarks are property of their respective owners.

 

 

Table of Contents

Who Should Read This White Paper?    3

Challenge: Siloed Information and Processes Limit Business Performance and
Consume IT Resources    4

Surface LOB Data in SharePoint 2010    5

Business Connectivity Services    5

Implement, Extend, and Improve Business Processes by Building Solutions on
SharePoint 2010    6

Business Solutions in SharePoint 2010    6

IT-Managed Solutions    6

Advanced User and Information Worker Solutions    7

Increase Productivity with Enterprise Search and Business Intelligence    9

Use the SharePoint Platform to Speed ROI and Decrease Business Risk    10

Conclusion    11

Resources    12

 

Who Should Read This White Paper?

Business users at every level of your organization should have access to important data and be connected to processes that enable them to support operations. However, critical business data often is stored in disparate systems and ad hoc processes block efficiencies. Users call on IT to assist them in reaching and reconciling this business data, which can divert important IT resources away from strategic work that positions IT as a business partner, rather than a cost center.

This white paper is intended for Chief Information Officers, Chief Technical Officers, infrastructure managers and information system managers who want to deliver business-critical solutions while driving business value and reducing business risk. These solutions can be implemented across the organization and empower business users to get more value out of line-of-business (LOB) systems, thus extending the reach of important business data and improving business processes. This can be accomplished by connecting LOB applications to Microsoft® SharePoint® 2010 to organize and facilitate broad access to previously siloed information.

This white paper explains how to:

  • Increase access to critical backend business data by surfacing it in SharePoint 2010.
  • Enhance the effectiveness of business processes by building on SharePoint 2010 and using its platform capabilities.
  • Deliver fast return on investment (ROI) by lowering business risk, decreasing training costs and enhancing compliance.

Challenge: Siloed Information and Processes Limit Business Performance and Consume IT Resources

Most business-based IT applications are built for and deployed within vertical business functions, such as product lifecycle management (PLM) for design and engineering, customer relationship management (CRM) for sales and service, and enterprise resource planning (ERP) for finance and human resources.

Your organization is familiar with—or has even deployed—a number of these solutions, like PeopleSoft, JDEdwards, Oracle Financials or SAP. These applications support structured decision making and processes within the business function; nevertheless, silos exist between the applications and organizational functions and processes. This environment can make it difficult for business users to drive collaborative decision making that crosses business units and spans multiple business functions.

Users who need access to the data that resides in these systems make frequent requests to IT to extract the information and organize it. These requests are compounded by users’ demands for anywhere, anytime access on a variety of devices, like laptops, smartphones, and tablets. This leaves IT to solve configuration, security, and compliance concerns, while stretching an already slim IT budget.

These issues become less critical when data is surfaced through a unified platform. SharePoint 2010 can connect a broad range of users to business data that currently resides in siloed systems and is accessible only to specialized users and IT professionals. It also can empower every user—from information workers to power users and professional developers—to build solutions based on this business data, solutions that streamline processes and result in better, faster decisions by the organization.

Surface LOB Data in SharePoint 2010

SharePoint 2010 offers a variety of ways for your organization to surface information buried in siloed LOB systems, so you can quickly begin to improve processes by building solutions that span departmental and even cross-organizational boundaries. For the purpose of this paper, we will highlight Business Connectivity Services (BCS), the SharePoint 2010 technology that provides the core line-of-business connectivity capabilities. However, there are other options for connecting your LOB applications to SharePoint, including Web Services using Windows Communication Foundation (WCF), and more.

Business Connectivity Services

Business Connectivity Services in SharePoint 2010 enables connectivity to external data sources, such as databases and LOB systems (Figure 1). When your LOB systems are connected to SharePoint, users can interact with business data from within the familiar Microsoft Office and SharePoint user interface, so they do not need to learn many complex applications to get their jobs done. This also enables IT to use a unified platform across a range of LOB systems to simplify administration and support.


Figure 1: BCS architecture diagram

Specifically, Business Connectivity Services can help your organization:

  • Bring external data into SharePoint and Office, helping users to read, edit, and write LOB data in familiar tools such as Microsoft Outlook®, Excel®, and Word. Example: An organization brings inventory data from its ERP system into SharePoint to give sales the ability to update the information in real time based on order changes.
  • Address users’ collaboration needs by extending SharePoint capabilities and the Office user experience to include business data and processes. Example: A manufacturing plant foreman searches SharePoint to identify his peers in other plants so that he can reach out and discuss a machinery issue.
  • Create fast, incremental user-driven solutions like workflows and templates that position IT as a strategic asset within the organization. Example: The service department leverages a pre-built workflow for problem resolution that reduces response time and increases customer satisfaction.

Implement, Extend, and Improve Business Processes by Building Solutions on SharePoint 2010

SharePoint 2010 allows IT to focus on executing high-priority projects that deliver strategic business advantages, while maintaining a stable infrastructure. This is because SharePoint provides the right environment for IT to meet business demands by enabling business-critical processes within and across organizational boundaries. In turn, users can become more empowered, translating into increased efficiencies for IT and improved productivity for your organization.

Certainly, user empowerment is key to keeping your organization agile and productive. You can increase organizational agility by using the SharePoint platform to help users access, visualize, and consume the business-critical data currently locked in LOB applications. Plus, IT can maintain control through centralized management and security tools, such as data storage management, backup, versioning, and records management.

It is important to remember that to achieve this level of business benefit, your organization must have a deployment plan that prioritizes business needs. By focusing on business needs, you can help to ensure that SharePoint 2010 is broadly adopted across the organization, which ultimately means that the right information can be delivered to the right people at the right time.

Business Solutions in SharePoint 2010

After connecting external systems to SharePoint 2010, you can begin to build solutions to improve the processes that are crucial to organizational success. SharePoint 2010 offers solution models to help your organization develop, improve, and extend business processes:

  • IT-managed solutions
  • Advanced user and information worker solutions

IT-Managed Solutions

When implementing or extending business-critical processes based on business data from operational systems, IT leads the way more often than not while ensuring that the right methodologies, security models and governance approach are being applied. Nevertheless, these IT-led project tend to be both complex and costly, making it hard for organizations to decide to take-on these challenges.

With SharePoint 2010, IT does not need to develop web applications from the ground up. Instead, they can use platform services to quickly create robust custom solutions. The typical application development lifecycle is a time-consuming and costly endeavor. Each application needs its own security model, workflow engine, repository for storing information, and more. SharePoint provides all of these capabilities out of the box. By building applications on top of SharePoint, your organization can get started faster and deliver value to the business more quickly.

The SharePoint partner ecosystem gives organizations access to an extensive range of solutions from independent software vendors (ISVs) as an alternative to writing custom code, thereby providing sophisticated solutions in a prebuilt package. In addition to the partner ecosystem, a community of systems integrators (SIs) can help users to plan and deploy these types of solutions.

Advanced User and Information Worker Solutions

Advanced users have a deeper understanding of the tools and technologies that IT professionals often use to develop and deploy solutions (for example, Microsoft SharePoint Designer, Microsoft Access® Services, and Microsoft Visio® Services). SharePoint integrates with design tools to give advanced users greater flexibility in building solutions, while still promoting quality assurance and allowing IT professionals to maintain control over finished products. Examples of advanced user solutions include creating a business connection from SharePoint to an LOB system and creating custom workflows to automate tedious business processes (Figure 2). To support the creation of these types of solutions, IT needs to identify advanced users and train them on self-service best practices, and establish governance to define how advanced user solutions are built and deployed.


Figure 2: Example of a workflow for an advanced user solution

 

Scenario: Advanced User Solution

Frank Zhang, a customer service representative for an engineer-to-order company, uses a SharePoint solution for order entry and change management. Business Connectivity Services provides integration with customer data, product catalogs, engineering specifications, on-hand stock availability, and pricing and discount information dynamically linked to various LOB sources, including master data repositories, engineering, and ERP.

Before implementing this process through SharePoint, Frank needed to access and assemble data from various systems to complete the order placement process. This often required exchanging multiple emails and Excel workbooks among various departments, such as Engineering, Manufacturing, and Finance. With simplified and integrated access to all needed data in his SharePoint-based workplace, Frank now typically can manage order completion on his own, with fewer errors and delays. When needed, automated workflows take care of the process across various cross-functional teams, with shared information maintained in a single workspace (such as through Excel Services), helping to eliminate the need for inefficient ad hoc communications.

Information workers can take advantage of out-of-the-box capabilities in SharePoint 2010 that increase productivity. For instance, they can leverage workflows and customizable views of their critical business data created by IT or advanced users. SharePoint 2010 also can create forms automatically based on templates within SharePoint and other Microsoft Office applications (Figure 3).


Figure 3: Example of a form for an information worker solution

Further, information workers can create lists and document libraries that allow them to collect information, collaborate on documents, and share information easily. IT is tasked with publishing templates for the most common solutions and with teaching users best practices for creating lists and collecting information. For additional information about the advantages of user-created solutions, refer to the resources at the end of this paper.

Scenario: Information Worker Solution

Nina Vietzen, a customer service representative, used Microsoft SharePoint and Word to create her own solution for tracking customer inquiries and associated documentation. Custom templates in Word allowed Nina to start with a standard document, while a centralized SharePoint framework provides document versioning, document metadata, and backup and restore.

Before Nina implemented this process through SharePoint, users had to create new Word documents for each customer inquiry and store them in a file share without version control. Now, versioning, metadata and search, and central backups provide Nina and her colleagues with a time-saving solution that keeps documents safe.

Increase Productivity with Enterprise Search and Business Intelligence

SharePoint 2010 includes multiple capabilities with built-in security and manageability that IT can deploy to help improve business user productivity based on accessing and visualizing the business data. Two of these key capabilities are Search and Insights.

SharePoint Search enables cross-platform search to help business users consume and manage important business data. SharePoint 2010 Search provides an interactive, visual search experience. Visual cues help people find information quickly, while refiners let them drill into the results and discover insights.

Example: An account manager receives a customer request to adjust a custom order. Before responding to her customer, she must determine whether any of her organization’s warehouses have the items in stock to amend the order. Her ERP system is connected to SharePoint 2010, so she opens up her team portal, searches for the part, and finds that it is available in two warehouses.

SharePoint 2010 Insights provides interactive dashboards and scorecards that can help people to define and measure success: key metrics can be matched to specific strategies and then shared, tracked, and discussed. Users can create meaningful visualizations that convey the right information the first time, aggregating content from multiple sources and displaying it in a web browser in an understandable and collaborate environment. Moreover, rich interactivity allows users to analyze up-to-the-minute information and work with data quickly and easily to identify key opportunities and trends. Figure 4 (next page) shows a user’s dashboard in SharePoint 2010 Insights.


Figure 4: Dashboard in SharePoint 2010 Insights

Use the SharePoint Platform to Speed ROI and Decrease Business Risk

Using SharePoint 2010 to surface business data from your LOB systems and build solutions can increase the ROI of your legacy systems, speed solutions’ time-to-market, and empower users to help themselves—all of which frees IT resources to focus on more strategic initiatives.

Business units can reduce training costs because SharePoint 2010 offers the familiar Microsoft Office experience that enables people to quickly and easily adopt SharePoint (as opposed to training users on a variety of more complex LOB applications).

SharePoint can speed time-to-market of otherwise time-consuming and resource-intensive solutions to streamline business-critical processes. In addition, powerful Search and BI capabilities provide self-service functionality, which boosts productivity, reduces costs, and increases user satisfaction.

Finally, SharePoint can help to reduce your organization’s overall risk by increasing the visibility of business-critical data. The ability to access accurate, real-time business data has a major impact on your organization. In his 2009 white paper, “Business Intelligence: A Guide for Midsize Companies,” MAS Strategies’ Founder and Principal Analyst Michael Schiff said, “All employees have the responsibility to make the best decisions possible, based upon the data available to them at that time. If their ability to analyze this data and transform it into useful information is improved, the overall quality of their decisions can be improved as well.”

When you surface all relevant data to the people who need it when they need it, you enable them to make better decisions faster. This can reduce mistakes that result from misinformation and decrease your organization’s business risk.

SharePoint also reduces risk by enhancing security, privacy, and compliance through a flexible authentication model. This authentication model can help your organization to maximize its SharePoint 2010 deployment while maintaining highly secure control over corporate assets to increase compliance.

Conclusion

This white paper has discussed extending the reach of your business-critical data across departmental and organizational boundaries to improve business-critical solutions. It also has shown the benefits of surfacing and visualizing this data in SharePoint 2010:

  • Surface LOB Data in SharePoint 2010: Identify business-critical data and the users who need it, and extend the reach of your data by connecting SharePoint to your LOB applications. SharePoint 2010 provides many ways to achieve this state, faster and easier than in previous versions and without complex, expensive custom development.
  • Implement, Extend, and Improve Business Processes: Find and visualize the information you need in SharePoint 2010. Take advantage of out-of-the-box platform capabilities like collaboration, social computing, and content management to enable the right people to access the right information at the right time. IT can design and administer solutions quickly so that users can build their own templates and workflows to connect business data to their processes.
  • Gain Additional Productivity with SharePoint 2010: SharePoint provides several capabilities, including Search and Insights, that can help organizations to improve workforce productivity and visualize their business data in real-time. These capabilities have built-in security and manageability to help ensure safe and easy use.
  • Speed ROI and Decrease Risk: Connecting SharePoint 2010 to your LOB applications can increase the ROI of these systems and decrease business risk by surfacing important data across the organization to users who need it, when they need it. Plus, out-of-the-box capabilities in SharePoint can speed the time-to-market of previously labor-intensive solutions. SharePoint also can reduce IT administrator and end user training costs by enabling users to access information through a familiar interface.

 

Resources

Learn more about the SharePoint capabilities outlined in this white paper by visiting the following:

SharePoint Deployment on Windows Azure Virtual Machines

Posted on by escapebusinesssolutions in Uncategorized

DISCLAIMER

This document is provided “as-is.” Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. 

Some examples are for illustration only and are fictitious. No real association is intended or inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

2012 Microsoft Corporation.  All rights reserved. 

 

Table of Contents

 

Executive Summary    4

Who Should Read This Paper?    4

Why Read This Paper?    4

Shift to Cloud Computing    5

Delivery Models for Cloud Services    6

Windows Azure Virtual Machines    7

SharePoint on Windows Azure Virtual Machines    7

Shift in IT Focus    8

Faster Deployment    8

Scalability    8

Metered Usage    8

Flexibility    9

Provisioning Process    9

Deploying SharePoint 2010 on Windows Azure    10

Creating and Uploading a Virtual Hard Disk    15

Usage Scenarios    16

Scenario 1: Simple SharePoint Development and Test Environment    16

Scenario 2: Public-facing SharePoint Farm with Customization    18

Scenario 3: Scaled-out Farm for Additional BI Services    20

Scenario 4: Completely Customized SharePoint-based Website    22

Conclusion    25

Additional Resources    25

 

 

Executive Summary

Microsoft SharePoint Server 2010 provides rich deployment flexibility, which can help organizations determine the right deployment scenarios to align with their business needs and objectives. Hosted and managed in the cloud, the Windows Azure Virtual Machines offering provides complete, reliable, and available infrastructure to support various on-demand application and database workloads, such as Microsoft SQL Server and SharePoint deployments.

While Windows Azure Virtual Machines support multiple workloads, this paper focuses on SharePoint deployments. Windows Azure Virtual Machines enable organizations to create and manage their SharePoint infrastructure quickly—provisioning and accessing nearly any host universally. It allows full control and management over processors, RAM, CPU ranges, and other resources of SharePoint virtual machines (VMs).

Windows Azure Virtual Machines mitigate the need for hardware, so organizations can turn attention from handling high upfront cost and complexity to building and managing infrastructure at scale. This means that they can innovate, experiment, and iterate in hours—as opposed to days and weeks with traditional deployments.

Who Should Read This Paper?

This paper is intended for IT professionals. Furthermore, technical decision makers, such as architects and system administrators, can use this information and the provided scenarios to plan and design a virtualized SharePoint infrastructure on Windows Azure.

Why Read This Paper?

This paper explains how organizations can set up and deploy SharePoint within Windows Azure Virtual Machines. It also discusses why this type of deployment can be beneficial to organizations of many sizes.

 

Shift to Cloud Computing

According to Gartner, cloud computing is defined as a “style of computing where massively scalable IT-enabled capabilities are delivered ‘as a service’ to external customers using Internet technologies.” The significant words in this definition are scalable, service, and Internet. In short, cloud computing can be defined as IT services that are deployed and delivered over the Internet and are scalable on demand.

Undeniably, cloud computing represents a major shift happening in IT today. Yesterday, the conversation was about consolidation and cost. Today, it’s about the new class of benefits that cloud computing can deliver. It’s all about transforming the way IT serves organizations by harnessing a new breed of power. Cloud computing is fundamentally changing the world of IT, impacting every role—from service providers and system architects to developers and end users.

Research shows that agility, focus, and economics are three top drivers for cloud adoption:

  • Agility: Cloud computing can speed an organization’s ability to capitalize on new opportunities and respond to changes in business demands.
  • Focus: Cloud computing enables IT departments to cut infrastructure costs dramatically. Infrastructure is abstracted and resources are pooled, so IT runs more like a utility than a collection of complicated services and systems. Plus, IT now can be transitioned to more innovative and strategic roles.
  • Economics: Cloud computing reduces the cost of delivering IT and increases the utilization and efficiency of the data center. Delivery costs go down because with cloud computing, applications and resources become self-service, and use of those resources becomes measurable in new and precise ways. Hardware utilization also increases because infrastructure resources (storage, compute, and network) are now pooled and abstracted.

Delivery Models for Cloud Services

In simple terms, cloud computing is the abstraction of IT services. These services can range from basic infrastructure to complete applications. End users request and consume abstracted services without the need to manage (or even completely know about) what constitutes those services. Today, the industry recognizes three delivery models for cloud services, each providing a distinct trade-off between control/flexibility and total cost:

  • Infrastructure as a Service (IaaS): Virtual infrastructure that hosts virtual machines and mostly existing applications.
  • Platform as a Service (PaaS): Cloud application infrastructure that provides an on-demand application-hosting environment.
  • Software as a Service (SaaS): Cloud services model where an application is delivered over the Internet and customers pay on a per-use basis (for example, Microsoft Office 365 or Microsoft CRM Online).

Figure 1 depicts the cloud services taxonomy and how it maps to the components in an IT infrastructure. With an on-premises model, the customer is responsible for managing the entire stack—ranging from network connectivity to applications. With IaaS, the lower levels of the stack are managed by a vendor, while the customer is responsible for managing the operating system through applications. With PaaS, a platform vendor provides and manages everything from network connectivity through runtime. The customer only needs to manage applications and data. (The Windows Azure offering best fits in this model.) Finally, with SaaS, a vendor provides the applications and abstracts all services from all underlying components.

Figure 1: Cloud services taxonomy


Windows Azure Virtual Machines

Windows Azure Virtual Machines introduce functionality that allows full control and management of VMs, along with extensive virtual networking. This offering can provide organizations with robust benefits, such as:

  • Management: Centrally manage VMs in the cloud with full control to configure and maintain the infrastructure.
  • Application mobility: Move virtual hard drives (VHDs) back and forth between on-premises and cloud-based environments. There is no need to rebuild applications to run in the cloud.
  • Access to Microsoft server applications: Run the same on-premises applications and infrastructure in the cloud, including Microsoft SQL Server, SharePoint Server, Windows Server, and Active Directory.

Windows Azure Virtual Machines is an easy, open and flexible, and powerful platform that allows organizations to deploy and run Windows Server and Linux VMs in minutes:

  • Easy: With Windows Azure Virtual Machines, it is easy and simple to build, migrate, deploy, and manage VMs in the cloud. Organizations can migrate workloads to Windows Azure without having to change existing code, or they can set up new VMs in Windows Azure in only a few clicks. The offering also provides assistance for new cloud application development by integrating the IaaS and PaaS functionalities of Windows Azure.
  • Open and flexible: Windows Azure is an open platform that gives organizations flexibility. They can start from a prebuilt image in the image library, or they can create and use customized and on-premises VHDs and upload them to the image library. Community and commercial versions of Linux also are available.
  • Powerful: Windows Azure is an enterprise-ready cloud platform for running applications such as SQL Server, SharePoint Server, or Active Directory in the cloud. Organizations can create hybrid on-premises and cloud solutions with VPN connectivity between the Windows Azure data center and their own networks.

SharePoint on Windows Azure Virtual Machines

SharePoint 2010 flexibly supports most of the workloads in a Windows Azure Virtual Machines deployment. Windows Azure Virtual Machines are an optimal fit for FIS (SharePoint Server for Internet Sites) and development scenarios. Likewise, core SharePoint workloads are also supported. If an organization wants to manage and control its own SharePoint 2010 implementation while capitalizing on options for virtualization in the cloud, Windows Azure Virtual Machines are ideal for deployment.

The Windows Azure Virtual Machines offering is hosted and managed in the cloud. It provides deployment flexibility and reduces cost by mitigating capital expenditures due to hardware procurement. With increased infrastructure agility, organizations can deploy SharePoint Server in hours—as opposed to days or weeks. Windows Azure Virtual Machines also enables organizations to deploy SharePoint workloads in the cloud using a “pay-as-you-go” model.
As SharePoint workloads grow, an organization can rapidly expand infrastructure; then, when computing needs decline, it can return the resources that are no longer needed—thereby paying only for what is used.

Shift in IT Focus

Many organizations contract out the common components of their IT infrastructure and management, such as hardware, operating systems, security, data storage, and backup—while maintaining control of mission-critical applications, such as SharePoint Server. By delegating all non-mission-critical service layers of their IT platforms to a virtual provider, organizations can shift their IT focus to core, mission-critical SharePoint services and deliver business value with SharePoint projects, instead of spending more time on setting up infrastructure.

Faster Deployment

Supporting and deploying a large SharePoint infrastructure can hamper IT’s ability to move rapidly to support business requirements. The time that is required to build, test, and prepare SharePoint servers and farms and deploy them into a production environment can take weeks or even months, depending on the processes and constraints of the organization. Windows Azure Virtual Machines allow organizations to quickly deploy their SharePoint workloads without capital expenditures for hardware. In this way, organizations can capitalize on infrastructure agility to deploy in hours instead of days or weeks.

Scalability

Without the need to deploy, test, and prepare physical SharePoint servers and farms, organizations can expand and contract compute capacity on demand, at a moment’s notice. As SharePoint workload requirements grow, an organization can rapidly expand its infrastructure in the cloud. Likewise, when computing needs decrease, the organization can diminish resources, paying only for what it uses. Windows Azure Virtual Machines reduces upfront expenses and long-term commitments, enabling organizations to build and manage SharePoint infrastructures at scale. Again, this means that these organizations can innovate, experiment, and iterate in hours—as opposed to days and weeks with traditional deployments.

Metered Usage

Windows Azure Virtual Machines provide computing power, memory, and storage for SharePoint scenarios, whose prices are typically based on resource consumption. Organizations pay only for what they use, and the service provides all capacity needed for running the SharePoint infrastructure. For more information on pricing and billing, go to Windows Azure Pricing Details. Note that there are nominal charges for storage and data moving out of the Windows Azure cloud from an on-premises network. However, Windows Azure does not charge for uploading data.

Flexibility

Windows Azure Virtual Machines provide developers with the flexibility to pick their desired language or runtime environment, with official support for .NET, Node.js, Java, and PHP. Developers also can choose their tools, with support for Microsoft Visual Studio, WebMatrix, Eclipse, and text editors. Further, Microsoft delivers a low-cost, low-risk path to the cloud and offers cost-effective, easy provisioning and deployment for cloud reporting needs—providing access to business intelligence (BI) across devices and locations. Finally, with the Windows Azure offering, users not only can move VHDs to the cloud, but also can copy a VHD back down and run it locally or through another cloud provider, as long as they have the appropriate license.

Provisioning Process

This subsection discusses the basic provisioning process in Windows Azure. The image library in Windows Azure provides the list of available preconfigured VMs. Users can publish SharePoint Server, SQL Server, Windows Server, and other ISO/VHDs to the image library. To simplify the creation of VMs, base images are created and published to the library. Authorized users can use these images to generate the desired VM. For more information, go to Create a Virtual Machine Running Windows Server 2008 R2 on the Windows Azure site. Figure 2 shows the basic steps for creating a VM using the Windows Azure Management Portal:

Figure 2: Overview of steps for creating a VM


Users also can upload a sysprepped image on the Windows Azure Management Portal. For more information, go to Creating and Uploading a Virtual Hard Disk. Figure 3 shows the basic steps for uploading an image to create a VM:

Figure 3: Overview of steps for uploading an image

Deploying SharePoint 2010 on Windows Azure

You can deploy SharePoint 2010 on Windows Azure by following these steps:

  1. Log on to the Windows Azure (Preview) Management Portal through your account.
  1. Create a VM with base operating system: On the Windows Azure Management Portal, click +NEW, then click VIRTUAL MACHINE, and then click FROM GALLERY.

  2. The VM OS Selection dialog box appears. Click Platform Images, select the Windows Server 2008 R2 SP1 platform image.

 

  1. The VM Configuration dialog box appears. Provide the following information:
  • Enter a VIRTUAL MACHINE NAME.
    • This machine name should be globally unique.
  • Leave the NEW USER NAME box as Administrator.
  • In the NEW PASSWORD box, type a strong password.
  • In the CONFIRM PASSWORD box, retype the password.
  • Select the appropriate SIZE.
    • For a production environment (SharePoint application server and database), it is recommended to use Large (4 Core, 7GB memory).

  1. The VM Mode dialog box appears. Provide the following information:
  • Select Standalone Virtual Machine.
  • In the DNS NAME box, provide the first portion of a DNS name of your choice.
    • This portion will complete a name in the format MyService1.cloudapp.net.
  • In the STORAGE ACCOUNT box, choose one of the following:
    • Select a storage account where the VHD file is stored.
    • Choose to have a storage account automatically created.
      • Only one storage account per region is automatically created. All other VMs created with this setting are located in this storage account.
      • You are limited to 20 storage accounts.
      • For more information, go to Create a Storage Account in Windows Azure.

 

  • In the REGION/AFFINITY GROUP/VIRTUAL NETWORK box, select the region where the virtual image will be hosted.

  1. The VM Options dialog box appears. Provide the following information:
  • In the AVAILABILITY SET box, select (none).
  • Read and accept the legal terms.
  • Click the checkmark to create the VM.

 

  1. The VM Instances page appears. Verify that your VM was created successfully.

  2. Complete VM setup:
  • Open the VM using Remote Desktop.
  • On the Windows Azure Management Portal, select your VM, and then select the DASHBOARD page.
  • Click Connect.

  1. Build the SQL Server VM using any of the following options:
  • Create a SQL Server 2012 VM by following steps 1 to 7 above—except in step 3, use the SQL Server 2012 image instead of the Windows Server 2008 R2 SP1 image. For more information, go to Provisioning a SQL Server Virtual Machine on Windows Azure.
    • When you choose this option, the provisioning process keeps a copy of SQL Server 2012 setup files in the C:\SQLServer_11.0_Full directory path so that you can customize the installation. For example, you can convert the evaluation installation of SQL Server 2012 to a licensed version by using your license key.
  • Use the SQL Server System Preparation (SysPrep) tool to install SQL Server on the VM with base operating system (as shown above in steps 1 to 7). For more information, go to Install SQL Server 2012 Using SysPrep.
  • Use the Command Prompt to install SQL Server. For more information, go to Install SQL Server 2012 from the Command Prompt.
  • Use supported SQL Server media and your license key to install SQL Server on the VM with base operating system (as shown above in steps 1 to 7).
  1. Build the SharePoint farm using the following substeps:
  • Substep 1: Configure the Windows Azure subscription using script files.
  • Substep 2: Provision SharePoint servers by creating another VM with base operating system (as shown above in steps 1 to 7). To build a SharePoint server on this VM, choose one of the following options:
  • Substep 3: Configure SharePoint. After each SharePoint VM is in the ready state, configure SharePoint Server on each server by using one of the following options:
    • Configure SharePoint from the GUI.
    • Configure SharePoint using Windows PowerShell. For more information, go to Install SharePoint Server 2010 by Using Windows PowerShell.
      • You also can use the CodePlex Project’s AutoSPInstaller, which consists of Windows PowerShell scripts, an XML input file, and a standard Microsoft Windows batch file. AutoSPInstaller provides a framework for a SharePoint 2010 installation script based on Windows PowerShell. For more information, go to CodePlex: AutoSPInstaller.

  1. After the script gets completed, connect to the VM using the VM Dashboard.
  2. Verify SharePoint configuration: Log on to the SharePoint server, and then use Central Administration to verify the configuration.

Creating and Uploading a Virtual Hard Disk

You also can create your own images and upload them to Windows Azure as a VHD file. To create and upload a VHD file on Windows Azure, follow these steps:

  1. Create the Hyper-V-enabled image: Use Hyper-V Manager to create the Hyper-V-enabled VHD. For more information, go to Create Virtual Hard Disks.
  2. Create a storage account in Windows Azure: A storage account in Windows Azure is required to upload a VHD file that can be used for creating a VM. This account can be created using the Windows Azure Management Portal. For more information, go to Create a Storage Account in Windows Azure.
  3. Prepare the image to be uploaded: Before the image can be uploaded to Windows Azure, it must be generalized using the SysPrep command. For more information, go to How to Use SysPrep: An Introduction.
  4. Upload the image to Windows Azure: To upload an image contained in a VHD file, you must create and install a management certificate. Obtain the thumbprint of the certificate and the subscription ID. Set the connection and upload the VHD file using the CSUpload command-line tool. For more information, go to Upload the Image to Windows Azure.

 

Usage Scenarios

This section discusses some leading customer scenarios for SharePoint deployments using Windows Azure Virtual Machines. Each scenario is divided into two parts—a brief description about the scenario followed by steps for getting started.

Scenario 1: Simple SharePoint Development and Test Environment

Description

Organizations are looking for more agile ways to create SharePoint applications and set up SharePoint environments for onshore/offshore development and testing. Fundamentally, they want to shorten the time required to set up SharePoint application development projects, and decrease cost by increasing the use of their test environments. For example, an organization might want to perform on-demand load testing on SharePoint Server and execute user acceptance testing (UAT) with more concurrent users in different geographic locations. Similarly, integrating onshore/offshore teams is an increasingly important business need for many of today’s organizations.

This scenario explains how organizations can use preconfigured SharePoint farms for development and test workloads. A SharePoint deployment topology looks and feels exactly as it would in an on-premises virtualized deployment. Existing IT skills translate 1:1 to a Windows Azure Virtual Machines deployment, with the major benefit being an almost complete cost shift from capital expenditures to operational expenditures—no upfront physical server purchase is required. Organizations can eliminate the capital cost for server hardware and achieve flexibility by greatly reducing the provisioning time required to create, set up, or extend a SharePoint farm for a testing and development environment. IT can dynamically add and remove capacity to support the changing needs of testing and development. Plus, IT can focus more on delivering business value with SharePoint projects and less on managing infrastructure.

To fully utilize load-testing machines, organizations can configure SharePoint virtualized development and test machines on Windows Azure with operating system support for Windows Sever 2008 R2. This enables development teams to create and test applications and easily migrate to on-premises or cloud production environments without code changes. The same frameworks and toolsets can be used on premises and in the cloud, allowing distributed team access to the same environment. Users also can access on-premises data and applications by establishing a direct VPN connection.

Getting Started

Figure 4 shows a SharePoint development and testing environment in a Windows Azure VM. To build this deployment, start by using the same on-premises SharePoint development and testing environment used to develop applications. Then, upload and deploy the applications to the Windows Azure VM for testing and development. If your organization decides to move the application back on-premises, it can do so without having to modify the application.

 

Figure 4: SharePoint development and testing environment in Windows Azure Virtual Machines


Setting Up the Scenario Environment

To implement a SharePoint development and testing environment on Windows Azure, follow these steps:

  1. Provision: First, provision a VPN connection between on-premises and Windows Azure using Windows Azure Virtual Network. (Because Active Directory is not being used here, a VPN tunnel is needed.) For more information, go to Windows Azure Virtual Network (Design Considerations and Secure Connection Scenarios). Then, use the Management Portal to provision a new VM using a stock image from the image library.
  • You can upload the on-premises SharePoint development and testing VMs to your Windows Azure storage account and reference those VMs through the image library for building the required environment.
  • You can use the SQL Server 2012 image instead of the Windows Server 2008 R2 SP1 image. For more information, go to Provisioning a SQL Server Virtual Machine on Windows Azure.
  1. Install: Install SharePoint Server, Visual Studio, and SQL Server on the VMs using a Remote Desktop connection.
  1. Develop deployment packages and scripts for applications and databases: If you plan to use an available VM from the image library, the desired on-premises applications and databases can be deployed on Windows Azure Virtual Machines:
  • Create deployment packages for the existing on-premises applications and databases using SQL Server Data Tools and Visual Studio.
  • Use these packages to deploy the applications and databases on Windows Azure Virtual Machines.
  1. Deploy SharePoint applications and databases:
  • Configure security on the Management Portal endpoint and set an inbound port in the VM’s Windows Firewall.
  • Deploy SharePoint applications and databases to Windows Azure Virtual Machines using the deployment packages and scripts created in step 3.
  • Test deployed applications and databases.
  1. Manage VMs:
  • Monitor the VMs using the Management Portal.
  • Monitor the applications using Visual Studio and SQL Server Management Studio.
  • You also can monitor and manage the VMs using on-premises management software, like Microsoft System Center – Operations Manager.

Scenario 2: Public-facing SharePoint Farm with Customization

Description

Organizations want to create an Internet presence that is hosted in the cloud and is easily scalable based on need and demand. They also want to create partner extranet websites for collaboration and implement an easy process for distributed authoring and approval of website content. Finally, to handle increasing loads, these organizations want to provide capacity on demand to their websites.

In this scenario, SharePoint Server is used as the basis for hosting a public-facing website. It enables organizations to rapidly deploy, customize, and host their business websites on a secure, scalable cloud infrastructure. With SharePoint public-facing websites on Windows Azure, organizations can scale as traffic grows and pay only for what they use. Common tools, similar to those used on premises, can be used for content authoring, workflow, and approval with SharePoint on Windows Azure.

Further, using Windows Azure Virtual Machines, organizations can easily configure staging and production environments running on VMs. SharePoint public-facing VMs created in Windows Azure can be backed up to virtual storage. In addition, for disaster recovery purposes, the Continuous Geo-Replication feature allows organizations to automatically back up VMs operating in one data center to another data center miles away. (For more information on geo-replication, go to Introducing Geo-replication for Windows Azure Storage).

VMs in Windows Azure infrastructure are validated and supported for working with other Microsoft products, such as SQL Server and SharePoint Server. Windows Azure and SharePoint Server are better together: Both are part of the Microsoft family and are thoroughly integrated, supported, and tested together to provide an optimal experience. They both have a single point of support for the SharePoint application and the Windows Azure infrastructure.

Getting Started

In this scenario, more front-end web servers for SharePoint Server must be added to support extra traffic. These servers require enhanced security and Active Directory Domain Services domain controllers to support user authentication and authorization. Figure 5 shows the layout for this scenario.

Figure 5: Public-facing SharePoint farm with customization


Setting Up the Scenario Environment

To implement a public-facing SharePoint farm on Windows Azure, follow these steps:

  1. Deploy Active Directory: The fundamental requirements for deploying Active Directory on Windows Azure Virtual Machines are similar—but not identical—to deploying it on VMs (and, to some extent, physical machines) on-premises. For more information about the differences, as well as guidelines and other considerations, go to Guidelines for Deploying Active Directory on Windows Azure Virtual Machines. To deploy Active Directory in Windows Azure:
  1. Provision a VM: Use the Management Portal to provision a new VM from a stock image in the image library.
  2. Deploy a SharePoint farm:
  • Use the newly provisioned VM to install SharePoint and generate a reusable image. For more information about installing SharePoint Server, go to Install and Configure SharePoint Server 2010 by Using Windows PowerShell or CodePlex: AutoSPInstaller.
  • Configure the SharePoint VM to create and connect to the SharePoint farm.
  • Use the Management Portal to configure the load balancing.
    • Configure the VM endpoints, select the option to load balance traffic on an existing endpoint, and then specify the name of the load-balanced VM.
    • Add another front-end web VM to the existing SharePoint farm for extra traffic.
  1. Manage VMs:

  • Monitor the VMs using the Management Portal.
  • Monitor the SharePoint farm using Central Administration.

Scenario 3: Scaled-out Farm for Additional BI Services

Description

Business intelligence is essential to gaining key insights and making rapid, sound decisions. As organizations transition from an on-premises approach, they do not want to make changes to the BI environment while deploying existing BI applications to the cloud. They want to host reports from SQL Server Analysis Services (SSAS) or SQL Server Reporting Services (SSRS) in a highly durable and available environment, while keeping full control of the BI application—all without spending much time and budget on maintenance.

This scenario describes how organizations can use Windows Azure Virtual Machines to host mission-critical BI applications. Organizations can deploy SharePoint farms in Windows Azure Virtual Machines and scale out the application server VM’s BI components, like SSRS or Excel Services. By scaling resource-intensive components in the cloud, they can better and more easily support specialized workloads. Note that SQL Server in Windows Azure Virtual Machines performs well, as it is easy to scale SQL Server instances, ranging from small to extra-large installations. This provides elasticity, enabling organizations to dynamically provision (expand) or deprovision (shrink) BI instances based on immediate workload requirements.

Migrating existing BI applications to Windows Azure provides better scaling. With the power of SSAS, SSRS, and SharePoint Server, organizations can create powerful BI and reporting applications and dashboards that scale up or down. These applications and dashboards also can be more securely integrated with on-premises data and applications. Windows Azure ensures data center compliance with support for ISO 27001. For more information, go to the Windows Azure Trust Center.

Getting Started

To scale out the deployment of BI components, a new application server with services such as PowerPivot, Power View, Excel Services, or PerformancePoint Services must be installed. Or, SQL Server BI instances like SSAS or SSRS must be added to the existing farm to support additional query processing. The server can be added as a new Windows Azure VM with SharePoint 2010 Server or SQL Server installed. Then, the BI components can be installed, deployed, and configured on that server (Figure 6).

Figure 6: Scaled-out SharePoint farm for additional BI services

Setting Up the Scenario Environment

To scale out a BI environment on Windows Azure, follow these steps:

  1. Provision:
  • Provision a VPN connection between on premises and Windows Azure using Windows Azure Virtual Network. For more information, go to Windows Azure Virtual Network (Design Considerations and Secure Connection Scenarios).
  • Use the Management Portal to provision a new VM from a stock image in the image library.
    • You can upload SharePoint Server or SQL Server BI workload images to the image library, and any authorized user can pick those BI component VMs to build the scaled-out environment.
  1. Install: If your organization does not have prebuilt images of SharePoint Server or SQL Server BI components, install SharePoint Server and SQL Server on the VMs using a Remote Desktop connection.
  1. Add the BI VM:
  • Configure security on the Management Portal endpoint and set an inbound port in the VM’s Windows Firewall.
  • Add the newly created BI VM to the existing SharePoint or SQL Server farm.
  1. Manage VMs:
  • Monitor the VMs using the Management Portal.
  • Monitor the SharePoint farm using Central Administration.
  • Monitor and manage the VMs using on-premises management software like Microsoft System Center – Operations Manager.

Scenario 4: Completely Customized SharePoint-based Website

Description

Increasingly, organizations want to create fully customized SharePoint websites in the cloud. They need a highly durable and available environment that offers full control to maintain complex applications running in the cloud, but they do not want to spend a large amount of time and budget.

In this scenario, an organization can deploy its entire SharePoint farm in the cloud and dynamically scale all components to get additional capacity, or it can extend its on-premises deployment to the cloud to increase capacity and improve performance, when needed. The scenario focuses on organizations that want the full “SharePoint experience” for application development and enterprise content management. The more complex sites also can include enhanced reporting, Power View, PerformancePoint, PowerPivot, in-depth charts, and most other SharePoint site capabilities for end-to-end, full functionality.

Organizations can use Windows Azure Virtual Machines to host customized applications and associated components on a cost-effective and highly secure cloud infrastructure. They also can use on-premises Microsoft System Center as a common management tool for on-premises and cloud applications.

Getting Started

To implement a completely customized SharePoint website on Windows Azure, an organization must deploy an Active Directory domain in the cloud and provision new VMs into this domain. Then, a VM running SQL Server 2012 must be created and configured as part of a SharePoint farm. Finally, the SharePoint farm must be created, load balanced, and connected to Active Directory and SQL Server (Figure 7).

Figure 7: Completely customized SharePoint-based website


Setting Up the Scenario Environment

The following steps show how to create a customized SharePoint farm environment from prebuilt images available in the image library. Note, however, that you also can upload SharePoint farm VMs to the image library, and authorized users can choose those VMs to build the required SharePoint farm on Windows Azure.

  1. Deploy Active Directory: The fundamental requirements for deploying Active Directory on Windows Azure Virtual Machines are similar—but not identical—to deploying it on VMs (and, to some extent, physical machines) on premises. For more information about the differences, as well as guidelines and other considerations, go to Guidelines for Deploying Active Directory on Windows Azure Virtual Machines. To deploy Active Directory in Windows Azure:
  1. Deploy SQL Server:
  • Use the Management Portal to provision a new VM from a stock image in the image library.
  • Configure SQL Server on the VM. For more information, go to Install SQL Server Using SysPrep.
  • Join the VM to the newly created Active Directory domain.
  1. Deploy a multiserver SharePoint farm:
  1. Manage the SharePoint farm through System Center:
  • Use the Operations Manager agent and new Windows Azure Integration Pack to connect your on-premises System Center to Windows Azure Virtual Machines.
  • Use on-premises App Controller and Orchestrator for management functions.

 

Conclusion

Cloud computing is transforming the way IT serves organizations. This is because cloud computing can harness a new class of benefits, including dramatically decreased cost coupled with increased IT focus, agility, and flexibility. Windows Azure is leading the way in cloud computing by delivering easy, open, flexible, and powerful virtual infrastructure. Windows Azure Virtual Machines mitigate the need for hardware, so organizations can reduce cost and complexity by building infrastructure at scale—with full control and streamlined management.

Windows Azure Virtual Machines provide a full continuum of SharePoint deployments. It is fully supported and tested to provide an optimal experience with other Microsoft applications. As such, organizations can easily set up and deploy SharePoint Server within Windows Azure, either to provision infrastructure for a new SharePoint deployment or to expand an existing one. As business workloads grow, organizations can rapidly expand their SharePoint infrastructure. Likewise, if workload needs decline, organizations can contract resources on demand, paying only for what they use. Windows Azure Virtual Machines deliver an exceptional infrastructure for a wide range of business requirements, as shown in the four SharePoint-based scenarios discussed in this paper.

Successful deployment of SharePoint Server on Windows Azure Virtual Machines requires solid planning, especially considering the range of critical farm architecture and deployment options. The insights and best practices outlined in this paper can help to guide decisions for implementing an informed SharePoint deployment.

Additional Resources

Retailers’ Mobile and Social Commerce Strategies Will Yield Minimal Revenue

Posted on by escapebusinesssolutions in Uncategorized

Predicts 2013: Retailers’ Mobile and Social Commerce Strategies Will Yield Minimal Revenue

30 November 2012 ID:G00231876

Analyst(s): Miriam Burt, Gale Daikoku, John Davison, Robert Hetu

VIEW SUMMARY

Tier 1 multichannel retailers will not gain significant benefits from mobile and social commerce strategies if they fail to understand how mobile and social customer interaction points can enhance and optimize cross-channel, customer shopping processes.


VIEW SUMMARY

Overview

Key Findings

  • Retailers will struggle to move significant numbers of consumers from cash and cards to Near Field Communication (NFC)-based mobile payments.
  • Retailers’ efforts to pursue location-based personalization offers will yield a very small rate of redemption.
  • Retailers will face a new threat to their profit margins and revenue-sharing models from emerging social shopping retailers.

Recommendations

For CIOs:

  • Ensure that you provide your customers with functionality on their mobile phones that they prefer to use, such as finding a store location or looking up stock availability, rather than NFC-based mobile payments.
  • Invest in multichannel analytical resources to help you increase revenue by defining more relevant coupon offers for all customers, while delivering personalized offers to a carefully chosen selection of customers.
  • Map your existing product catalog to the key demographics for social media to take advantage of the current low cost of entry to use the Facebook commerce (F-commerce) as a testing ground for social selling.

Analysis

What You Need to Know

Tier 1 multichannel retailers are still struggling to provide the everyday “business as usual” multichannel experience that their customers desire. Organizational and technology silos continue to hamper the delivery of a consistent and contiguous cross-channel customer shopping experience. Moreover, this is exacerbated by retailers investing in hyped-up mobile and social commerce solutions, rather than focusing on delivering the customer basics. For example, in store, some of the key customer basics are stock availability, an informed and available staff, and fast check-out.

Gartner predicts that retailers will struggle to move significant numbers of consumers from cash and cards, especially if they implement market-hyped, NFC-based, mobile wallet payment solutions. Our research confirms that customers’ preferences to use their mobile phones to find a store location, compare prices, look up stock availability and receive promotions are far ahead of their preferences to use their mobile phones to order and pay.

We predict retailers’ efforts to pursue context-aware personalized offers, such as location-based offers through mobile phones, will yield a very small rate of redemption. Our research shows that consumers favor paper coupons. In the near term, paper coupons will remain the dominant form of retail offer over electronic coupon redemption.

We predict that retailers will face a new threat to their profit margins and revenue-sharing models from emerging social shopping retailers. Our research shows that, with 93% share in the U.S., Facebook could become a virtual retailer by connecting manufacturers and distributors of consumer goods directly to the consumer (see Note 1). This shift could dramatically alter the profit margin and revenue-sharing models within the retailer and supplier networks, making it even more difficult for retailers to remain competitive.

Return to Top

Strategic Planning Assumptions

Strategic Planning Assumption: By 2014, less than 2% of consumers globally will adopt NFC-based mobile payments.

Analysis by: Miriam Burt and John Davison

Key Findings:

A Gartner consumer survey in 3Q11 in 10 countries showed that, on average:

  • Almost two-thirds of the consumers (62%) surveyed indicated that they did not use their mobile phones to conduct any type of financial transaction using mobile payment services.
  • Sixty percent of consumers indicated that concerns about the security of personal and payment data were the biggest barriers to using their mobile phones to make mobile payments. This is a 7% increase from the equivalent survey in 2010 (53%).
  • Seventy-nine percent of consumers indicated that the store is the main channel through which they were willing to make a purchase when conducting a cross-channel shopping event.

Market Implications:

This topic has been very hot in the past 12 months in all the major Tier 1 retail markets, with tremendous hype and publicity regarding solutions from a multitude of vendors of hardware, software, card payment services and, particularly, the NFC-based solutions, including those from Orange and Barclaycard, as well as the NFC-based mobile wallets from Isis and Google. Non-NFC-based solutions, such as the Starbucks mobile phone payment solution via its stored-value loyalty card and mobile bar codes, and PayPal’s mobile payment solution for Home Depot stores, have also been in the headlines.

About one-third of customers are using their mobile phones for financial transactions, although these are largely confined to functionality such as topping up prepaid mobile plans or purchasing digital products, not physical products. Moreover, this pertains to consumer usage of all types of mobile phones, not just to the usage of NFC-based mobile phones.

As part of a cross-channel shopping process, NFC-based mobile payments could address the need for speed of throughput and convenience during check-out in a store. This is important in some retail segments (such as grocery and convenience stores), but less important in other segments (such as luxury fashion).

If payment transaction fees using mobile devices are lower than traditional credit and debit cards, then there are clear savings for the retailer. However, in a 3Q11 retailer survey, Tier 1 retailers indicated that they expect the mobile channel, on average, to generate just under 2% of revenue through 2016, compared to 85% through the store and 12% through e-commerce. Hence, they do not see a robust business case for upgrading point of sale (POS) terminals to accept NFC-based, mobile contactless payments that include factors such as the cost of NFC-based POS terminal readers and the cost of merchant interchange fees.

Moreover, the speed of adoption of mobile payments will be dictated by consumers, so NFC-based payment solutions must demonstrate how they can support a secure, hassle-free, convenient and fast check-out — the latter being a key in-store customer service basic.

Current retailer trials of NFC-based stickers for promotions; the growing use of mobile coupons; the increasing use of mobile bar codes at the POS; and contactless payments using prepaid services for transportation applications, such as ticketing; may speed up the general adoption of NFC technology for mobile devices. For the most part, these are currently done through cards that customers touch on contactless readers, and do not involve NFC-enabled mobile phones in the payment process.

Benefits from NFC-based mobile payment transactions will only be gained if consumers are convinced that NFC mobile payments are secure, convenient and fast. They also need to have a compelling reason to make the switch. For example, retailers could give them incentives to choose this type of payment over others, such as tying loyalty programs into NFC-based mobile payments. In addition, a single set of standards needs to be agreed on by the banks, payment processing companies and retailers for NFC payments to succeed. We have not yet seen this consistency emerge yet.

Recommendations:

For CIOs:

  • Don’t let the projected rate of smartphone adoption or the hype around NFC-based contactless mobile payments drive investments in this solution.
  • Investigate how NFC can be used for nonpayment processes. For example, customers can use NFC stickers to access promotions or as a replacement for quick response (QR) code scanning.
  • Where appropriate, invest in secure, mobile POS applications in stores to enable store associates to provide the key customer basics of a fast and hassle-free check-out experience.
  • Ensure that you provide customers with functionality on their mobile phones that they prefer to use, rather than NFC-based mobile payments. This should include the capability to use their mobile phones to find a store location, compare prices, look up stock availability and receive promotions.
  • Trial mobile payments through lower-risk, stored-value payment solutions, preferably in conjunction with a loyalty solution.

Related Research:

“Hype Cycle for Retail Technologies, 2012”

“Distinguish How Consumers Want to Shop on Their Mobile Devices for Best Investment Decisions”

Strategic Planning Assumption: By 2015, less than 1% of all redeemed coupons will be location-aware offers sent by Tier 1 retailers.

Analysis by: Gale Daikoku and Robert Hetu

Key Findings:

  • Many Tier 1 retailers are pursuing personalization strategies that include the delivery of real-time offers on a customer’s mobile device while they are shopping in stores.
  • Paper coupons, including those provided directly to the customer, are the dominant format preferred by consumers.

Market Implications:

Retailers see personalization as a competitive necessity for building meaningful relationships that foster loyalty, yet many have work to do to support this level of engagement with customers. Gartner notes that nearly every Tier 1 retailer we speak with has prioritized the ability to gain a single view of its customers as a business priority, as personalization is dependent on good knowledge and segmentation of cross-channel customers. In fact, in the next few years, we expect that just over two-thirds (70%) of leading Tier 1 retailers will have improved the quality or way they develop customer offers. This can be anything from promotions to coupons or personalized offers. However, many retailers will be challenged by their abilities to deliver and execute location-aware offers to customers who are shopping in their stores.

Coupons are the most common form of retail offers. According to industry sources, even though coupon distribution is down overall due to more limited funding by manufacturers, paper coupons, many of which are delivered via free standing inserts (FSIs) that are mailed to a customer’s home, are expected to remain the dominant form of retail offer for some time. Gartner research shows that the vast majority of consumers prefer to use some form of paper coupon — in particular, those sent directly to the customer, delivered as a separate sheet with a sales receipt or from the retailer’s in-store mailer.

Electronic coupon distribution is an alternate, cost-efficient way to reach customers with offers. There are two types of e-coupons that are rated fairly high for potential use: those emailed to customers that can be printed out on paper and coupons that can be saved to loyalty accounts. However, customers are still getting used to these newer forms of couponing, and redemption will remain constrained in the near term, due to process and technology challenges in stores.

As mobile technology improves, customers will get more comfortable with using mobile devices as part of the shopping process. However, for retailers, communicating the perfect multichannel offer at the right moment in the shopping process, although theoretically attractive, is difficult to execute in real time (for example, sending location-based, context-aware, personalized mobile coupons in real time while customers are shopping in their stores).

Apart from customer readiness, there are customer privacy challenges around sending mobile coupons to customers’ personal mobile devices. In Gartner’s consumer survey, 39% of respondents said they had smartphones. However, when asked if they were willing to use a mobile app to receive coupons or rewards in stores, almost one-third (29%) indicated that that were not willing to use a smartphone app. Furthermore, 40% said that they were not willing to register their phones so that they can be tracked to receive offers while they were shopping in the store.

Recommendations:

For CIOs:

  • Maintain the capability and processes that support the delivery and redemption of paper coupons in stores.
  • Invest in the multichannel analytical resources to help you target a small, but lucrative, set of customers who are favorable to receiving personalized mobile offers.

Related Research:

“Hype Cycle for Retail Technologies, 2012”

“Consumer Survey Shows What’s Ahead for Retail Coupon Management”

“Personalization and Context-Aware Technology’s Impact on Multichannel Customer Loyalty”

“Marketing Service Provider Capabilities in Retail”

Strategic Planning Assumption: By 2015, a new social shopping retailer will emerge, accounting for 2% of U.S. Tier 1 retail sales.

Analysis by: Robert Hetu

Key Findings:

  • U.S. consumers in the 18- to 24-year-old demographic have the highest preference for using social media while shopping (48%), with expected continued growth in this activity as they grow older.
  • Facebook’s overwhelming share (93%) of social media provides it with an opportunity to generate a direct retail revenue stream via a virtual store approach.

Market Implications:

According to a Gartner consumer survey conducted in 3Q11, young adults aged 18- to 24-years-old conduct the highest levels of shopping-related social activity, including the propensity to look for special offers and check product reviews on a social networking site. The depth of personal information willingly supplied on social networks provides unmatched visibility into the lives, interests and personal networks of consumers.

Retailers primarily view the customer through their interactions via shopping activity. As they seek to expand their knowledgebases, they have been experimenting with social media through various partnerships. Most Tier 1 retailers have built a relationship with Facebook as a convenient avenue to access social networks via Facebook commerce (or F-commerce), but have found little revenue.

As experienced by retailers when they were part of the Amazon platform, the extent of information flowing between partner organizations (for example, sales and inventory data at a SKU level, as well as customer transaction data) can be used by a partner as it builds its own retail strategy. Complicating this further is that the social environment is dominated by a single player that has 93% share in the U.S. As a result, the partnership with Facebook, or with other social players that currently exist or may rise as rivals of Facebook, could provide the competitive content required to quickly enable a virtual social retailer, connecting manufacturers and distributors of consumer goods directly to the consumer. This shift could dramatically alter the profit margin and revenue-sharing models within the retailer and supplier networks, making it even more difficult for retailers to remain competitive.

Amazon grew to be a top U.S. retailer by taking advantage of a specific weakness in the retail environment — namely, the slow adoption of technology-enabled shopping. It is continuing to exploit this weakness with its forays into mobile shopping and social enablement. Amazon now includes the setup of a profile with personal information, including a picture and preferences, with the ability to share previously purchased items with others to help facilitate social shopping. Customers can also create lists of products and purchase guides to support various activities (for example, photography). As a result, Amazon is far ahead of multichannel, Tier 1 retailers in its ability to meet this new threat posed by social networks, such as Facebook.

Recommendations:

For CIOs:

  • Map your existing product catalog to the key demographics for social media to take advantage of the low cost of entry to use the F-commerce platform as a testing ground for social selling.
  • Connect e-commerce and m-commerce sites with Facebook using social plug-ins and custom applications to ensure a consistent flow for the cross-channel customer.
  • Approach F-commerce with the awareness that Facebook will use this as a revenue-generating model by planning alternative approaches for social commerce when it becomes disadvantageous to engage in F-commerce.

Related Research:

“Use Facebook to Test Social Commerce Strategy”

“Information Innovation Powers Customer-Centric Merchandising”

“Hype Cycle for Retail Technologies, 2012”

Return to Top

A Look Back

In response to your requests, we are taking a look back at some key predictions from previous years. We have intentionally selected predictions from opposite ends of the scale — one where we were wholly or largely on target, as well as one we missed.

On Target: 2012 Prediction — By 2013, the U.K. retail market will be the world’s most advanced multichannel market.

Analysis by: Miriam Burt, Van L. Baker, Gale Daikoku, John Davison, Robert Hetu

Previously Published: “Predicts 2012: Retailers Turn to Personalized Offers Through Mobile and Social but Will Struggle With Multichannel Execution”

In 2Q11 and 3Q11, Gartner surveyed leading retailers in the U.S., Canada, the U.K., France, Germany, Brazil, Russia, India, China and Japan. The survey asked retailers to estimate the percentage of revenue coming from each of their selling channels. In this survey, U.K. retailer estimates for the percentage of e-commerce sales were higher than in any other country surveyed, with 13.22% of sales compared to the 10-country average of 9.22% of sales for this channel. The same finding was made in other channels — mail order and catalog, call center, and mobile — where the U.K. had a higher estimated percentage than the 10-country average.

The corollary of this is that the U.K. retailers surveyed anticipated a smaller percentage of their revenue through 2014 coming from brick-and-mortar stores (79.28%), compared to the 10-country average of anticipated revenue in 2014 (88.56%). Thus, the U.K. is on target to be the world’s most advanced multichannel market through 2013 and beyond.

Missed: 2012 Prediction — By year-end 2012, 90% of Tier 1 and Tier 2 retailers will have an active presence on social media sites.

Analysis by: Miriam Burt, Van L. Baker, Gale Daikoku, John Davison, Robert Hetu

Previously Published: “Predicts 2012: Retailers Turn to Personalized Offers Through Mobile and Social but Will Struggle With Multichannel Execution”

In defining active presence, the intent was to incorporate some commerce activity within the social networking activity. Facebook commerce was, at the time, being pursued by many Tier 1 retailers. Over time, they learned that selling products on social media was not a straightforward process and retreated. Presently, most Tier 1 retailers have a presence on social media. However, they have not successfully monetized through F-commerce or other sales activities. Many even failed to be responsive to customer comments, making it a one-way communication vehicle.

Return to Top

SharePoint Lifecycle Management Solution with Project Server 2010 – Setup Guide

Posted on by escapebusinesssolutions in Uncategorized


 

 

SharePoint Lifecycle Management Solution with Project Server 2010 – Setup Guide

 

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

Some examples depicted herein are provided for illustration only and are fictitious.  No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

 

© 2011 Microsoft Corporation.

 

Whitepaper: SharePoint Lifecycle Management Solution with Project Server 2010 – Setup Guide

Authors:    Scott Jamison and Mark Candelora, Jornata LLC; Christophe Fiessinger, Senior Technical Product Manager, Microsoft

Published: May 2011

Applies to: Microsoft SharePoint Server 2010 and Microsoft Project Server 2010

Summary: This white paper provides instructions for configuring the Microsoft SharePoint
Server 2010 lifecycle management solution that runs atop Microsoft Project Server 2010. (37 printed pages)

 

Contents

Contents    3

About this white paper    5

Option 1: Step-by-Step Configuring your SharePoint Lifecycle Management Project Application    6

Create custom fields    6

Lookup tables    6

Create Custom Fields    7

Business drivers    8

Create Drivers    8

Prioritize Drivers    9

Create Project Detail Pages (PDP)    10

Create workflow stages    10

Install & configure the Workflow Visualization Web Part    12

Install the web part    12

Configure the web part    13

Install & configure Dynamic Workflow WSP    14

Install the Dynamic Workflow solution    14

Configure the Dynamic Workflow solution    14

Create generic resources    19

Install Project site template    19

Install Custom Site Template    20

Create EPT    20

Option 2: Restoring From Database Backup Files    21

Restore & Attach Databases    21

Attach Content DB    22

Deploy Dynamic Workflow and Workflow Visualization solutions    22

Provision a new Project Web Application    24

PWA Settings    25

Installing the Solution in the Information Worker Demonstration and Evaluation Virtual Machine    26

Using the Solution    28

Creating a Proposal    28

Gathering Information    28

Getting Approval    31

Project Selection    31

Project Execution    34

Post Mortem    34

Where to go from here    35

Business demonstration    35

Adapt for your organization    35

Making it real    35

Learn about Microsoft Project Server 2010    35

About the Authors    36

Artifacts    36

List of Figures    37

References    38

 

 

About this white paper

Microsoft SharePoint Server 2010 provides a vast number of capabilities that empower both business users and IT to create solutions quickly. For this reason, many organizations consider implementing SharePoint as a central platform for a wide array of business solutions.

 

For those organizations that have put SharePoint in place to handle the wide array of business needs (or are planning to do so), it’s likely that they’ll need a good way to track, manage, and prioritize those business requests. The white paper entitled SharePoint Lifecycle Management Solution with Project Server 2010 provides a suggested solution based on Microsoft Project Server 2010. The white paper is available for download at http://go.microsoft.com/fwlink/?LinkID=218030.

 

This white paper (the one you reading) is an adjunct paper that will walk through the steps necessary to begin using Microsoft Project Server 2010 to help prioritize and schedule the SharePoint projects that have been requested by a business user or group. This document assumes you will be working in a farm that has a working installation of SharePoint 2010 with Microsoft Project Server 2010 fully configured (please refer to Project Server 2010 Tech Center: http://technet.microsoft.com/en-us/projectserver/ee263909).

 

There are two ways to begin using this Solution:

  1. A manual, step-by-step creation of each Microsoft Project Server 2010 PWA instance, or
  2. By restoring the database backups provided in the download.

 

With the step-by-step process outlined in the section entitled Option 1, you will learn about what each entity does and have the opportunity to customize them as they are created. In the section entitled Option 2, you’ll instead use database backups, which enables you to get started right away. The backups also come pre-populated with sample projects that can be used to view portfolios or create reports. It is not recommended to use the backups as anything other than a demo that can be used to learn and experiment with or to present to others. To implement a solution in a production environment, the entities should be created and finalized in a development or test environment, thoroughly tested, then moved to production using the DM Import/Export tools (found in the Microsoft Project 2010 Solution Starters download).

 

This solution requires artifacts from a separate download. Before starting this instruction guide, please download the required artifacts from http://go.microsoft.com/fwlink/?LinkID=218030 and the deployment download from http://archive.msdn.microsoft.com/P2010SolutionStarter. Please see the Artifacts section in this document for specific details on the files.

 

 

Option 1: Step-by-Step Configuring your SharePoint Lifecycle Management Project Application

Once your Project Server Web Application (PWA) has been properly configured, open your browser to the PWA home page (by default this is http://<server>/pwa).

Create custom fields

Custom fields provide the ability to collect information on different project related objects as required by the business.

  1. In the quick links, under Settings, select Server Settings. Under Enterprise Data, select Enterprise Custom Fields and Lookup Tables.

Lookup tables

Lookup tables provide a specific list of values for fields that require a limited number of options. These    lookup tables will be the options for the fields we create.

  1. Scroll to the second section called Lookup Tables for Custom Fields and click New Lookup Table
  2. Create the following tables with the specified values.

Roles

Name: Role_LT

Type: Text

Code
Mask: leave default

Lookup Table:

Level

Value

1

Developer

1

Administrator

1

QA Engineer

1

Architect

1

DBA

1

Business Liaison

1

Analyst

Display order: By row number

Project Types

Name: ProjectType_LT

Type: Text

Code
Mask: leave default

Lookup Table:

Level

Value

1

Feature activation

1

Search content source

1

Business workflow process

1

Executive BI dashboard

1

Business entity (BCS) connection

1

SharePoint custom software development

1

Other

Display order: By row number

Risk Levels

Name: Risk_LT

Type: Text

Code
Mask: leave default

Lookup Table:

Level

Value

1

Very High Risk

1

High Risk

1

Medium Risk

1

Low Risk

Display order: By row number

 

 

One To Ten

Name: OneToTen_LT

Type: Number

Lookup Table:

Value

Description

1

The project went terribly – possibly due to budgeted time, resources or other estimates were incorrect; changing scope, technology or implementation issues, etc.

2

  

3

  

4

  

5

The project was ok – there were some aspects of the project that went smoothly, but other aspects were not.

6

  

7

  

8

  

9

  

10

The project was a complete success – all aspects of the project went according to plan, the project finished on time and under budget.

Display order: By row number

Create Custom Fields

Project Custom Fields

    Project custom fields are used to collect information on projects for reporting or workflow purposes. For more information on how to manage Enterprise Custom Fields and Lookup Tables (Project Server 2010) please go to http://technet.microsoft.com/en-us/library/gg709725.aspx .

  • Under the section called Enterprise Custom Fields, click New Field

    Create the following fields with the specified values. Attributes not listed in the table below can be left at     their default values.

 

Name

Description

Entity

Type

Custom Attributes

Behavior

Estimated Cost

How much it will cost for additional resources i.e. additional hardware, software, licenses, etc.

Project

Cost

  

Controlled by workflow

Estimated Effort

An estimate of FTE days to complete this project, including all planning, execution, testing, etc…

Project

Duration

  

Controlled by workflow

Additional Technical Description

Any additional technical information to take note of during planning stages of this project.

Project

Text

  

Controlled by workflow

Risk

How likely it is that this project will remain within it’s time/scope/budget estimates

Project

Text

Lookup Table

-Risk_LT

Controlled by workflow

SharePoint Project Type

The type of project this is

Project

text

Lookup Table

-ProjectType_LT

Controlled by workflow

Post Mortem Notes

Notes gathered from the post mortem discussions. What did we do well? What could have been improved? Any changes to documentation / forms / process?

Project

Text

  

Controlled by workflow

Overall Project Health Rating

1 – 10 rating of the project’s planning & execution.

Project

Number

Lookup Table

-OneToTen_LT

Controlled by workflow

 

    Resource custom fields

    Resource custom fields are used to add information about resources for reporting or workflow purposes.

  • Under the section called Enterprise Custom Fields, click New Field.

 

    Create the following fields with the specified values. Attributes not listed in the table below can be left at     their default values.

Name

Description

Entity

Type

Custom Attributes

Behavior

Role

  

Resource

Text

Lookup Table

-Role_LT

Required

 

Business drivers

Business drivers are key reasons why your business implements software. Business drivers are used to assess project strategic value and to assure that project selection supports the organizational strategy. These drivers are priorities that allow the business to be more effective. For more information please refer to Portfolio Analysis with Microsoft Project Server 2010 (white paper)

 

Create Drivers

  • From your PWA home page, under Strategy, select Driver Library from the quick links menu.
  1. Click the New button in the ribbon

Create the following drivers with the specified values. Attributes not listed in the table below can be left at their default values.

 

Name

Description

Project Impact Statements

Cost Avoidance

Is there clear cost avoidance as a result of implementing this solution?

**Refer to Project Impact Statement table below**

Employee Retention

Note: this driver comes with the base Project Server installation

How will our employee retention improve as a result of implementing this solution?

**Refer to Project Impact Statement table below**

Productivity Gains

Improvements in a group or department’s effectiveness that would allow them to get work done faster, or with better quality.

**Refer to Project Impact Statement table below**

Quality Improvements

Will we attain a measurable quality improvement as a result of implementing this solution?

**Refer to Project Impact Statement table below**

Risk

Will we reduce risk as a result of implementing this solution?

**Refer to Project Impact Statement table below**

ROI

Will we obtain a clear return on investment (ROI) as a result of implementing this solution?

**Refer to Project Impact Statement table below**

Time-to-Market

How much faster can we get a product to market as a result of implementing this solution?

**Refer to Project Impact Statement table below**

 

Use these Project Impact Statements for each of the above drivers. Be sure to replace “<driver>” with the name of the driver.

Project Impact Level

Description

None

No measurable impact.

Low

Little impact on <driver> – potentially 0-3% increase in <driver>.

Moderate

Some impact on <driver> – potentially 3-10% increase in <driver>.

Strong

Strong impact on <driver> – potentially 10-20% increase in <driver>.

Extreme

Very large impact on <driver> – potentially over 20% increase in <driver>.

 

Prioritize Drivers

Driver prioritization will help automatically rank projects based on their strategic rating. Your project portfolio will be sorted based on the priorities you set here.

  1. Select Driver Prioritization in the quick links menu. Click New in the ribbon.
  2. Enter a Name and Description for your driver prioritization.
  3. Select your prioritization type – Calculated or Manual.
    1. Calculated – you’ll walk through a wizard that will compare the priority of each driver against all the others.
    2. Manual – you will manually enter the percentage value of each priority. If the amounts do not add up to 100% when save is clicked the numbers will be adjusted to equal 100%
  4. Select all of your drivers then click Next: Prioritize Drivers.
  5. After walking through the wizard or entering percentages, review your priority levels and click Save.

    *Note – If using the Calculated Wizard you cannot edit the Priority Percentages

Create Project Detail Pages (PDP)

Project detail pages collect information from the user at specific points in the project lifecycle. Each stage must have a project detail page associated. Please refer to Workflow and Project Detail Pages for more information.

  • In the quick links, select Server Settings. Under the Workflow and Project Detail Pages section, select Project Detail Pages.
  • In the Documents tab in the ribbon, click New Document
  • Always select “Full Page, Vertical” as Layout Template.

Create the following project detail pages with the specified values. Attributes not listed in the table below can be left at their default values.

Name

Web Part

WP Configuration

Initial SharePoint Proposal

Project Fields

Fields:

Project Name
Description

Initial SharePoint Requirements

Project Fields

Fields:

Estimated Cost

Estimated Effort

Additional Technical Description

Risk

Project Type

Initial SharePoint Requirements Strategic Impact

Project Strategic Impact

  

SharePoint Schedule

Project Details

  

SharePoint Post Mortem

Project Fields

Fields:

Post Mortem Notes

Overall Project Health Rating

 

Create workflow stages

Stages represent specific points in a project lifecycle that a project must move through in order to complete. Each stage has specific requirements that must be met before a project can move on to the next stage.

  1. From the Server Settings page, select Workflow Stages under Workflow and Project Detail Pages.
  2. Click New Workflow Stage.

Create the following project detail pages with the specified values. Attributes not listed in the table below can be left at their default values.

Name

Description

Description for Submit

Workflow Phase

Visible Project Detail Pages

Required Custom Fields

Strategic Impact Behavior

Project CheckIn Required

01 – SP Initial Proposal

A new proposal was submitted.

Upon submission, this proposal will be forwarded to an administrator who will gather information on requirements, business drivers, and usage to create estimates for budget, timeline, resources, etc.

Create

– Initial SharePoint Proposal

  

Read Only

No

02 – SP Initial Requirements

SharePoint Project Administrator will gather the project details to create a baseline estimate for the proposal.

Upon submission, this proposal will be slated for review.

Create

– Initial SharePoint

Requirements

– Initial SharePoint

Requirements

Strategic Impact

– SharePoint Schedule

– Estimated Cost

– Estimated

Effort

– Risk

– SharePoint

Project Type

Required

Yes

03 – SP Project Selection

The proposal is being reviewed for selection. If selected, it will move into the planning stages where resources will be assigned and task estimates created.

Upon submission the proposal will become a project where baseline estimates will become real tasks for resources to be assigned.

Select

– Initial SharePoint

Requirements

– Initial SharePoint

Requirements

Strategic Impact

– SharePoint Schedule

  

Read Write

No

04 – SP Execution

Work is being done on this project, see the project plan for more specifics on the progress of the project.

Upon submitted, this project will move into the post-mortem stage where the project execution is reviewed and potential improvements identified.

Manage

– SharePoint Schedule

  

Read Only

Yes

05 – SP Post Mortem Assessment

    

Finished

– SharePoint Post

Mortem

– Initial SharePoint

Requirements

– Initial SharePoint

Requirements

Strategic Impact

– SharePoint Schedule

– Overall Project

Health Rating

Read Only

No

Install & configure the Workflow Visualization Web Part

The Workflow Visualization Web Part (available as part of the Microsoft Project 2010 Solution Starters) provides a graphical display of the project workflow highlighting the current stage that the proposal is currently in. Deploying this Web Part is optional and is not required to implement this solution, it helps visualize workflow execution.

Install the web part

  1. Under the MS Project Solution Starters Deployment download, open the Workflow Visualization Webpart folder.

Figure 1 – Deploying the Dynamic Workflow solution

  1. Open and read the ReadMe.docx then edit the DeployPowerShell.cmd script. Edit the SiteUrl parameter so it equals the URL of your new PWA web app.
  2. Deploy the solution by running the DeployPowerShell.cmd script.

Configure the web part

  1. In the PWA site collection, go to Site Actions -> Site Settings, then under Site Collection Administration, select Site collection features
  2. Find and activate the Workflow Visualization feature.
  3. Go to Server Settings -> Workflow Stages. Open each workflow stage and expand the System Identification Data section at the bottom. Write down the name of the stage with the GUID that appears for each stage.
  4. Go to Site Actions -> More Options... In the Create dialog filter by Library on the left, then choose Asset Library in the middle section then click More Options. Enter StagesLibrary for the name, then check No for both Navigation and Item Version History, then click Create at the bottom of the window.
  5. In your newly created list, in the Documents tab of the ribbon go to Upload Document -> Upload Multiple Documents
  6. In the file system, find the WF Images folder, then drag and drop all of the images into the upload window.
  7. Once the files have finished uploading, click the Create Column option in the Library tab of the ribbon.
  8. In the Create Column dialog, enter StageUID for the name, leave all other options as blank then click OK.
  9. Under the Library tab in the ribbon, select Datasheet View. For each image enter the GUID into the StageUID field for each stage retrieved from Step 1. The arrow image will have a blank StageUID.
  10. Once this is complete, go back to Server Settings -> Project Detail Pages. Open the ProposalStageStatus page, then click Site Actions -> Edit Page.
  11. In the Full Page zone, click Add a Web Part. In the section that opens, select Project Server Add-Ons for the category. The Workflow Visualization web part will be the only option displayed. Click the add button to add the web part to the page.
  12. In the upper-left corner of the Workflow Visualization web part, click the down arrow, then click Edit Web Part. Enter the following information into the properties:

Property

Value

Document Library Name

StagesLibrary

Image for the arrow

Arrow.png

Arrow Image Width

70

Stage Image Width

120

Stage image Padding

0

Rich Highlighting

Checked

 

  1. Click OK to close the properties dialog, and then click Stop Editing in the Page ribbon.

Install & configure Dynamic Workflow WSP

The Microsoft Project Server 2010 Dynamic Workflow (available as part of the Microsoft Project 2010 Solution Starters) will guide your project through the required lifecycle stages, prompting for approvals and portfolio commitments along the way. The Dynamic Workflow solution starter provides an easy mechanism to build sequential workflows in PWA without using Visual Studio. The tool inserts a custom InfoPath form into the workflow definition process that gathers the custom workflow information. That information is then used to dynamically construct a workflow for the demand management process.

Install the Dynamic Workflow solution

  1. Under the MS Project Solution Starters Deployment download, unzip the dynamic workflow component.
  2. Open and read the ReadMe.docx then edit the DeployPowerShell.cmd script. Edit the SiteUrl parameter so it equals the URL of your new PWA web app.
  3. Deploy the solution by running the DeployPowerShell.cmd script.
  4. In the PWA site collection, go to Site Actions -> Site Settings, then under Site Collection Administration, select Site collection features
  5. Find and activate the DemandManagement DynamicWorkflow feature

Configure the Dynamic Workflow solution

  1. Go back to the Site Settings page and under Site Administration, click Workflow settings
  2. Click Add a workflow and select DM DynamicWorkflow. Name it SharePoint Lifecycle Management, and leave the other defaults on the form.
  3. In each section of the form, fill out the details as shown below then click the Add Stage button at the top of the section to add a new stage.

Be sure not to click the Submit button until all stages have been completed.

 


Figure 2 – Stage 1: Initial Proposal

Figure 3 – Stage 2: Initial Requirements

Figure 4 – Stage 2: Approval

Figure 5 – Stage 3: Project Selection

Figure 6 – Stage 3: Approval

Figure 7 – Stage 4: Execution

Figure 8 – Stage 5: Post Mortem Assessment

The image below shows how the finished workflow configuration should look.

Figure 9 – Dynamic Workflow Stage Definition

  1. Once the workflow has been completely configured, click Submit.
  2. Go to Server Settings -> Change or Restart Workflows and enter the following information into the form:

Field

Value

Choose Enterprise Project Type

SharePoint Enhancement

Choose Projects

<all>

Choose new Enterprise Project Type or restart current workflow

Associate projects with a new Enterprise Project Type: SharePoint Enhancement

Choose Workflow Stage

Skip until the current workflow stage

 

  1. Once the form has been filled out, click OK at the bottom of the page.

Create generic resources

Generic resources allow you to create estimates and assign resources by skill in order to get baseline cost and timeline estimates without having to assign an actual team. Please refer to http://office.microsoft.com/en-us/project-server-help/create-resources-to-represent-capacity-HA101865477.aspx for more information.

  1. In the quick links menu, under Resources, click the Resource Center link.
  2. In the ribbon on the Resource Center page, click New
    Resource:

 

Attributes not listed in the table below can be left at their default values.

 

Display Name

Generic

Role

SharePoint Developer

Yes

Developer

SharePoint Administrator

Yes

Administrator

SharePoint Analyst

Yes

Analyst

SharePoint Architect

Yes

Architect

Business Liaison

Yes

Business Liaison

QA Engineer

Yes

QA Engineer

 

 

Install Project site template

A Project site template provides a starting point for your projects. The template can contain predefined tasks, milestones, resource allocations and other items that can then be changed or customized for each project.

  1. Open MS Project Professional. Connect to your SharePoint Lifecycle Management PWA application.
  2. From MS Project Professional, open the project template from the SharePoint Lifecycle Solution Accelerator download.
  3. Under the Resource tab in the ribbon, click Add Resources -> Build Team from Enterprise…
  4. In the window that opens, select each of the generic resources that were created in the above step and add them to the project. It may help to set the Group by drop down to Generic then collapse the No group.
  5. Once all generic resources have been added, click the OK button to close the dialog.
  6. Add one or two resources for each task in the project template.
  7. Click File -> Save as… In the Save to Project Server dialog, type SharePoint Enhancement Template as the name. Select Template as the type. Leave the other defaults and select Save.
  8. In the Save as Template dialog box, select all the checkboxes and click Save.

Install Custom Site Template

The custom site template provides custom lists, pages, and other items that can provide a customized starting point for your project collaboration site.

  1. Open the Solution Gallery – go to Site Actions -> Site Settings, and under Galleries select Solutions.
  2. In the Solutions tab in the ribbon, select Upload Solution. Browse to the solution files, and find the SPEnhProj.wsp file under the Site Template folder. Click Activate when the popup appears.
  3. Open the Site collection features (go to Site Actions -> Site Settings then select Site collection features). Find and enable the feature Web Template feature of exported web template SharePoint Enhancement Site Template.

Create EPT

The EPT or Enterprise Project Type combines a workflow (consisting of stages & PDPs) to a project template and a custom site template. Each EPT can we used to model specific work request type in your PWA instance. For more information please refer to http://office.microsoft.com/en-us/project-server-help/enterprise-project-types-HA100955930.aspx

  1. Open the Project Server Settings page (from the PWA home page, select Server Settings). Under the Workflow and Project Detail Pages section, select Enterprise Project Types.
  2. Click New Enterprise Project type:

Property

Value

Name

SharePoint Enhancement

Description

A new enhancement for SharePoint 2010

Site Workflow
Association

SharePoint Lifecycle Management

Default

No

Departments

<blank>

Image

<blank>

Order

Position at end

Project
Plan
Template

SharePoint Enhancement Template

Project
Site
Template

SPEnhProj

 

Option 2: Restoring From Database Backup Files

This method will use a five database attach (Draft, Published, Reporting, Archive and Content databases) to restore the SharePoint Lifecycle demo server. The backup files come as backups of the 5 databases required for MS Project Server to function: Archive, Draft, Published, Reporting, and the SharePoint Content database. For more detailed instructions on the five databases attach restore method go see Database-attach full upgrade to Project Server 2010. Please note that the sample databases were created using the February 2011 Project Server 2010 Cumulative Update and hence you will need to apply this update to your test farm in order to perform the database attach procedure: http://support.microsoft.com/kb/2475879 .

 

Restore & Attach Databases

On your database server, using SQL Server Management Studio, restore each database backup file using the name of the file as the database name.


Attach Content DB

Attach the content database to the farm in central administration. For this restore option to work, the site collection restored from the content database must use the path http://<server>/PWA. If this path is not available, another web application must be selected to restore the content database to.

  1. Open the Central Administration console, and click Application Management. Under Databases click Manage content databases.
  2. In the Manage Content Databases page, click Add a content database.
  3. In the Add Content Database page, select the web application to add the content database to, then enter the SP_PWAContent for the Database Name.
  4. Leave the other settings with their defaults and click OK.

     


 

Deploy Dynamic Workflow and Workflow Visualization solutions

Deploy the workflow:

  1. Under the MS Project Solution Starters Deployment download, unzip the dynamic workflow component.
  2. Open and read the ReadMe.docx then edit the DeployPowerShell.cmd script. Edit the SiteUrl parameter so it equals the URL of your new PWA web app.

  1. Deploy the solution by running the DeployPowerShell.cmd script.

Deploy the workflow visualization web part:

  1. Under the MS Project Solution Starters Deployment download, open the Workflow Visualization Webpart folder.

Figure 13 – Deploy Workflow Visualization webpart

  1. Open and read the ReadMe.docx then edit the DeployPowerShell.cmd script. Edit the SiteUrl parameter so it equals the URL of your new PWA web app.
  2. Deploy the solution by running the DeployPowerShell.cmd script.

 

Provision a new Project Web Application    

  1. If you have not already done so, create a new Project Server Service Application under Application Management -> Service Applications-> Manage Service Applications
  2. In the Project Server Service Application Management page, click Project
    Server.
    1. **Note – the configuration requires the path http://<server>/pwa. If this site already exists, it must be removed before your new PWA can be attached to the service application.
  3. Click Create Project Web App Site
  4. Select the web application that your content database was restored to, then for each database name, enter the name of the corresponding database that was restored as part of the Restore & Attach Databases section. Be sure to leave the Project Web App path as its default PWA value since this is the path that the restored Site Collection is located.


PWA Settings

  1. Open your browser to your new PWA site. In the quick links bar, select Server Settings, then under Workflow and Project Detail Pages click Project Workflow Settings
  2. Change Workflow Proxy User account to the administrator account used to provision your PWA site and click Save.
  3. Go to Site Actions -> Site Settings and under Site Administration, select Workflow Settings.
  4. Click Add a workflow. Follow the steps detailed above in Configure the Workflow to set up the workflow.
  5. Once the workflow has been configured, go back to the PWA site and go to Server Settings -> Enterprise Project Types (under work flow and project detail pages). Select the SharePoint Enhancement EPT item.
  6. Under Site Workflow Association, select the newly created workflow and click Save.


 

Once you have all of the databases restored and the additional configuration changes made, you should see all of the projects in various stages and will be able to modify the settings to suit your organization’s needs.

 

Installing the Solution in the Information Worker Demonstration and Evaluation Virtual Machine

If you would like to evaluate this SharePoint Lifecycle Management Solution with Project Server 2010 in an existing demonstration environment, it can easily be achieved by leveraging the publically available demo virtual machine: 2010 Information Worker Demonstration and Evaluation Virtual Machine (RTM).

To deploy and configure this solution in the IW Demonstration & Evaluation VM do the following steps:

  1. Download, deploy and configure the IW Demonstration & Evaluation VM
  2. Download and apply at a minimum the February 2011 Project Server 2010 Cumulative Update: http://support.microsoft.com/kb/2475879  (since the PWA database schema has be patched with this latest CU)
  3. Restore all five databases within the VM’s SQL instance (you can mount the databases as an ISO file)
  4. Go to the Central Administration and add the Content database to the Intranet site

Figure 17 – Adding Content DB to IW Demo VM

  1. Deploy the Dynamic Workflow and Workflow Visualization WSP to http://intranet.contoso.com (see detailed steps by steps in earlier)
  2. Provision a new PWA instance to http://intranet.contoso.com (make sure you use the same database names you used when you restored them: ie. SPPWA_Published, etc.)

Figure 18 – Provision PWA instance in IW Demo VM

Figure 19 – PWA instance under Intranet.Contoso.Com

Additionally please check out the Microsoft Project 2010 Demonstration and Evaluation Installation Pack to further test out Microsoft Project and Project Server 2010 its capabilities and scenarios (contains hands-on-labs / demo scripts which showcases Microsoft Project 2010).

 

 

Using the Solution

Once the solution has been created, we can begin to create new project proposals.

Creating a Proposal

In your SharePoint Lifecycle PWA, go to your project center. In the projects ribbon, select New -> SharePoint Enhancement.


In the Initial Proposal form that displays, enter a name and description for the proposal. This information will assist the SharePoint Analyst in determining the requirements of the project and creating the initial scope.

Gathering Information

In this stage, the Business Analyst will gather information on the new project proposal, determine the technical requirements, create baseline estimates, and estimate the strategic impact of the project.

  1. Go to the Project Center. Find and click the new proposal.
  2. In the Proposal Stage Status page, you will see information regarding what needs to be done for the current stage. Click the Next button in the Project ribbon.
  3. The Initial Requirements form prompts the SharePoint Analyst to estimate the cost and effort for the project along with the risk associated with completing the project. The page also asks for additional technical information that might be required to complete the project and the SharePoint Project Type.


  1. Once the information has been entered, click Next in the Project ribbon. This will take you to the Scheduling page.
  2. A project template has been created with some baseline estimates. These estimates can be changed through the web interface or using MS Project Professional. In the Project tab in the ribbon, click Resource Plan.
  3. From the Plan tab, click Build Team. The Team Builder page will allow you to assign generic resources to your project to create your baseline estimates for the project. Select some generic resources for this project then click Save & Close.


  1. Once the resources have been selected, you will be able to create your resource plan. In the Plan ribbon, set Work Units to Full-time Equivalent, and Timescale to Months.
  2. Estimate the amount of FTE’s that will be needed for each resource type. Once that’s done click Save and then close the Resource Plan page.

After finalizing the estimates for your project, click the Next button in the Project tab in the ribbon.

  1. The strategic impact page will help evaluate the project proposal against the strategic goals of the company.


  1. After all strategic impact rankings have been made; click the Commit button in the Project ribbon. This will start the approval process for the project proposal.

Getting Approval

As a part of the Initial Requirements stage, executive approval is required for the project to continue. Once the project requirements, baseline estimates, and strategic impact have been submitted, the SharePoint governance team must meet to discuss the project and its merit along with any potential implications it may have.

  1. In the quick links bar, click the Workflow Approvals link.
  2. The approvals page will contain numerous approval tasks that represent pending approvals. Find the approval task for the new project and edit the item.


  1. Click the approve button in the upper-left corner. Once the project has been approved, it will move to the selection stage.

Project Selection

The project selection process will rank each project by their strategic value, then allow you to apply budget or resource constraints against them to determine what projects are feasible to implement.

  1. In the quick links bar, select Portfolio Analysis. Click New in the Analyses ribbon to create a new portfolio for analysis.
    1. Note that before doing this it is common to set up project dependencies to account for projects that may be mutually exclusive, or require other projects to be completed before another project to begin. To do this, select Project Dependencies from the Analyses tab.
  2. In the new analysis creation page, enter a name and description for your portfolio analysis. Next select the projects you wish to include. Also, be sure the Primary Cost Constraint is set to Estimated Cost.
  3. To include resource analysis in your project selection project check the Analyze time-phased project resource requirements checkbox. Be sure to set the following fields:
  • Resource role custom field – Role
  • Resource capacity impact – Committed and proposed assignments affect capacity
  1. Click Next: Prioritize Projects to see all of the projects’ strategic impact rankings. Here you can make modifications from what was originally entered in the Initial Requirements stage.


  1. Click Next: Review Priorities to review the relative priorities of all of the projects in your portfolio. No changes can be made in this screen.
  2. Click Next: Analyze Cost to see the estimated total cost of all projects in your portfolio. Enter a new budget amount under Cost Limits here to see which projects will be chosen based on your new budget constraint. Click Save As in the ribbon to save your current budget scenario.


  1. Click Next: Analyze Resources to see which projects can be selected based on resource constraints. Note that projects that were not selected due to budget constraints are not displayed in this part of the analysis. Under each project’s New Start column, select a new start date to stagger the start of your projects throughout the year. You can also choose to hire more resources to bring in more projects into your selected portfolio.
  2. Click the Requirements Details button in the Analysis ribbon to view the actual resource breakdown of how many hours are required for each type of resource for each project. In the top section, you can see how many FTE’s of each resource are available for allocation for that time period. Cells highlighted in red display a deficit in the required resources for that time period. Click Save As in the ribbon to save your current scenario.

 


  1. Once you have the projects that will be selected, the selected projects must be approved by the portfolio manager. In the quick links bar, select Workflow Approvals. Find the pending approval tasks for each of the selected projects to edit, then approve them.

Project Execution

This stage is where project work is executed. This includes actual task planning, detailed requirements gathering, developing test plans, as well as any development, testing, documentation, etc. Much of the task planning, and scheduling can be done by connecting MS Project Professional to your PWA instance. Project teams will have access to the team site for this project as well for collaboration on documentation, discussions, etc.

  • Once the execution phase is completed, open the Project Stage Status page and click Submit in the Project ribbon to move to the next stage.

 

Post Mortem

The Post Mortem stage is where we gather the “lessons learned”. It is common to hold a formal meeting with the project team and stakeholders to gather what went well and what did not.

 

  1. Open the Project Stage Status page and click the Next button in the Project ribbon.
  2. Enter a 1-10 rating of the overall project health (a simple measure of the project’s execution), then enter the notes and action items from the post mortem meeting.
  3. Click Submit in the Project ribbon to complete the project proposal.
  4. Check-in the project plan by clicking Close in the ribbon

 

Where to go from here

Once you’ve completed your evaluation using our sample data and solution accelerator, you can use the example solution to accomplish several additional tasks. For example, use the 5db backup with sample data to demonstrate the solution, update the configuration parameters to accommodate your particular organization’s particular business needs, or even work with a consulting partner to design a full solution based on these concepts; the following outlines what steps you might take.

Business demonstration

Using the backups with sample data provided gives a great starting point for a discussion around Demand Management and how this solution might be employed in your organization.

Adapt for your organization

Since the solution is based on simple configuration with no custom code, it is reasonably straight forward to configure & customize this solution to meet your organization’s business needs. The following items are some simple examples of what can be changed to drastically improve the relevance of this solution for your business:

  • Business Drivers
  • Project Plan Template
  • Enterprise Project Types
  • Workflow Phases and Stages
  • Project Detail Pages and Custom Fields

Making it real

In order to fully realize the benefit of this solution, you’ll want to engage a Microsoft Partner http://pinpoint.microsoft.com or Microsoft Consulting Services, who can help you with the following items:

  • SharePoint Governance & Maturity Model
  • Solution Customization
  • Business Process Engineering
  • End user training

Learn about Microsoft Project Server 2010

For more information on workflows in Project Server please visit Project Server 2010 Demand Management site for more information: http://technet.microsoft.com/en-US/projectserver/ff899331.aspx.

 

About the Authors

Scott Jamison is Chief Architect and CEO at Jornata (www.jornata.com), the worldwide experts in implementing collaborative solutions based on Microsoft SharePoint. Scott is an experienced leader with almost 20 years directing managers and technology professionals to deliver a wide range of business solutions for customers. Scott is co-author of Essential SharePoint 2010.

 

Mark Candelora is a Managing Consultant and Development Manager at Jornata. Mark is an experienced developer and architect with almost 10 years’ experience developing and delivering business-centric applications using various technologies including Microsoft SharePoint.

 

Christophe Fiessinger is Senior Technical Product Manager for Microsoft. As part of the Microsoft Office Division Product Marketing Group he focuses on the enterprise project and portfolio management solution. Be sure to check out the latest blog entries: http://blogs.msdn.com/chrisfie

 

 

Artifacts

The files below are required to complete the installation of this solution.

 

The following files are available for download at http://go.microsoft.com/fwlink/?LinkID=218030

  • 5db.zip – contains 5 database backup files (the files are used as part of Option 2: Restoring From Database Backup Files)
  • SPEnhProj.wsp – a SharePoint solution file containing the custom site template that will be used for projects in this solution
  • SharePoint Enhancement.mpt – a project template for use with the enterprise project type in this solution
  • WF Images.zip – contains the images that will be used for the Workflow Visualization web part.

 

The following files are available as part of the deployment download from the Microsoft Project Server 2010 Solution Starters at http://archive.msdn.microsoft.com/P2010SolutionStarter

  • DynamicWorkflow.zip – contains the deployment artifacts for the Dynamic Workflow solution
  • Workflow Visualization WebPart.zip – contains the deployment artifacts for the Workflow Visualization Web Part

 

 

List of Figures

Figure 1 – Deploying the Dynamic Workflow solution    12

Figure 2 – Stage 1: Initial Proposal    15

Figure 3 – Stage 2: Initial Requirements    16

Figure 4 – Stage 2: Approval    16

Figure 5 – Stage 3: Project Selection    17

Figure 6 – Stage 3: Approval    17

Figure 7 – Stage 4: Execution    18

Figure 8 – Stage 5: Post Mortem Assessment    18

Figure 9 – Dynamic Workflow Stage Definition    19

Figure 10 – Database Restore    22

Figure 11 – Add Content Database    23

Figure 12 – Deploy Dynamic Workflow Solution    24

Figure 13 – Deploy Workflow Visualization webpart    24

Figure 14 – New Project Web Application    25

Figure 15 – Site Workflow Association    26

Figure 16 – Site Workflow Association    26

Figure 17 – Adding Content DB to IW Demo VM    27

Figure 18 – Provision PWA instance in IW Demo VM    28

Figure 19 – PWA instance under Intranet.Contoso.Com    28

Figure 20 – Create a new SharePoint Enhancement    29

Figure 21 – Initial Requirements form asks the business analyst to enter baseline estimates for the proposal.    30

Figure 22 – Use generic resources to create a baseline estimate of tasks. This will allow you to substitute in actual resources once the execution phase begins.    30

Figure 23 – Creating a resource plan allows you to schedule your resource allocation when planning your project portfolio.    31

Figure 24 – The strategic impact page allows you to rank a project proposal against the company’s business drivers to estimate the gains realized by implementing the project.    31

Figure 25 – Approving the project allows the proposal to move to the selection stage.    32

Figure 26 – Review your projects’ strategic impact rankings.    33

Figure 27 – View your cost constraints for your project portfolio.    34

Figure 28 – View details of resource requirements for each project to see what you’re lacking for each resource/time period.    35

 

References

Microsoft Project 2010 Resources:

Product information

End-User Product Help

Interactive content – Videos & Sessions & Webcasts

Project Professional 2010 and Project 2010 Demo Image:

IT Professional related – TechNet

Developer related – MSDN

Got Questions? Search or ask in the official Microsoft Forums!

SharePoint 2010 Products:

Guide to the Data Development Platform for . NET Developers

Posted on by escapebusinesssolutions in Uncategorized

MSDN Library

.NET Development

Articles and Overviews

Data Access and Storage

ADO.NET

XML for Analysis Specification

Microsoft Data Development Technologies At a Glance

Guide to the Data Development Platform for .NET Developers

Hello, Data

Microsoft Data Development Technologies: Past, Present, and Future

OData by Example

Testability and Entity Framework 4.0


Guide to the Data Development Platform for .
NET Developers


SQL Server Technical Article

Published: November 2009

Applies to: SQL Server 2008

Summary: This whitepaper covers all facets of the .NET data development platform. This includes both client-side and service-based APIs but also .NET APIs for programming at a server level inside the SQL Server 2008 database and for developing and testing a SQL Server database application. It also includes information on future directions of the .NET and SQL Server development platform.

Introduction

The large majority of applications use a database to store, query, and maintain their data. Almost all of them use a relational database that is designed using the principals of data normalization and queried with set-based queries using the SQL query language.  Application programmers need to tie user activities, such as ordering products and browsing through lists of items for sale, to database activities like SELECT statements and stored procedure execution in a visually pleasing and responsive graphical user interface. They need data access and data binding to the user interface that is encapsulated in easy-to-use components that employ the same object-oriented concepts that are used in the rest of the application. To develop responsive, robust applications on-time and on-budget, programmers need to reduce the time from model to implementation, as well as to easily write automated test cases to ensure the application is robust and scalable.

But data usage doesn’t end there.  One of the key challenges today is to make sense of the data in timely manner.  This means applications often add Business Intelligence to the OLTP (online transaction processing system) data that is managed in a relational database for timely decision making within OLTP applications. The OLTP data can be imported, exported, combined with other data and transformed using ETL (extract, transform, and load) technologies, and analyzed using an OLAP (online analytical processing) system. Patterns and insights can be teased from the data using smart data mining algorithms. User interactions can be real-time, but reports serve an important role in data presentation and can be embellished graphically using charts, gauges, maps and other user-interface controls.  Programmers enhancing applications with Business Intelligence should be able to leverage existing experience by using the same basic programming APIs and methodology as they use in application development.

Since the .NET framework 1.0 was released in 2002, it has become programmers’ preferred framework for developing software, including data-driven applications, on Microsoft platforms. .NET 1.0 includes a set of classes known as ADO.NET to provide a substrate for database development. ADO.NET is based on a provider model. Databases products (like SQL Server, Oracle, and DB2) hook into the model by delivering their own data provider. .NET 1.0 shipped with three data providers; bridge providers to existing ODBC drivers and OLE DB providers, and the SqlClient provider for the SQL Server database. Over time, not only has the SqlClient provider been improved to keep pace with innovations in the SQL Server database, but SQL Server has expanded the number of integration points between the database and its related features and the .NET framework. Today .NET APIs are an integral part of the SQL Server product. In this whitepaper, I’ll describe the current state of the Microsoft Data Development Platform and illustrate the deep integration between .NET and SQL Server to show why SQL Server is the preferred database product for .NET developers. The .NET database platform has been enhanced since the advent of .NET to include functionality like object-relational mapping and a programming language integrated query language.  I’ll also discuss the direction that platforms and integration will take in the future so you can plan your overall database development strategy.

Flexible Data Access Frameworks for Productive .NET Application Development

Data Access Frameworks are what developers normally consider to be “database programming”.  .NET developers have a choice of using the low-level ADO.NET object model or the higher-level conceptual model as defined by the ADO.NET Entity Framework. You can also program you client-level database access using through a well-known REST-based services model, exposing the database as a service in the world of Software As A Service. Each of the client-level data access frameworks integrate seamlessly with multi-tier applications and with visual controls in the .NET platform.

ADO.NET – The Substrate

ADO.NET is the base database API and the basis for all of today’s .NET-based data access frameworks. All existing .NET applications in releases prior to .NET 3.5 use this API. Programmers should use ADO.NET when they want direct control of the SQL statements used to communicate with the underlying database and also to access specific database features not supported by the ADO.NET Entity Framework.

Microsoft ships three SQL Server-related ADO.NET data providers. System.Data.SqlClient is the provider for the SQL Server database engine, mentioned previously. Microsoft.Data.SqlCe is a provider for SQL Server Compact Edition that works on the desktop or on compact devices. There is also a specialized data provider for SQL Server Analysis Services and Data Mining functionality (the ADOMD.NET provider), which will be mentioned later.

 ADO.NET uses the “connection-command-resultset” paradigm. Programmers open a Connection to the database, and issue Commands consisting of stored procedures or SQL statements that can either perform insert, update, and delete operations, or select statements that return a DataReader (resultset). Additional classes encapsulate transactions, procedure parameters, and error handling. ADO.NET 1.0 and 1.1 used an interface-based design; that is, programmers used database-specific classes that implemented the well-known IDbConnection, IDbCommand, and IDataReader interfaces. ADO.NET 2.0 added base classes so database-specific classes derived from the generic DbConnection, DbCommand, and DbDataReader classes. The base classes or interfaces provide the generic functionality; database-specific functionality is encapsulated in provider-specific classes (e.g. SqlConnection, SqlCommand, and SqlDataReader). For more information reference “Generic Coding with ADO.NET 2.0 Base Classes and Factories” at http://msdn.microsoft.com/en-us/library/ms379620(VS.80).aspx.

Database transactions are accommodated in the model in two ways. Local transactions can be represented by a DbTransaction object which is specified as a property of the connection object. Alternatively, .NET developers can use the System.Transactions library. System.Transactions supports local and distributed transactions and passes and tracks transactions through an instance of the TransactionScope class.

The specific server, database, and other connection parameters are specified in a connection string. The preferred method for specifying a connection string is to include it in the application’s .NET configuration (.config) file. There is a standard configuration file location used for ADO.NET connection strings.

Programmers use the .NET type system in their applications, but relational data types in the database, so there needs to be a low-level .NET type system-to-database type system correspondence. Database types mostly follow the ISO-ANSI SQL standard type system, but database vendors can include database-specific types. In ADO.NET, common database types are represented using an enumeration for the common types, System.Data.DbTypes. Programmers rely on documented mapping of their database’s data type to DbTypes, but providers can also add database-specific types.  SqlClient accommodates SQL Server-specific types by using a SqlDbTypes enumeration. A type correspondence mismatch occurs because most types in the .NET type system have nothing that distinguishes database NULL from empty instance of a type. A set of SQL Server-specific types in the System.Data.SqlTypes namespace not only encapsulate the concept of database NULL by implementing an INullable interface, but are isomorphic with the data types in SQL Server. .NET 2.0 adds a generic nullable type that works with .NET value types (structures). Nullable types can be used as an alternative to System.Data.SqlTypes.

The SqlClient provider is the lowest-level .NET data access APIs and also the closest to SQL Server. It is the only client database API that supports all of the latest enhancements in SQL Server. Programmers must use native ADO.NET calls when they need access to these features. For example, only ADO.NET and SqlClient directly support the Filestream storage with streaming I/O and the table-valued parameters features of SQL Server 2008. SqlClient is also the only database API to directly support SQL Server data types implemented in .NET, such as SQL Server 2008’s spatial data types and hierarchyid type, as well as SQL Server UDTs (user-defined types) implemented in .NET in the server and on the client. The SQL Server and .NET teams work closely together; an example of this is the introduction a new .NET primitive type, the System.DateTimeOffset data type in .NET 3.5 that corresponds to SQL Server 2008’s datetimeoffset data type. This makes it straightforward for programmers to take advantage of the very latest SQL Server features.

ADO.NET also provides for a “disconnected update” scenario using a class called the DataSet that contains of collection of DataTable instances with collections of DataRow and DataColumn objects. The DataSet interacts with the database through a DataAdapter class, and includes relational database-like features such as indexing, primary keys, defaults, identity columns, and constraints, including referential constraints between DataTables. Programmers can use the DataSet as an in-memory object model for database data, although with the advent of the ADO.NET Entity Framework a model based on entities is preferred because you program against business objects rather than a relational-style object model.

LINQ – Native Data Querying integrated into .Net Languages

LINQ is a SQL-like, strongly-typed, query language that is implemented using built-in programming language constructs in .NET. This provides the ability to query any implementation of a .NET IEnumerable<T> class using SQL syntax. LINQ defines a standard set of query operators that allow both set-based (e.g. projection, selection) and cursor-based (e.g. traversal) queries using programming language syntax. LINQ uses a provider model to allow access to domain-specific stores such as relational database, XML, Windows Active Directory, collections of objects, and ADO.NET DataSets. In most cases, the providers work by translating LINQ queries to queries against the domain-specific store.

These Using LINQ queries that produce SQL query statements gives programmers compile-time syntax checking and IntelliSence, reduces data access coding time compared to coding with raw SQL strings, and also provides protection from SQL injection. This is an improvement for programmers who don’t use stored procedures and code SQL strings directly in application code, because these SQL strings are not syntax-checked by the underlying programming language compiler.

Two implementations of LINQ over ADO.NET were introduced in .NET 3.5. LINQ to SQL is a SQL Server-specific implementation over a collection of tables represented as objects. It includes a mapping tool that maps tables to objects, stored procedure support, and implements almost all LINQ operators by translating them into T-SQL. Programmers can insert, update, and delete rows from database tables through objects using a DataContext class, which extends the basic LINQ functionality.  A similar LINQ provider also exists for the SQL Server Compact Edition. 

Programmers use the ADO.NET DataSet object because it represents its data as a familiar collection of tables that contain columns and rows.  There is also a LINQ provider over the DataSet class that provides functionality not included in the original DataSet classes using standard SQL syntax. This implementation allows selection, projection, joins, and other LINQ operators against the DataSet’s collection of DataTables. Programmers that use the DataSet in existing applications can use the LINQ provider to extend the base functionality. Using LINQ to DataSet enhancements does not change the paradigm that filling the DataSet with database data and updating the database when changes are made to the DataSet is controlled by the DataAdapter class.

ADO.Net Entity Framework and the Entity Data Model Overcome the Object Relational Impedance Mismatch

Object-oriented programming concepts have taken hold in the development community, enjoying the same widespread popularity as the relational normalization-based design and set-based programming has in the database community. Therefore, there is a need to bridge the gap between relational sets to collections of objects and map operations on object instances to changes in the database. Programmers can bridge this gap, known as the object-relational impedance mismatch, by using the ADO.NET Entity Framework.

The ADO.NET Entity Framework should be considered the development API of choice for .NET SQL Server programmers going forward. The Entity Framework raises the abstraction level of data access from logical relational database-based access to conceptual model-based access. For more information on this, reference the whitepaper “Next-Generation Data Access: Making the Conceptual Level Real” at http://msdn.microsoft.com/en-us/library/aa730866(VS.80).aspx.

The Entity Framework consists of five main parts. These are the Entity Data Model (EDM), the EntityClient provider and Entity SQL language, the ObjectServices libraries, and the LINQ to Entities provider.

The Entity Data Model (EDM) specifies the conceptual model that you program against and it’s mapping to underlying database primitives. With EDM, programmers specify the classes and relationships that will be used in the object model (conceptual schema), how tuples and relations are represented in the database (physical schema), and how to map between the two (mapping schema). EDM directly supports table-per-type mapping as well as modeling inheritance using either table-per-hierarchy or table-per-concrete-type. But mapping supports more than just database tables. Mapping entities to stored procedure resultsets and projections as a ComplexType, database views, and on-the-fly projection mapping as anonymous types are also supported. The EDM is specified by using a Visual Studio designer against a set of XML schemas. This metadata can be represented as an XML file (edmx) which can compiled into a program resource.

Entity Framework is layered over ADO.NET and most ADO.NET data providers have been enhanced to work with it. Programmers can work directly with an Entity Data Model using ADO.NET by using the EntityClient provider. To use EntityClient, you specify both the Entity Data Model and the underlying data provider (e.g. SqlClient) in the ADO.NET connection string.

The entity framework defines a language called Entity SQL that extends traditional SQL with some additional object-oriented query constructs.  Entity SQL is the low-level query language of the Entity Framework that is used to query against the conceptual model.  Finally, the Entity Framework includes an implementation of the LINQ query language, known as LINQ to Entities. LINQ to Entities implements almost all of the constructs in Entity SQL.

You can program the Entity Framework using three different programming models: LINQ to Entities, ObjectServices, or “native” ADO.NET using the EntityClient provider. Both EntityClient and ObjectServices use Entity SQL strings directly. Although some lower-level query functionality can only be expressed using EntitySQL, using LINQ to Entities is the overwhelming favorite among programmers that use the Entity Framework. LINQ to Entities is preferred for most development because using LINQ query language instead of EntitySQL gives programmers compile-time syntax checking and IntelliSence, reducing coding time.

The Entity Framework includes a type system that supports Entity Types, Simple Types, Complex Types, and Row Types. Entity Types contain a reference to an instance of the type called an EntityKey. The set of supported Simple Types is a subset of data types found in most relational databases. A complex type is a set of multiple simple types, e.g. a resultset from a stored procedures or a projection from a SQL query. Entities and Associations between entities live in containers known as EntitySets and AssociationSets.  EntityContainer is the top level object that usually maps to a database or application’s scope.

The programming model supports insert, update, and delete operations against the entities by tracking changes with an ObjectContext. The model can generate the SQL required for these operations or they can be mapped to a set of user-written stored procedures. Both lazy and eager loading of related entities is supported. Transactions are accommodated by using System.Transactions.

Entity Framework 4.0, which will ship with .NET 4.0, adds a variety of new features that support a model-first methodology, test-driven development (make it easier to create a mocking layer for testing ), and persistence ignorance. In addition, there are improvements in the SQL generation layer, both in the translation of LINQ to Entities queries and in the SQL Server provider specifically that make the generated SQL more performant. EF 4.0 also directly supports a representation of foreign keys between entities which is useful for data binding.

ADO.NET Data Services – RESTful Data Access

All of the database client APIs that I’ve mentioned so far require a direct connection to the database from either the client or middle-tier. But popular programming models such as Silverlight and AJAX (Asynchronous Javascript and XML) interact with all data sources directly from browser code, using either XML or JSON (JavaScript Object Notation) payloads.  This pattern requires programmers to use a middle-tier service that’s exposed through either SOAP or REST-based APIs. ADO.NET Data Services is the database API that exposes the data as a REST-based Web Service. The service is a WCF-based (Windows Communication Foundation) service that exposes database operations through the standard HTTP operations against a REST endpoint like GET, POST, PUT, and DELETE.

ADO.NET Data Services can use an Entity Framework model as an endpoint directly. It’s also able to work with other LINQ-based data sources (including LINQ to SQL) for reading. If you would like your non- EF data source to be available over ADO.NET data services for updating, you must provide an implementation of the IUpdateable interface.  SQL clauses like the WHERE, GROUP BY, and ORDER BY clause are implemented as parameters on the URL.

ADO.NET Data Services exposes entities (or any sets of data, such as SharePoint lists in SharePoint 2010) as either ATOMPub feeds (ATOM is a standard XML format used for data feeds) or collections of JSON objects. JSON objects are especially useful with AJAX clients. Because security needs to be considered in any service available on the Internet, granular data access security is enabled by using a set of data access expressions coded in the InitializeService method in combination with the authentication and authorization scheme of your choice. By default, no resources or associations are available unless you override the default InitializeService implementation. To build a robust application, ADO.NET Data Services clients may also require metadata; that is, information about the data available through a specific service endpoint.  A REST resource endpoint that supplies metadata for discovery purposes is also part of the model.

In addition to the capability to serve data, ADO.NET Data Services includes a client consumer API. In this API, the client “connects” to the endpoint by using the REST URL. Programmers query the data with a special LINQ provider that fetches updateable EDM object instances. This pattern encapsulates ADO.NET Data Services access so that it looks more like traditional data access code rather than raw HTTP and XML calls.

Programmers can use rich application frameworks that start with ADO.NET Data Services as a REST-based substrate for interacting with a database. One such framework is ASP.NET Dynamic Data, a template-based framework that allows you to build a data driven application quickly. It can use ADO.NET Data Services and LINQ to SQL or Entity Framework to access the relational database. Another framework is .NET RIA Services, which provides a set of components that allow building rich Internet applications (hence the acronym) using common patterns in ASP.NET, service-oriented architecture, and Silverlight 3.0 as a graphic user interface. It includes a pair of service models that hook directly into the ADO.NET Entity Framework, LINQ to SQL, or ADO.NET Data Services. This allows for data to be fetched though a service (a la ASP.NET AJAX) and bound to Silverlight data controls. These are mapped by .NET RIA Services to database maintenance functionality in the service models. .NET RIA Services is currently part of Visual Studio 2010 Beta2.

Harness your .Net skills within the Database for Scalable, Reliable and Manageable Applications

Data access APIs are normally considered as programming outside the database server. A specific database product, such as SQL Server, exposes programming models for in-server programming as well. Beginning with SQL Server 2005, SQL Server permitted in-server programming of database objects such as user-defined functions using .NET APIs as addition to traditional programming in native database languages such as SQL, MDX , and DMX. In addition, SQL Server provides mechanisms to extend and customize the product itself using .NET APIs, and .NET support extends deep into all facets of the server product. With SQL Server as the database, programmers can leverage their existing knowledge of .NET programming to add value to existing applications and also leverage knowledge of .NET APIs such as ADO.NET, which can be used both “outside” and “inside” the server. .NET integration with the database engine includes support for both application programmers and database administrators.

Programming SQL Server with .NET

Since SQL Server 2005, programmers have had the ability to incorporate .NET in database object code as an adjunct and alternative to using T-SQL. Hosting .NET code inside the SQL Server engine (that is, using SQL Server as a .NET runtime host) is commonly referred to as “SQLCLR”. One use of this .NET support is as a means to access external resources such as the registry or file system directly from database code, without resorting to unsafe mechanisms such as enabling xp_cmdshell or using undocumented system extended stored procedures. In addition, the programmer has a choice of .NET or T-SQL when writing user-defined functions, stored procedures and triggers. This allows a choice of best-of-breed mechanisms, that is, the ability to use .NET code where it’s more performant (such as regular expressions and complex calculations) and T-SQL when it’s more performant (database access and complex joins).

The SQL server team designed its .NET integration with security and reliability in mind, as well as performance. Safe SQLCLR code is limited to a subset of system assemblies that have been hardened and tested with SQL Server. These assemblies are guaranteed not to leak memory or other external resources, such as file handles, by the use of such coding primitives as SafeHandle and ConstrainedExecutionRegion introduced in .NET 2.0. Changes to the .NET 2.0 hosting APIs allow .NET resources such as memory allocation, thread pooling and scheduling, and exception handling, to be controlled by and integrated with the rest of the SQL Server engine. SQLCLR code runs in process, as part of the SQL Server service process (sqlservr.exe) for best performance, not in a separate service process.

Data access in SQLCLR database objects follows the ADO.NET object model with optimizations for in-server processing. SQLCLR code can directly execute T-SQL code, rather than requiring a separate ODBC or OLE DB connection (as extended stored procedures require) that use more memory and must be enlisted in the current transaction manually. This is accomplished by using a mechanism to give the appearance of an ordinary SqlConnection instance, but hook up memory pointers for direct access to database buffers, parameters, transaction space, and lock space. The SQLCLR code is part of the current connection. This internal connection is requested by a special connection string parameter “Context Connection=true”.

Server-side programmers use the SqlClient provider and the traditional Connection-Command-DataReader paradigm with a few extensions to accommodate server-specific constructs. This not only lets programmers familiar with ADO.NET leverage their existing expertise, but also allows the production of data access code that’s relatively portable from in-server to middle-tier and even to client should the need arise, with few changes to the code. The additional constructs for server-side code include a SqlContext and DbDataRecord. The SqlContext provides access to the code executor’s identity for access to external resources and a reference to a SqlPipe object. The SqlPipe is used to directly output resultsets or messages.  The DbDataRecord allows the programmer to synthesize resultsets from external resources. You can use this to represent Web Service results or files as a resultset from a stored procedure. In ADO.NET 3.5 and SQL Server 2008 a collection of DbDataRecord objects can also be used to construct table-valued parameters in client code.  Server-side programmers can also write .NET-based table-valued user-defined functions that expose any computed or refactored data as a set, in keeping with the T-SQL set-based paradigm.

There is a well-defined mapping between SQL data types and .NET data types and SQLCLR code must adhere to the mapping to be interoperable with T-SQL code. Although this mapping is mostly the same as client-side data type mapping, some data types (e.g. varchar, the non-Unicode string data type) do not have direct .NET equivalents and these cannot be used in SQLCLR result sets or parameters. In addition, .NET data types that do not have SQL Server equivalents (e.g. System.Collections.Hashtable) cannot be returned to T-SQL code, unless they are transformed into types (such as a collection of DbDataRecord objects returned from a stored procedure as a resultset) that T-SQL can work with.

Server-side programmers can also write SQLCLR user-defined aggregates and user-defined types (UDTs). User-defined aggregates extend the reach of aggregate functions to allow functions that are not built into T-SQL, such as covariant of a population. Programmers can use UDTs to create new scalar data types that extend the SQL Server type system.  UDTs and user-defined aggregates must be coded in SQLCLR, not T-SQL. The execution engine recognizes the new aggregate functions and includes them in the query plan, even allowing the aggregation function to be parallelized as with other query plan iterators.

Extended .Net Data Types and Code in SQL Server

In SQL Server 2008 some new features of the SQL Server engine itself were written in .NET code.  The new Policy Based Management and Change Data Capture features use .NET internally. This indicates that the SQL Server team has made a commitment to use SQLCLR code as an adjunct to unmanaged code where it is useful. Although there is a database option to disallow user-written SQLCLR code, this does not prevent the execution of system CLR code which is always enabled. Thus, in SQL Server 2008, .NET code is used as part of the engine’s standard operating environment.

Three new data types were introduced in SQL Server 2008 that are implemented using the SQLCLR user-defined type architecture; these are the spatial data types, geometry and geography, and the hierarchyid data type. The code that implements these data types in contained in a .NET assembly, Microsoft.SqlTypes.Types.dll. This assembly is required for normal server operation.  One reason for the implementation is that many different operations can be performed on these data types, especially the spatial types and the natural way to expose these is through type-specific methods.  The geometry data type, for example, has over 60 domain-specific methods and many properties that can be manipulated with the SQL-1999 standard “instance.method” syntax in T-SQL, instead of including a library of system functions that apply only to a single data type. This serves to better encapsulate the data type’s functionality.

Implementing built-in data types in .NET code also provides benefits for programmers that use the types. Firstly, the programming model for these types is identical whether you’re writing a SQL query or client-side object manipulation code.  This allows server programmers writing SQL with the instance.method syntax to leverage their expertise when writing either client code or SQLCLR stored procedures. In addition, Microsoft.SqlTypes.Types is distributable as part of the SQL Server Feature Pack, and useable directly on the client-side (in map-drawing applications for example) without a need to connect to the server at all. The server is used for data storage, SQL queries that may include spatial functions, and spatial indexing.

Programmatic Management with SQL Server Management Objects

.NET is used in programming administrative functions in the SQL Server Engine as well through the SQL Server Management Objects (SMO) libraries. SMO is a set of libraries that are used to manipulate and manage database objects (e.g. create a new database, configure network protocols, backup and restore the database). These libraries are used by SQL Server Management Studio (SSMS) Object Explorer and are available to programmers and administrators as well. This provides an additional choice to manage the database, serving as an alternative to SQL DDL statements and system stored procedures. Because they are used to write the SQL Server Management Studio utility, internal Policy Based Management code, and SQL Server Configuration Manager, the SMO libraries are quite comprehensive. These libraries also contain a special Scripter class that can be used to produce a SQL DDL script for any supported database object. This functionality is used extensively in SSMS. The two main functions of the library are DDL type operations and configuration operations in which the library is actually a .NET wrapper around WMI (Windows Management Instrumentation) operations. In addition, there is a set of classes that enable creating and controlling trace objects and a set of classes for creating and managing SQL Server Agent jobs. SQL Server also ships with a .NET library for configuring replication. The RMO (Replicate Management Objects) library can be used as an alternative to the system stored procedures for configuring SQL Server replication.

PowerShell Support in SQL Server

SQL Server 2008 introduced support for PowerShell, a new Windows shell and successor to the command shell. This product is part of the Microsoft Common Engineering Criteria and is supported as an administrative interface by almost every product in the Windows platform. IIS, Exchange, and Active Directory are parts of the platform that currently ship with PowerShell support. Third-party products such as VMWare are beginning to provide PowerShell support as well.

Because systems administrators or programmers often assume database administrator responsibilities in smaller installations and PowerShell is a common administration tool it is thought that “multi-product administrators” will leverage their PowerShell expertise to manage SQL Server, while specialized SQL Server-only DBAs will continue to use SQL scripts and system stored procedures. Experience with .NET object models is a useful skill when working in the PowerShell environment.

PowerShell is based on programming and scripting objects, rather than passing text between commands in scripts as traditional shells (e.g. CMD, csh) do. PowerShell uses cmdlets to manipulate the system; these are programmed in .NET code. In addition, products can expose administrative functionality through a provider model. PowerShell providers assist in generic knowledge transfer by representing system configuration using the file system paradigm combined with a standard set of operations such create-alter-delete on product nodes. SQL Server 2008 includes both cmdlets and a provider. SQL Server ships a custom shell (named SQLPS.exe) in which the cmdlets and the provider are pre-installed to prevent naming conflicts. You can also manually install the cmdlets and provider for use in the main PowerShell shell.

The provider exposes a few directories under the SQLSERVER: virtual directory.

  • SQL  – The SQL database engine objects (servers, databases, tables)
  • SQLPolicy – SQL Server policy-based management
  • SQLRegistration – Server registrations in SSMS
  • DataCollection –  Data collection for the Management Data Warehouse

   In SQL Server 2008 R2, two additional subdirectories are added

  • SQLUtility –The SQL Server Utility (Enterprise Multi-Server Management)
  • SQLDAC –DAC (Data-tier application databases)

There are only a few cmdlets in SQL Server 2008, including cmdlets for mapping between SQL Server database object names and PowerShell object names, a cmdlet with similar functionality to SQLCMD and cmdlets for executing policy-based management functions. Because the provider uses SMO objects to represent database objects internally, programming in PowerShell with SQL Server consists of quite a bit of navigation to the appropriate place in the provider virtual directory and then using SMO objects to perform operations.

The ability to use the SSMS Object Explorer context menu to bring up a PowerShell window at the appropriate position in the database object provider hierarchy as well as the ability to invoke PowerShell script as a SQL Agent job step round out the SQL Server 2008 PowerShell support.

Enrich your Applications with Embedded Business Intelligence

Business Intelligence is a set of tools and technologies designed to improve business decision making and planning. It generally includes reporting, quantitative analysis, and predictive modeling. Programmers can use their .NET skills along with tools in SQL Server in the programming of decision support systems.

.NET APIs are not only integrated with the database engine, but are pervasive in every part of the SQL Server product. .NET is used for four main Business Intelligence functionality groups:

  • Programming Business Intelligence client applications
  • Application programming at the server product level
  • Administration and Configuration programming
  • Writing product extensions

In this section, I’ll briefly describe the .NET facilities used by SQL Server Reporting Services (SSRS), SQL Server Analysis Services (SSAS), and SQL Server Integration Services (SSIS).

Integrate Reports & Data Visualization within your application with SQL Server Reporting Services

SQL Server Reporting Services is the tool of choice for programming reports to display in custom Web or Windows applications using the ReportViewer control , or publication in a central repository for convenient, secure, user access. Programmers can use SQL Server or Microsoft Office SharePoint Server as a repository or include report definitions directory in applications.  All of the entry points into the SSRS product allow programmers to leverage their .NET coding experience.

The report definition language (RDL) is a XML document format based on a published XML schema. SSRS reports are usually designed and published using Business Intelligence Development Studio (BIDS) or the graphic Report Builder tool. Reports can use Data Sources as input (SQL Server and a variety of other data sources are supported) and work with tables, views, and stored procedure data directly, or Report Models that expose metadata and relationships in a more user-friendly form. Report Models are designed and programmed in Visual Studio and can be generated against a SQL Server or Oracle database or an Analysis Services database.

The Report Manager is an ASP.NET-based application that allows administrators to manage reports, common data source defintions, and subscriptions, and also provides the ability for users to view reports in HTML format. Reports can be rendered in a variety of formats and will even be exposed as data (AtomPub feeds) in SQL Server 2008 R2. 

There is not a custom object model for building RDL, but because RDL is XML-based the .NET XML APIs can be used to built raw RDL with built-in schema validation. Programmers can integrate reports into their own applications or manage reports using a Web Service interface, as the functionality is exposed using Web Services. The Report Viewer control can use URL access or Web Service calls to access published reports, or use reports in local RDL files.

Reports can use custom code that extends the functionality provided by the SSRS built-in expression language. This custom code must be provided in .NET assemblies, which are registered with Report Manager and in the development environment. Once registered, custom .NET classes are accessed just as the built-in expressions are.

Administrators can administer the server programmatically by either using the Report Manager Web Service APIs or by writing scripts that are executed using the rs.exe utility. Administrative scripts for the rs.exe utility are programmed using Visual Basic .NET scripting language.

Reporting Services uses a componentized architecture and, as such, allows user-written extensions. These extensions use SSRS-specific .NET objects models. Some of the components that can be extended with custom .NET code are: data processing extensions, delivery extensions, rendering extensions, and security extensions.  The object models are provided in two .NET libraries, Microsoft.ReportingServices.DataProcessing and Microsoft.ReportingServices.Interfaces that are shipped with SQL Server.

ADOMD.NET – Client side Analysis Services

Programmers that expose Business Intelligence functionality, such as information from OLAP cubes in custom .NET applications must use an API to extract the information from the OLAP server just as they use an API to extract information from the relational engine.  Although many Business Intelligence users use packaged client applications such as Excel, writing custom web applications that extract analysis and data mining information, such as the familiar “people who brought product X are also interested in product Y” is becoming more and more commonplace. ADOMD.NET is to OLAP applications as ADO.NET is to relational database applications.

ADOMD.NET is a specialized ADO.NET data provider that not only exposes the traditional Connection-Command-DataReader paradigm, but contains additions for multidimensional data and data mining. With OLAP cubes, metadata is not only more complex (it can also be hierarchical in the case of hierarchical dimensions) but also more important, because OLAP client applications often rely on discovery through metadata at runtime. ADOMD.NET exposes this metadata in three different ways: through provider-specific properties (e.g. Cubes and MiningModelCollection properties) on the Connection object, through an extensive set of SchemaRowsets (accessed through the GetSchemaRowset method as with traditional ADO.NET providers), and through an ADOMD.NET-specific object model devoted to metadata, rooted at the CubeDef class. Because analysis queries can be long-running, the ADOMD.NET provider supports multiplexing Connection instances through a Session class. Programmers use ADOMD.NET on the client by adding a program reference to the Microsoft.AnalysisServices.AdomdClient.dll.

Multidemensional resultsets are also different than rectangular relational resultsets. These resultsets can have multiple axes as opposed to the two-axis column and row relational rowsets. The ADOMD.NET provider supports using the AdomdCommand class to execute MDX (MultiDimensionalExtensions) or DMX (DataMiningExtensions) language commands. These commands can return a two-dimensional AdomdDataReader or a multidimensional CellSet. The CellSet is shaped like a cube rather than a table, and programmers access data consisting of CellCollections and retrieve axis information using the Axes property. You can also use a DataAdapter to get cells in the form of a DataSet. The provider is currently read-only but supports composing multiple operations in a read-committed level transaction.

ADOMD.NET ships with SQL Server or is available separately as part of the SQL Server 2005-2008 Feature Pack.

ADOMD.NET – Server side Analysis Services

In SSAS you can write stored procedures or user-defined functions in .NET using the ADOMD.NET library. The server library has almost the exact same object model as the client library but slightly different functionality. To write in-server .NET code, you reference Microsoft.AnalysisServices.AdomdServer.dll.

User-defined functions are used in the context of either an MDX or DMX statement, as part of the statement itself.  Stored procedures are executed stand-alone with the MDX or DMX CALL statement. Both UDFs and stored procedures can take parameters, but UDFs can return a variety of data types including instances of the ADOMD.NET Set object. Stored procedures can only return sets, as IDataReader or DataSet. A stored procedure can also have no return value. The reason for using ADOMD.NET with in-server objects is to decrease network latency (one single network call as opposed to possibly many network calls from the client) and to encapsulate data logic for reuse.

In server-side ADOMD.NET, you do not get a Connection object in order to access the data. Instead, you start by obtaining a Context object that represents your current connection. The Context object does not expose any schema rowsets, you must use the object model to access metadata. A UDF is created for either MDX or DMX and cannot mix functionality in a single module. Therefore, depending on the calling context, either a CurrentCube or CurrentMiningModel property is available from the context object. ADOMD.NET functions and stored procedures are read-only. They do not support writeback and therefore do not support transactions.

Analysis Management Objects

AMO is an object model for creating, altering, and managing Analysis Services cubes and Data Mining models. It is analogous to SMO in the database engine. In addition to a comprehensive set of classes to manipulate metadata, AMO contains classes that allows the .NET administrator to automate backup and restore of Analysis Services databases,  a Trace class for monitoring, replay, and management of SSAS profiler traces, and CaptureLog and CaptureXML classes for scripting capturing and scripting operations performed by SSAS SMO statements. Once again, administrators with the ability the write .NET code can leverage their expertise with AMO. SSAS also supports an XML-based scripting language for management known as ASSL.

Microsoft does not ship a formal PowerShell provider or cmdlets for SSAS. However, you can use AMO classes with PowerShell in an analogous way as the SMO classes are used. In addition, a PowerShell snapin including a provider and cmdlets called PowerSSAS is available on the CodePlex website at http://www.codeplex.com/powerSSAS .

Integrated Your Data with Ease with SQL Server Integration Services

SSIS is an extract-transform-load (ETL) tool that is used to build database workflows called packages. SSIS packages are designed for fast data transformation and loading using an in-memory “data pipeline”, often using the bulk extract-bulk insert functionality of databases directly. SSIS allows programmers to concentrate on writing workflow logic rather than building custom code for each ETL scenario. Programmers often use SSIS packages to populate Analysis Services cubes and train Data Mining models, as well as import and export of relational data.

The SSIS data flow engine is not written in .NET code, but an ADO.NET provider can be used as either as source or destination of data. SSIS packages are usually created in the Business Intelligence Development Studio (BIDS) environment, but you can also create them programmatically through a .NET object model.  The entire SSIS package model is available in .NET, rooted in a class named Package.  You can also manage and execute packages programmatically using the classes in Microsoft.SqlServer.Dts.Runtime.dll. Management classes allow you to run packages locally or on a remote machine, load the output of a local package, load, store, and organize packages into folders, and enumerate existing packages stored in the SQL Server database or file system. SQL Server Agent is usually used to run the packages on a schedule or an ad-hoc basis.

SSIS packages consist of control flows and data flows. Control flows contain tasks, with special enumerator components to allow the execution of a task over a collection of items (e.g. executing an FTP task over a collection of files). Data flows can contain source providers, destination providers, and transformation components. Data source connection information for the providers (e.g. connection strings to databases) is managed by Connection Manager components. Components log information to an informational and error log.

SSIS ships with a rich set of ETL components. It is interesting to note that each SSIS component that is part of the product is implemented in a separate assembly, that is, they are implemented in .NET. Therefore programmers with .NET expertise can use SSIS object models to create custom components that extend the base functionality of implement SSIS integration with ETL items that are not supported by SSIS “in the box”. These components include not only an execution portion but also a designer portion so that the custom component can be used to design packages in BIDS. There’s a .NET object model to enable programmers to build:

·         Custom tasks

·         Custom connection managers

·         Custom log providers   

·         Custom enumerators

·         Custom data flow components – source, destination, or transformation components

Note that you cannot write custom components that derive from system components.

Finally, SSIS ships with script components that allow custom logic in control flows or data flows. This allows script-based lightweight access for customizations without having to build an entire custom component. SSIS includes both a Script Task for task flow and a Script Component for the data flow. In SQL Server 2008, these scripts use Visual Studio Tools for Applications (VSTA) and the script code can be written in either VB.NET or C# (only VB.NET is supported in SQL Server 2005).  Once again, the .NET programmer or administrator should be right at home with SSIS script coding and component customizations.

Sync Services for ADO.NET enables collaboration and offline scenarios for applications, services and devices

As compact devices get more prolific, a common pattern in data access emerges in which data is stored on in a corporate database, with many individual devices needing replication/synchronization of their-specific portion of the database tables. An example would be salespeople that need a local copy of their customers’ data for offline use while on the road. The salesperson also needs to be able to take orders offline then upload changes to the corporate database. For this type of scenario, ADO.NET Sync Services fills the bill nicely, allowing client-directed synchronization between databases such as SQL Server and SQL Server Compact Edition. Sync Services for ADO.NET is part of the Microsoft Sync Framework, which also supports file system sync and feed sync (for RSS and ATOM feeds).

Sync Services works with a provider model and ships with a SyncClient provider for SQL Server Compact Edition. The SyncServer provider works with any ADO.NET data provider. Sync Services support unidirectional synchronization, bidirectional synchronization, or complete refresh with each synchronization. Version 1.0 supports hub-and-spoke style synchronization, Version 2.0 also supports peer-to-peer style synchronization.

The Server provider works with the set of synchronization commands, but also allowed for programmable conflict resolution if the data in the server database changes but there is also a change on the client. SQL Server 2008 includes a Change Tracking feature that allows distinguishing which changes are made on client or server, and allows selective synchronization on a per-client basis. Sync Services for ADO.NET is available for SQL Server Compact Edition on the desktop and on compact devices.

Microsoft Visual Studio Team System 2008 Database Edition

Regardless of which features that programmers use to interact with the database, they need a strong development environment that includes testing, profiling, version control, and software workflow capabilities. Visual Studio Team System (VSTS) is Microsoft’s flagship software development lifecycle product. In addition to application and service development, database programmers and administrators can also use to develop, test, and deploy database objects and schemas using the same software development lifecycle they are already familiar with. VSTS Database Edition includes SQL Server 2000-2008 support and provides a provider mechanism for other database support as well. In addition to the built-in functionality there are customization points for .NET programmers who wish to extend the product.

VSTS Database Edition unit tests are encapsulated in .NET modules and a .NET extension API is available to customize the tests. You’d do this when you need to test for special conditions that aren’t covered by the graphic user interface. Programmers also use VSTS Database Edition to generate test data. A standard set of test data generators are provided, but you can extend this with custom generators, using the classes in Microsoft.VisualStudio.TeamSystem.Data.Generators.dll.

The latest version of the product (VSTS 2008 Database Edition GDR) includes database object refactoring and static code analysis. You can extend the facilities of both of these features by writing custom .NET code.  Custom database refactoring code (e.g. SQL code casing) and custom refactoring targets (e.g. text files) are supported by using a framework of classes in the VSTS Database Edition interop DLLs. Static code analysis rules can be plugged in to the existing rule infrastructure by using Microsoft.Data.Schema.ScriptDom.dll.

Finally, VSTS Database Edition ships with a SQL parser that developers can use to parse and validate T-SQL and generate T-SQL scripts. The parser is not only used by the product but available as a component for programmer use. For further information and an example of the parser’s use, reference http://blogs.msdn.com/gertd/archive/2008/08/21/getting-to-the-crown-jewels.aspx .

The Future Frontier

So far, we’ve covered how you can use .NET as a substrate to develop with databases outside the server, and with SQL Server inside the server product as well. In future products, the vast amount of functionality available to .NET data access programmers as well the synergy between SQL Server and .NET continues to increase.  Here’s a survey of some of the features for .NET programmers coming in the near future.

SQL Azure Database – Database in the Cloud

SQL Azure Database is a cloud-based database built on SQL Server. Because it uses SQL Server underneath, any database API, including Entity Framework, ADO.NET, and others, can be used with SQL Azure Database simply by changing the connection string. Version 1.0 of SQL Azure Database supports almost all of the database objects (e.g. tables, views, and stored procedures) in the SQL Server database engine, but Version 1.0 does not yet support in-database .NET programming (SQLCLR).

Self-Service Business Intelligence

SQL Server PowerPivot For Excel is an Excel 2010 add-in that facilitates the creation of analysis models in Excel using multiple data sources, and also allows the user to define relationships between tables in disparate data sources. This enables an entire population of non-programmers to write Excel apps against their corporate data in a way they never could before, enabling new scenarios in Business Intelligence. The data in the models can be set to automatically update on a schedule and the Excel-based model can be published to a SharePoint server farm. Using SQL Server PowerPivot for SharePoint 2010, the SQL Server PowerPivot Excel-based data can be exposed as an Analysis Services 2010 cube data source. SQL Server PowerPivot For Excel will support ATOMPub data feeds, meaning that any data source that’s exposed through ADO.NET Data Services will be available. This allows SQL Server PowerPivot For Excel to import data exposes through EDM object models, as well as SharePoint 2010 List data. In addition, SSRS 2008 R2 will expose any report that’s published to the SSRS Report Manager as an ATOMPub feed, permitting the creator of a SQL Server PowerPivot For Excel data model to import report data for further analysis.

SQL Server Master Data Services– Manage Your Most Important Data Assets

SQL Server Master Data Services is a feature of SQL Server 2008 R2 that allows a business to manage its most vital and precious data such as customer and product lists. Often multiple copies of this data are stored in different databases and it’s easy to end up with duplicate but slightly different versions of information such as addresses. MDS allows you to resolve such data inconsistencies. For more information about Master Data Management concepts, reference the whitepaper “The What, Why, and How of Master Data Management” at http://msdn.microsoft.com/en-us/library/bb190163.aspx.

SQL Server MDS includes a SQL Server-based repository, a Web-based front end that allows domain experts and user analysts to manage the data, and a Web Service-based interface for programmability. The feature implements fuzzy matching and domain-specific indexes. The fuzzy indexing algorithms were developed by Microsoft Research.

The programming functionality is included as a set of SQLCLR system functions (e.g. similarity and regular expression style functions) and index building and lookup procedures for multicolumn domain-specific indexes, implemented in the MDM repository database. Because they are written in SQLCLR these functions could be leveraged in future in other parts of the SQL Server product such as SSIS (for bulk matching and data cleansing), SSRS (to produce fuzzy match reports) and IFTS (Integrated Fulltext Search) as an adjunct and alternative to word-stemming based text matching.

Real-time Applications with Data in Flight (StreamInsight)

In most data use cases, data is first stored in a database and analyzed after it’s stored. However, with some use cases (e.g. power grid or heart monitor readings, stock ticker prices) the act of storing the data introduces too much latency for the analysis to be useful. StreamInsight is a Complex Event Processing (CEP) system that allows data to be analyzed and events to be summarized “in flight”.

The StreamInsight produce in built on a service process and input and output providers. Input providers (written in .NET using the StreamInsight APIs) translate input data into a standard set of input stream messages. The streaming applications themselves are also written using a .NET API. Each message type can be registered with one or more LINQ queries (StreamInsight supplies the LINQ provider) for in-flight querying, message aggregation and output through message output providers. Input and output providers for SQL Server data are among those included with the current samples.

Simplify Data Development with Modeling

Modeling with M, EDM, and the Entity Framework

Programmers can be more productive when the transition from model to application is part of the development process. Future versions of SQL Server will support several features to facilitate this process, including a programming language to create models (“M” language) and SQL Server Modeling Services, a repository for storing, querying, and programming with models. The current CTP includes “M” to T-SQL generation for defining and populating models  and the Quadrant tool for creating and compiling “M”-based models and DSLs (domain-specific languages), and browsing and editing the repository.

One near-term usage of this product will be using the “M” language as an eventual replacement of XML-based EDMX model files as part of the generation of the Entity Framework mapping model and code. This will allow greater model scalability compared to using EDMX. Programmers can either use the EDM graphic designer in Visual Studio or code their Entity Data Model in “M”, and the model will be used by the Entity Framework directly at runtime.

In addition to Entity Framework 4.0, which supports the major of object programming patterns used today, Entity Framework will also be more closely integrated with the SQL Server product in the future. A most likely first step is to allow programmers to design reports in SQL Server Reporting Services directly against the conceptual model. The conceptual model could be integrated with other models in the framework as well.

SQL Server Modeling Services Repository

The SQL Server Modeling Services repository resides in a SQL Server database, so it can be populated and queried with either the “M” or T-SQL languages. SQL Server-specific facilities such as Change Data Capture can be used to provide versioning of the repository.  By using the “M” language, programmers can model not only the data and object models, but also applications, workflows and business processes, services, configuration information, and the operating system infrastructure such as SMS (Microsoft Systems Management Service) configuration and installation information. By storing the information is a single repository, it’s possible to define relationships among domain-specific models for a complete picture of business and computer-based processes. SQL Server Modeling Services ships with a Base Domain Library (BDL) to provide a service layer to model-driven applications, including pre-built domains for UML and CLR and tools that support them.

The storage of data and application models in the SQL Server Modeling Services Repository, along with code generation to make repository information synergistic with data applications, will assist in recording and using models to make SQL Server Modeling Services the provider of a single, integrated, auditable, version of truth for the organization’s data models, application service models, configuration information, software management, and business processes.

Conclusion

As you can see programmers use .NET as the underlying substrate for extending and customizing all parts of the SQL Server family including deep support in SQL Server Analysis Services, Integration Services, and Reporting Services. In future SQL Server will evolve into a database that not only integrates with the relational model of data but provides direct support of the conceptual model as well.

The current .NET data access APIs are always based on a provider model that defines a base functionality with extensibility points that allow database-specific functionality to be fit into the model. A SQL Server implementation of each client API is able to serve as a reference implementation and also as a model to illustrate provider extensibility. SQL Server carries the data models into the database itself, enabling in-database .NET programming and extensibility.  

For more information:

http://www.microsoft.com/sqlserver/ : SQL Server Web site

http://technet.microsoft.com/en-us/sqlserver/ : SQL Server TechCenter

http://msdn.microsoft.com/en-us/sqlserver/ : SQL Server DevCenter 

Real World Migration Story from BizTalk 2006/2006R2 to BizTalk 2010

Posted on by escapebusinesssolutions in Uncategorized

Real World Migration Story from BizTalk 2006/2006R2 to BizTalk 2010

 

Writer:

Srikiran Tadikonda, Microsoft

Banupriya Thirumaran, Microsoft

 

Reviewed By:

Puneet Gupta, Microsoft

Shridhar Nadgaundi, Microsoft

Scott Zimmerman, Microsoft

Mandi Ohlinger, Microsoft

 

Contributors:

Gopinath P, Microsoft

Prethish Kumar Puthenparampil, Microsoft

Rachna Srivastava, Microsoft

Rajesh Ramamirtham, Microsoft

Sumant Kumar Pandey, Microsoft

Sneha Aswani, Microsoft

 

Published: January 2013

Applies to: BizTalk Server

Summary: The BizTalk Server infrastructure used by Microsoft IT (MSIT) included BizTalk Server 2006 R2 and BizTalk Server 2006 environments. Until recently, the platforms that hosted these applications were not Disaster Recovery enabled. Since BizTalk Server 2006 doesn’t support the Applicability Statement 2 (AS2) protocol, AS2 transactions are sent to BizTalk Server 2006 R2. Then, the transactions are routed back to BizTalk Server 2006 for further processing. As a result, there are many cross-server dependencies. With the transactions passed through multiple BizTalk servers, the architecture of these applications was complex. The integration team wanted to remove these cross-server dependencies and simplify the application architecture.

To simplify the architecture, the BizTalk Server 2006 R2/2006 environments are migrated to BizTalk Server 2010. The objectives of the migration are:

  • Enable Disaster Recovery for business critical applications.
  • Reduce the number of servers required to deploy the applications and reduce the support costs for the servers.
  • Minimize the number of pipelines deployed by leveraging the Pipeline Consolidation Framework of BizTalk Server 2010.
  • Enable MSIT to support a broader range of internal applications.
  • Stay current with the technologies: Windows Server 2008 R2, SQL Server 2008 R2, Visual Studio 2010, and the .NET Framework 4.

 

The migration to BizTalk Server 2010 is a great success, including the following results:

  • Reduced the physical servers for 32 applications from 51 servers to 22 servers
  • Uptime is greater than 99.9% and about 400,000 messages are processed per week
  • Utilize the benefits of Hyper-V
  • Zero major issues post-production

 

This white paper discusses the steps taken to migrate to BizTalk Server 2010.

Copyright

 

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

Some examples depicted herein are provided for illustration only and are fictitious.  No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

© 2013 Microsoft. All rights reserved.

 

 

 

Contents

Introduction    5

Introduction to BizTalk Server 2010    5

New Features of BizTalk Server 2010    5

Easier Integration of Enterprise Applications    5

Platform Support for Latest Microsoft Technologies    6

Simplify Solution Manageability for IT Pros    6

Enhanced Enterprise Interoperability    7

Deprecated Features of BizTalk Server 2010    7

Common Reasons for Migrating to BizTalk Server 2010    7

Microsoft IT (MSIT) BizTalk 2010 Migration Story    8

Problem Statement    8

Objectives    8

Migration Process    8

Development Strategy    8

Architecture – Before and After Migration    9

New Features used in Migration    10

Testing Strategy    11

Benefits of Migration    13

Challenges faced in Migration    14

Connect Bug: AS2 encryption and signing issue details    14

Host instance and hosts going down intermittently    14

BizTalk 2010 is stopping when input file is more than 5 MB    15

Best Practices    15

Appendix    16

Volume/ Throughput after migration    16

Weekly Volumes on BizTalk Servers    16

Patches applied to fix issues in BizTalk 2010    17

Conclusion    18

Acknowledgements    18

 

 

Introduction

This white paper describes the following:

  • Motivations for moving to BizTalk Server 2010
  • The process that Microsoft IT went through to migrate 32 mission-critical applications from BizTalk 2006/2006 R2 to BizTalk 2010
  • Suggestions for smooth administration of BizTalk Server 2010.

 

The new BizTalk installation went into production in phases from February 2011 to November 2011. Since then, uptime is greater than 99.9% while processing about 400,000 messages per week. This migration project allowed Microsoft IT to reduce the physical servers for the 32 applications from 51 to 22 servers, which includes eight 4-proc servers running Hyper-V to host BizTalk Server 2010. There have been zero major issues during post-production support. In many respects, the project has been a great success.

Introduction to BizTalk Server 2010

In most organizations today, there is a backlog of requirements to integrate different IT systems. Yet connecting software means more than just exchanging bytes. As organizations continue to move toward a service-oriented world, the real goal—creating effective business processes that unite disparate systems into a coherent whole—is now within reach. Microsoft BizTalk Server 2010 supports this goal.

Microsoft BizTalk Server is an integration server and connectivity solution. It helps organizations meet the challenge of integrating diverse systems. Including over 25 multiplatform adapters and a robust messaging infrastructure, BizTalk Server provides connectivity between core systems inside and outside of your organization. In addition to integration functionality, BizTalk also provides strong durable messaging, a rules engine, EDI connectivity, Enterprise Service Bus, Business Activity Monitoring (BAM), RFID capabilities, industry-standard SWIFT and HL7 support, and IBM Host/Mainframe connectivity.

New Features of BizTalk Server 2010

BizTalk Server 2010 introduces enhancements and new features in four main areas.

Easier Integration of Enterprise Applications

BizTalk 2010 accelerates time to solution with better and more accessible tools:

  • BizTalk Server 2010 includes easy to use tools that integrate Line-of-Business applications with Windows Server AppFabric Workflows. This includes Workflow activities that provide access to the BizTalk Transformation Engine and Mapper, and the BizTalk Line-of-Business adapters.
  • The BizTalk 2010 Mapper has been updated with a new user interface that improves development of large transformations and provides new search and predictive matching functionality. The new mapper also improves productivity by adding cut, copy, paste, move, and undo functions and stronger support for documenting maps and readability.
  • BizTalk Services are now accessible via SharePoint 2010 Business Connectivity Services.
  • BizTalk RFID includes new event handlers and event delivery capabilities. The new event handlers enable removal of duplicate events and filters based on Electronic Product Codes (EPC) and visibility requirements. The new delivery capabilities send events directly to .NET applications, BizTalk Server, and EPC Information Services (EPCIS).
  • Easier access to IBM applications and data sources. BizTalk 2010 Adapter for Host Systems offers efficient development tools and native runtime protocol engines to connect BizTalk 2010 to existing line-of-business programs and data on IBM mainframe systems.
  • Excellent support for industry-standard web services. BizTalk 2010 SOAP adapter enables sending and receiving messages by using SOAP over HTTP, thus enabling it to interact with the universe of web services. BizTalk 2010 also includes seven adapters and easy-to-use “wizards” that enable communicate to and from BizTalk Server and Web services-based applications via the Windows Communication Foundation (WCF). 

Platform Support

for Latest Microsoft Technologies

BizTalk Server 2010 supports the Microsoft platform technologies, including Windows Server 2008 R2, SQL Server 2008 R2, Visual Studio 2010, and the .NET Framework 4.

  • BizTalk Server 2010 support for Windows Server 2008 R2 offers modular, minimal installation, improved network performance and control. It also includes improved high availability features, enhanced administration, and management with Server Manager, Windows PowerShell command-line, and enhanced virtualization with Hyper-V. By utilizing Windows Server 2008 R2 clustering, BizTalk Server can be deployed in multisite cluster scenarios, where cluster nodes can reside on separate IP subnets and avoid complicated VLANs.
  • BizTalk Server 2010 takes advantage of the virtualization improvements included as part of Windows Server 2008 R2 Hyper-V. These improvements lead to reduced costs through lower hardware, energy, and management overhead; plus the creation of a more dynamic IT infrastructure.
  • BizTalk Server 2010 support for SQL Server 2008 and SQL Server 2008 R2 offers better manageability and scalability, an optimized virtual SQL Server implementation and improved performance, especially in a 64-bit.
  • Support for Visual Studio 2010 and NET Framework (v4) introduces a number of improvements to the underlying Visual Studio-based BizTalk project system. Improvements include debugging support for artifacts and BizTalk maps (XSLT), enhanced BizTalk artifact property pages integrated into Project Designer tabs, the Visual Studio Project Update Wizard, and support for release and debug builds.

Simplify Solution Manageability for IT Pros

With BizTalk Server 2010 it’s now much easier to get things done faster.

 

  • A new settings dashboard consolidates all settings into a single place, providing granular settings at the group, host, or host instance level. The Settings Dashboard supports automated settings deployment with scriptable export/import operations.
  • The new System Center Operation Manager Pack includes a new model with separate application and deployment views, new alerts and diagnostics, and improved multimachine monitoring and cluster awareness.
  • Enhancements to the setup and upgrade include a new Smart Setup which scans and installs BizTalk upgrades automatically, upgrading from BizTalk Server 2006 R2 and BizTalk Server 2009.

Enhanced Enterprise Interoperability

BizTalk Server 2010 has better options for connecting with business partners and applications.

 

  • Enhanced Trading Partner Management provides easy on-boarding and lifecycle management of trading partners. A new partner management model reflects typical B2B relationships and is easier to manage large-scale TPM deployments. An improved user interface includes agreement templates and business profiles for rapid configuration and a new TPM operator role which provides access to users other than administrators.
  • New and enhanced adapters provide higher levels of security, better performance, and support for the latest releases of line-of-business applications.

Deprecated Features of BizTalk Server 2010

The following features and tools are available in previous BizTalk versions and are replaced with new features in BizTalk Server 2010.

  • BizTalk Web Services Publishing Wizard: This wizard is replaced with the new BizTalk WCF Publishing Wizard, which provides broader and more flexible support to publish web services from BizTalk.
  • BizTalk Adapter Pack 2010 version (BAP 2010): The SAP adapter, Oracle adapter, SQL Server adapter and Siebel adapter in BAP 2010, which comes with BizTalk 2010 replaces the corresponding adapters in BAP 1.0, which comes with BizTalk 2006/2006 R2.
  • Add Web Reference: The Consume WCF Service wizard is used to consume web services from BizTalk.
  • EDI Pipeline and Context Properties: The new Trading Partner Management feature in BizTalk Server 2010 provides the same capabilities as the EDI Pipeline and Context Properties and replaces these features of older BizTalk Server Versions.
  • Web Services Enhancements Adapter: This adapter is replaced with the WCF Adapter using a custom transport.

Common Reasons for Migrating to BizTalk Server 2010

  • Stay Current: Being able to use newer communication protocols makes it easy to stay current and take advantage of new software features and technologies, such as improved encryption.
  • Support: BizTalk Server 2006/R2, Visual Studio 2005, and SQL Server 2005 are no longer covered by Microsoft’s Mainstream Support Policy. To keep active with support for mission-critical IT assets, it is essential to upgrade to the latest versions.
  • Virtualization: BizTalk 2010 takes advantage of the latest virtualization improvements of Windows Server 2008 R2 Hyper-V. The improvements can lead to reduced costs through lower hardware, energy, and management overhead and in the creation of a more dynamic IT infrastructure.
  • Clustering: By using Windows Server 2008 R2 clustering, BizTalk Server can now be deployed in multisite cluster scenarios, where cluster nodes can reside on separate IP subnets and avoid complicated VLANs.

Microsoft IT (MSIT) BizTalk 2010 Migration Story

Problem Statement

The Microsoft Board of Directors mandates that Microsoft’s internal business critical applications, like Volume licensing and Order Processing, must have Business Continuity/Disaster Recovery (BC/DR). Until recently, the platforms hosting these applications (that is BizTalk 2006/2006 R2) were not DR enabled. Since BizTalk Server 2006 doesn’t support the AS2 (Applicability Statement 2) protocol, AS2 transactions went to BizTalk Server 2006 R2. Then, the transactions were routed back to 2006 for further processing. This led to many cross-server dependencies. With the transactions being passed through multiple BizTalk servers, the architecture of these applications was complex. The integration team wanted to remove these cross-server dependencies and simplify the application architecture.

Objectives

The objectives of this migration were:

  • To enable Disaster Recovery for business critical applications.
  • To reduce the number of servers required to deploy the applications and to reduce the support costs for them.
  • To minimize the number of pipelines deployed (by leveraging the Pipeline Consolidation Framework of BizTalk 2010).
  • To enable MSIT to support a broader range of internal applications.
  • To stay current with the technologies: Windows Server 2008 R2, SQL Server 2008 R2, Visual Studio 2010, and the .NET Framework 4.

Migration Process

Development Strategy

The migration process started with estimating the number of maps for the applications, frequency of transactions per LOB (Line of Business), size of the messages and so on; and then setting these values as the production benchmark. Based on the estimation, and the criticality of the applications and the business needs, the integration team divided the entire migration process into four phases.

In each phase, a set of applications were migrated from BizTalk 2006/2006R2 to BizTalk 2010. Once the applications were determined for each phase, the old BizTalk solutions were rebuilt using Visual Studio 2010. BizTalk solutions that contained legacy adapters like SAP and SQL are replaced with WCF-Custom adapters in BizTalk 2010. Due to the growing number of pipeline and pipeline components, an internal framework called Pipeline Consolidation Framework (PCF) was developed. The framework offers flexibility for pipeline configuration, minimizes the number of pipelines and pipeline components, centralizes the storage in the .NET global assembly cache (GAC) and makes the pipelines reusable with more shared functionality. Refer to this (http://blogs.msdn.com/b/srikiran/archive/2012/12/22/pipeline-consolidation-framework-pcf-biztalk-2010.aspx) blog for more information on PCF. Changes to the existing solutions are done in configuring adapters, pipelines, and party settings. In other artifacts like maps, schemas and orchestrations no changes were required.

The final solutions were built and deployed in System Integration Test (SIT), User Acceptance Test (UAT), and Production environments. This entire migration process took about 15 months for 32 applications. This migration process started with less critical applications like HR and Supply Chain applications, then moved to highly critical applications like Finance, and Order to Cash in later phases.

Architecture – Before and After Migration

Previously, there were two versions of BizTalk (BizTalk 2006, 2006R2) in which all applications were deployed (see the following diagram). This architecture was previously supported by 22 Production servers. This architecture led to many problems. See Figure1. For example, when one application was updated, it used to affect the other applications on the same server. Any patches applied to BizTalk 2006/2006R2 affected all the applications. To overcome these major problems, the integration team came up with the new, appropriate architecture. In the new architecture, three BizTalk 2010 LOB Servers are maintained. The applications which are high-critical are deployed on first server, medium-critical on second server and low-critical on third server. When changes are made to low-critical applications, it does not impact the high-, or medium-critical applications on the other BizTalk servers and vice versa. The maintenance of the applications is greatly simplified. The alerts on the host instances are set according to the criticality of the applications, which helps the business owners handle any issues effectively. See Figure2 for each LOB server configuration.

 

            Figure 1. Old and New Architecture

 

 


Figure 2. BizTalk 2010 LOB Server Configuration

New Features used in Migration

Several BizTalk 2010 new features are used to help development productivity and solve some of the challenges faced with BizTalk 2006 and BizTalk 2006 R2:

  • Using the BizTalk 2006/2006R2 Trading Partner Management (TPM), the application could not use multiple AS2 transaction setups with AS2 (different AS2 Id with single MDN receive endpoint) for the same business partner. In BizTalk 2010, transaction setup is not only at the Party level but also at the Agreement level. So now, there is fine-grained control over Transaction Setup within the Agreement Level.
  • The BizTalk Mapper improvements reduced the time from 16 hours to develop complex maps to 4 hours. The key mapper improvements were:
    • Organizing maps in multiple pages
    • Ability to search in Schemas
    • Predictive Matching
    • Cut/Copy/Paste       
  • BAM User Experience Improvements – There were no indexed properties in earlier versions of BAM tracking. BTS2010 has indexed based sorting. The indexed-based sorting resolves a performance issue when returning the most recent tracking records.
  • SQL and SAP Adapters are migrated to WCF adapters to give a uniform experience when configuring the settings, and more importantly, improved performance.

     

Testing Strategy

Three months of test data was captured from the archives from the production systems. This data was later cleaned up and categorized for testing. The business partners for testing were selected based on criteria, including production volume, encoding, location, and so on. The testing scenarios cover new changes, regression of the as-is functionality, data comparison, and negative testing. 

Test is organized into four main phases.

Isolated Environment Testing:

In isolated environment testing, the BizTalk 2010 solution is tested with five or six partners per transaction. Individual BizTalk artifacts were tested including adapters, maps, schemas, receive locations and send ports. And the output for each transaction in BizTalk 2010 is compared with the old environment.

Volume testing was also done and ensured that the outputs between old and new translators were same.  

System Integration Testing:

For system integration testing, the BizTalk 2010 system is connected with internal tools and downstream systems. Partner testing is simulated here. For inbound transaction, HTTP posts to the gateway system are simulated. For outbound transactions, transactions are sent to local AS2 sites to verify the correctness of the output. 

User Acceptance and Rollback Testing:

For user acceptance and roll-back testing, there is collaboration with the partners at the start of the release so their environments are available during UAT testing.

Test certificates are used and shared with partners during this testing. The UAT test scenarios include positive and negative scenarios agreed with the partners. Scenarios include missing mandatory fields in purchase orders sent by partners, or incorrect EDI qualifiers or delimiters in TPM Agreement settings. Another criteria for successful UAT testing and signoff is that there is a minimum of five files tested with each partner.

Rollback testing happened in UAT. Scenarios were created depending on where the rollback happens. For example, different rollback procedures in gateway applications compared to BizTalk application-level procedures are used. The system was rolled back to the old environment and tested with one file for each transaction to verify that files were flowing through the old environment as expected.

Once the rollback tests were successful, again the automated deployment to switch to the new environment was initiated. This is followed by smoke testing the new applications in BizTalk 2010. This level of rollback testing obviously takes time, but it’s the best way to ensure that the project team is ready to make the switch, without any disruption to the business.

Performance Testing:

For Performance Testing, benchmark statistics are collected from BizTalk 2006 and BizTalk 2006 R2 production systems, covering a period of more than six months. The metrics included number of messages per transaction, size of the messages, peak volume, and frequency of the messages.

Using LoadGen (a simulation tool to measure the impact of clients on server), configuration sets are created for different adapters such as File, HTTP, WCF, and so on. The appropriate performance counters for a BizTalk clustered environment and SQL were selected and started before running LoadGen scenarios. For this project, the PAL (Performance Analysis of Logs) tool is used to analyze the performance counter logs and thresholds. It also documents the system, server, and network issues and presents it in HTML format.

The extreme scalability tests include increasing the message count and the size of the messages to manifold. In some cases, an increase up to 60 times more than the actual volume and 40 times more than the actual size of messages in production are used. See the following Figure 3; which shows the different scenarios. The team was satisfied that the measured CPU rates demonstrated the ability to handle peak load far beyond the requirements. A breakpoint scenario is also tested to ensure the stability and availability of the server even under high CPU utilization. It was seen that the files continued to be processed, even in the breakpoint scenario where there was more than 95% CPU utilization across the servers.

Figure 3. The CPU Utilization of BizTalk 2010 LOB1 during extreme load testing.

Preproduction Validation Testing:

The production deployment scripts, assemblies, and artifacts were validated and smoke tested before the Go-Live event. As the environments were new, there were network and firewall issues when connecting to external customer URLs and internal business systems such as SAP. So, the transactions are smoke tested with local end points and customer UAT URLs to ensure that the firewall issues or connectivity issues were overcome. After installing the BizTalk solutions and before the new receive ports and orchestrations were enabled, Build Validation Tests (BVT) are performed in the production environment. The BVTs verify that all the necessary artifacts are present.

Post Production Support:

Using System Center Operations Manager (SCOM) 2010, we monitored the following events:

  • Receive Location and Send port availability
  • BizTalk to BizTalk server services
  • Suspending Q monitoring
  • DB availability
  • Any disconnection between Database and processing nodes

Monitoring was automated. Alerts are sent for failed messages and SLAs were created based on the criticality of the applications. The 24/7 support team from Operations and Engineering ensures that any critical issue is resolved based on the SLA.

The post production support for applications is extended for a month from the date of Go-Live. Any suspended messages in BizTalk or any issues raised by customers are attended immediately with the 24/7 support model. There have been no major issues during the post-production support period.

Benefits of Migration

The migration of several applications from BizTalk 2006/2006R2 to BizTalk 2010 achieved many significant benefits for Microsoft.

  • The Integration Team successfully deployed 32 unique applications that have 150+ project artifacts (maps) and 300+ partners to production with zero severity-1 bugs or issues.
  • It resulted in the retirement of 51 servers, 70% of them being physical servers. This is a large cost savings.
  • There was no adverse impact on any business processes, and customers are using new applications without any issues.
  • It resulted in a reduction of the pipelines from 108 to 18, which significantly increased the performance of the applications.
  • After migration, the applications are able to take advantage of state-of-the-art virtualization technologies with Hyper-V, which leads to reduced costs through lower hardware, less management overhead and more flexible administration.
  • The migration has the applications on a newer platform, including Windows Server 2008 R2, SQL Server 2008 R2, and .NET Framework 4.0 which helped to leverage the dynamic features of all these new technologies.
  • BizTalk 2010 takes advantage of N-node Active-Active clustering of Windows Server 2008 which improves the availability and utilization of the resources for the applications. BizTalk 2006 R2 uses 2-Node Active-Passive clustering in Windows Server 2003.

Challenges faced in Migration

There were some challenges faced during the migration of applications to BizTalk 2010. The most important of them are listed here.

Connect Bug: AS2 encryption and signing issue details

 

Issue:

A signed and encrypted message from partner fails to be processed in BizTalk 2010 Beta, and was getting processed in BizTalk 2006 R2. A partner is able to send signed and encrypted message but the AS2 Decoder fails to decrypt the file and returns the following error

:“An error occurred when decrypting an AS2 message.
The sequence number of the suspended message is 1. “
Additional error generated in the event log whenever this error occurs is:
“The AS2 Decoder encountered an exception during processing. Details of the
message and exception are as follows: AS2-From :”<<Some Value>>” AS2-To:”<<
Some Value>>” MessageID:”<< Some Value>>” Message Type: “unknown”
Exception: “An error occurred when decrypting an AS2 message””

    The more detailed exception is available here (http://connect.microsoft.com/BizTalk/feedback/details/564504/error-while-decrypting-as2-files-as2-over-file-scenario)

 

Fix:

This issue was caused by code issue in BizTalk 2010 beta version and fixed in the BizTalk 2010 RTM version.

 

Host instance and hosts going down intermittently

Issue:

When performing transactions on BizTalk 2010, some of the following errors occurred:

  • Error code: 0xC0002A21, an error occurred while attempting to access the SSO database.
  • A transport-level error has occurred when receiving results from the server. (Provider: TCP Provider, error: 0 – The semaphore timeout period has expired.) SQL Error code: 0x00000079

Fix:

To fix this problem, follow these steps:

BizTalk 2010 is stopping when input file is more than 5 MB

Issue:

When you install BizTalk 2010 in Windows 7, Windows 7 Service Pack 1 (SP1), Windows Server 2008 R2 or Windows Server 2008 R2 Service Pack 1 (SP1) and use AS2 decoder to decrypt an encrypted message larger than 5 MB, it fails with an “out of memory error while decrypting files larger than 5MB” exception.

Fix:

To resolve this issue, apply the hotfix described in Microsoft Knowledge Base (KB) article 2480994 (http://support.microsoft.com/kb/2480994)

Best Practices

The following are some of the best practices used to help the migration process go smoothly:

  • Ensure consistent monitoring and tracking of messages is present across all applications.
  • Avoid overloading individual hosts. Instead, use the appropriate number of BizTalk 2010 Servers and decide how many applications to deploy on each of them in the design phase.
  • Validate your message volume per host and modify default settings as appropriate.
  • Ensure that the BizTalk SQL Agents Jobs run successfully in SQL Server to release all the resources.
  • Ensure that you have a backup and replication strategy in place.
  • When possible, avoid large schemas with mostly optional fields.
  • Ensure that appropriate permissions exist for SSO, SQL, and BizTalk.
  • Modify default retries and timeouts as appropriate.

 

The optimizations performed by the Integration team to improve the performance of applications in the migration process include:

  • Utilized the SQL Server Backup Compression feature in BizTalk Server which eventually reduced the size of BizTalk backups and the time consumed for backing up the Database.
  • In BizTalk Server 2010 Management Pack for System Center 2010, set the discovery interval to lower levels which fixed the performance issues related to resource utilization in previous BizTalk versions.
  • To achieve higher performance, the Max receive interval is tuned at the host Level in BizTalk 2010. Performance was further improved by tuning the Max polling interval and Orchestration polling interval. Refer to this (http://blogs.msdn.com/b/prethishkumar/archive/2012/02/04/high-performing-biztalk-server-2009-biztalk-server-2010-server-setup.aspx) blog for more information on improving performance.
  • Upgraded the hardware (for example, increasing the RAM and disk space) which reduced disk I/O Latency.

 

Appendix

Volume/ Throughput after migration

The following table and graph show the typical weekly number of messages that flow through the new BizTalk Servers.

Weekly Volumes on BizTalk Servers

 

BizTalk Server

Weekly Volume

BizTalk 2010 LOB1

326,534

BizTalk 2010 LOB2

36,467

BizTalk 2010 LOB3

4,062

 

 

 

Patches applied to fix issues in BizTalk 2010

Certain patches that are applied to BizTalk 2010 to fix issues based on customer feedback received by the Microsoft BizTalk team. In the following table, some of the patches that helped the integration team in migrating the applications from BizTalk 2006/ 2006R2 to BizTalk 2010 are listed.

KB

Description

2570450

FIX: “Unable to sign outbound message” error after you upgrade to BizTalk Server 2006 R2 SP1 or to BizTalk Server 2010 (http://support.microsoft.com/kb/2570450)

2587237

FIX: Host instance of BizTalk Server 2010 does not resume operation after the connection for a BizTalk Management database resumes (http://support.microsoft.com/kb/2587237)

2563979

FIX: Performance decreases when a server that is running BizTalk Server processes a large batch of messages after you customize the settings of the “Large Message Threshold” option and of the “Large Message Fragment size” option (http://support.microsoft.com/kb/2563979)

2555003

FIX: AS2 messages remain active after they are processed when the “Track Message Bodies – Request message after port processing” option is enabled on a send port in a BizTalk Server environment (http://support.microsoft.com/kb/2555003)

2580202

XML Ex operation processes EDI messages in BizTalk Server 2010 if the separator is set to the less than sign (<) or to 0x3C (http://support.microsoft.com/kb/2580202)

2627804

FIX: A hotfix that improves the performance of the tracking feature in BizTalk Server 2010 is available (http://support.microsoft.com/kb/2627804)

2563231

FIX: “The formatter threw an exception while trying to deserialize the message” error when the WCF-SAP adapter in BizTalk Adapter Pack executes an RFC or BAPI to an SAP system (http://support.microsoft.com/kb/2563231)

2552332

FIX: WCF-SAP adapter may incorrectly keep connections open when the adapter sends some RFC requests (http://support.microsoft.com/kb/2552332)

2539412

A hotfix is available for the WCF-based SAP adapter to let you disable the behavior that trims leading zeros of NUMC values in BizTalk Adapter Pack (http://support.microsoft.com/kb/2539412)

2539769

FIX: BizTalk Server stops receiving IDOCs from a WCF-based SAP adapter in BizTalk Adapter Pack (http://support.microsoft.com/kb/2539769)

2556304

FIX: Remote function calls do not work after the WCF-based SAP adapter receives the RFC_FAILURE error code in BizTalk Adapter Pack (http://support.microsoft.com/kb/2556304)

2554806

FIX: “System.InvalidOperationException: Category does not exist” error if some registry keys for the WCF-based SAP adapter are corrupted in BizTalk Adapter Pack (http://support.microsoft.com/kb/2554806)

2598894

FIX: Error occurs or NULL values incorrectly inserted by the WCF-based SQL adapter in BizTalk Adapter Pack 2010 if an input message contains empty elements (http://support.microsoft.com/kb/2598894)

2478956

WCF-SQL Adapter does not return useful errors (http://support.microsoft.com/kb/2478956)

2481676

WCF-SQL adapter sometimes locks sql spid/table after PDAS stopping all operations against database (http://support.microsoft.com/kb/2481676)

2522459

Poll Statement is executed even if PolledDataAvailableStatement returns 0 (http://support.microsoft.com/kb/2522459)

2530219

BAP SAP Keep Context for WCF-SAP (http://support.microsoft.com/kb/2530219)

2505757

[Port from BAP 2010 CU1 to BAP 2.0 CU2] New WCF based SAP Adapter does not sort the string correctly in the String scenario with multiple grouped IDocs (Collated String option fix) (http://support.microsoft.com/kb/2505757)

899599

A BizTalk Server Host instance fails, and a “General Network” error is written to the Application log when the BizTalk Server-based server processes a high volume of documents (http://support.microsoft.com/kb/899599)

970406

TCP settings that can impact BizTalk Server (http://support.microsoft.com/kb/970406)

 

Conclusion

BizTalk 2010 is implemented by many organizations worldwide today for their mission-critical integration. BizTalk 2010 is for high volume, high availability, and high scalability which are required for business process solutions. Migrating to BizTalk 2010 from BizTalk 2006, BizTalk 2006 R2 gives many benefits to the organization and to the customers.

Acknowledgements

Many thanks for the technical information and input provided by Gopinath P, Prethish Kumar Puthenparampil, Rachna Srivastava, Rajesh Ramamirtham, Sumant Kumar Pandey and Sneha Aswani. We would like to acknowledge the leadership and support provided by Puneet Gupta and Shridhar Nadgaundi. A special thanks to Scott Zimmerman and Mandi Ohlinger for the technical review and valuable insights on this white paper.

For more information:

http://www.microsoft.com/sqlserver/: SQL Server Web site

http://technet.microsoft.com/en-us/sqlserver/: SQL Server TechCenter

http://msdn.microsoft.com/en-us/sqlserver/: SQL Server DevCenter

 

Did this paper help you? Please give us your feedback. Tell us on a scale of 1 (poor) to 5 (excellent), how would you rate this paper and why have you given it this rating? For example:

  • Are you rating it high due to having good examples, excellent screen shots, clear writing, or another reason?
  • Are you rating it low due to poor examples, fuzzy screen shots, or unclear writing?

This feedback will help us improve the quality of white papers we release.

Send feedback.

Active Directory LDAP Compliance

Posted on by escapebusinesssolutions in Uncategorized

 

 

 

Active Directory LDAP Compliance

 

Microsoft Corporation

Published: October 2003

 

 

 

 

Abstract

Directories are public or private stores containing essential identifying information typically used in daily enterprise activities. Many application providers capitalize on directories offering integration into existing directories to extend their application’s functionality. Network operating systems also house vital network information, such as users and computers, within directories.

Lightweight Directory Access Protocol (LDAP) is a directory standard founded on the legacy X.500 directory. LDAP’s initial implementations provided gateway services between X.500 directory servers and clients. While LDAP was initially created to meet this requirement, it became clear that a parting from the cumbersome X.500 directory standard was needed to simplify deployments. In 1994, LDAP was transformed into a directory specification with its own database and structuring conventions.

This paper discusses the origins of LDAP within Microsoft products and, specifically, the implementation of, and conformance to, the LDAPv3 Proposed Standard within Microsoft Windows 2000 Server and Microsoft Windows Server 2003. Included for reference are matrixes detailing supported RFCs.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

© 2003 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Visual Basic, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

 

 

Contents

Introduction    2

Directory Foundation: X.500    2

X.500: The Need for a Lightweight Alternative    2

What Is LDAP?    3

LDAP: First Generation    3

Enhancements with Version 2    3

The Current State of LDAP    3

What Does It Mean to Be LDAP Compliant?    5

Achieving Compliance: IETF Applicability Statement    5

Achieving Compliance: Third-Party Test Suites    5

The Open Group LDAP Certifications    5

Setting an LDAP Compliance Baseline    6

Active Directory’s LDAP Compliance    8

Windows 2000 Server    8

Windows Server 2003    8

Compliance Misconceptions    10

inetOrgPerson    10

Native LDAP Calls    10

Directory Interoperability    11

LDAP API    11

Active Directory Services Interface    11

Development Environments    12

Active Directory Application Mode    12

Directory Services Markup Language    12

Microsoft Identity Integration Server 2003, Enterprise Edition    12

Additional Resources    14

Lightweight Directory Access Protocol Version 3    14

Open Group and the Directory Interoperability Forum    14

Developing with Active Directory Services Interface    14

Miscellaneous    14

Introduction

Directories—public or private resource lists containing names, locations, and other identifying information—are essential tools often taken for granted in our daily activities. Typically these directories provide information about people, places, or organizations as part of an overall solution. For example, a telephone is virtually useless without a directory to correspond names with telephone numbers. Historically, most directories were only available in printed form.

As the computer revolution forged ahead, printed directories gave way to an electronic counterpart. Many application providers capitalized on the directory concept offering proprietary versions that extended their application’s functionality. Network operating systems also provided directories, typically housing user and device information. Unfortunately, these first generation directories were often developed with little or no concern for interoperability. Isolated and specific in function, they performed admirably. However, it was obvious directories needed to interact within a larger network ecosystem. This idea grew into the definition of the X.500 standard.

Directory Foundation: X.500

In 1988, the International Organization for Standardization (ISO) and the International Telecommunications Union (ITU) introduced the X.500 standard. X.500 defines the protocols and the information model for an application and network platform agnostic directory service. As a distributed directory based on hierarchically named information objects, X.500 specifications characterized a directory that users and applications could browse or search.

The X.500 paradigm includes one or more Directory System Agents (DSAs)—directory servers—with each holding a portion of the Directory Information Base (DIB). The DIB contains named information objects assembled in a tree structure—defined by a Directory Information Tree (DIT)—with each entry having an associated set of attributes. Every attribute has a pre-defined type and one or more associated values. Object classes, containing mandatory and optional attributes, are defined within a directory schema. End users communicate with an X.500 DSA using the Directory Access Protocol (DAP) while the Directory System Protocol (DSP) controls interaction between two or more DSAs.

X.500: The Need for a Lightweight Alternative

Understanding the need for a streamlined directory standard, several implementers proposed a lightweight alternative for connecting to X.500 directories. Ultimately, the first iteration of LDAP gained traction as a simple alternative to the X.500 Directory User Agent (DUA). The new LDAP definition:

  • Simplified protocol encoding
  • Used text encoding for names and attributes
  • Mapped directly onto the TCP/IP stack
  • Supplied a simple Application Programming Interface (API)

What Is LDAP?

Organized development of LDAP occurred on several fronts. However, the most notable work, and the first freely available implementation, was completed by the University of Michigan in 1993. The University focused efforts on developing a simpler TCP/IP version of X.500’s DAP. DAP was considered cumbersome as it pushed much of its workload to the client.

Although LDAP is well rooted as a simplified component of the X.500 directory, it has become the de facto directory protocol on the Internet today.

LDAP: First Generation

LDAP’s initial implementations provided gateway services between X.500 directory servers and clients. The clients communicated with an LDAP gateway through LDAP-enabled software. In turn, the gateway handled transactions—on behalf of the client—with the X.500 DSA. This model promoted directory interoperability allowing application providers to easily develop client software capable of communicating with an LDAP gateway service, regardless of the backend platform. While LDAP was initially created to meet this requirement, it became clear that a parting from X.500 was needed to simplify deployments. In 1994, LDAP was transformed into a directory specification with its own database and structuring conventions.

Once transformed, the LDAP specifications reflected a true client-server model with clients making requests directly to servers for information or operations. One or more directory servers may either perform the operation or refer the client to another directory server that may be able to provide the requested information, or perform the requested operation. The LDAP client will see the same view of the directory no matter which server is contacted. If necessary, the LDAP server can authenticate the client to the operating system in use. Once received, the LDAP server will convert a request into an appropriate format for the accessed directory. For X.500 directories, the LDAP server would convert the LDAP request into a DAP request.

Enhancements with Version 2

As interest in LDAP increased, several new developments extended its core functionality while streamlining its footprint. In 1995, Request for Comment (RFC) 1777 was introduced for LDAP Version 2. RFC 1777 eliminated many of the impracticable components of X.500 that were central to the original LDAP specifications. Furthermore, network connectivity was changed from the X.500 Open Standards Intercommunication (OSI) model to the TCP/IP model.

LDAPv2 is officially defined by the following RFCs:

  • RFC
    1777     – Lightweight Directory Access Protocol (v2)
  • RFC
    1778     – The String Representation of Standard Attribute Syntaxes
  • RFC
    1779     – A String Representation of Distinguished Names

The Current State of LDAP

Developed by the Internet Engineering Task Force (IETF) in 1997, the current LDAPv3 implementation is a renovation of LDAPv2, which primarily tackles deployment limitations identified within the previous version. LDAPv3 also enriches compatibility with X.500 along with enhanced integration with non-X.500 directories. LDAPv3 encompasses LDAPv2 within a new set of RFCs.

LDAPv3 is a Proposed Standard currently defined by RFC 3377, which includes, the RFCs below in addition to support for the LDAPv2 RFCs listed above:

  • RFC 2251 – Lightweight Directory Access Protocol (v3)
  • RFC 2252 – LDAP (v3): Attribute Syntax Definitions
  • RFC 2253 – LDAP (v3): UTF-8 String Representation of Distinguished Names
  • RFC 2254 – String Representation of LDAP Search Filters
  • RFC 2255 – The LDAP URL Format
  • RFC 2256 – A Summary of X.500(96) User Schema for use with LDAP (v3)
  • RFC 2829 – Authentication Methods for LDAP
  • RFC 2830 – LDAP (v3): Extensions for Transport Layer Security

What Does It Mean to Be LDAP Compliant?

Claiming LDAP compliance seems to be a common occurrence among today’s directory-capable vendors. In truth, applications may be either directory-aware—capable of reading an LDAP directory—or directory-enabled—capable of reading and performing other defined LDAP operations on a directory. Both implementations should be considered LDAP-compliant if proposed standards are followed in achieving an application’s desired level of LDAP functionality. Compliance with standards, however, does not ensure interoperability. Standards may lack sufficient clarity, even after formalization, leading to varying guideline interpretations.

The compliance task for directory server vendors weighs on their ability to conform to defined standards while ensuring interoperability with those standards. In addition, the process of standard formalization is an ongoing effort compounding the difficulty of achieving full compliance. For example, LDAPv2 has been formalized with a defined set of RFCs, however, LDAPv3, a Proposed Standard, is theoretically still a work in progress as it moves toward an Internet Standard. In summary, a vendor’s compliance statement should be viewed as their ability to implement a known set of RFCs, accurately interpret those RFCs to ensure interoperability, and provide a framework capable of incorporating new RFCs.

Achieving Compliance: IETF Applicability Statement

The IETF has established an LDAPv3 Working Group—ldapbis—with the charter of steering the previously mentioned LDAPv3 RFCs through the Internet Standards process. In September 2002, the working group submitted a revised LDAP specification for consideration as a draft standard. Currently, no IETF LDAPv3 conformance or compliance directive exists; however, one of the most pervasive documents to be delivered by the working group is an LDAP applicability statement.

Once complete, an LDAP applicability statement will guide independent vendors toward IETF-based LDAP compliance. In all probability, third-party test suites, based on the applicability statement, will be developed, allowing vendors and end users to accurately determine LDAP compliance. Until then, vendor declarations combined with existing third-party testing suites are the most suitable alternatives.

Achieving Compliance: Third-Party Test Suites

Several LDAP compliance testing and certification solutions exist in the marketplace today. One of the most prominent test suites is offered by the Open Group, a technology-neutral consortium. The Open Group also sponsors the Directory Interoperability Forum (DIF), which compromises architects, strategists, and product managers from customers and vendors of directories and directory-enabled applications. According to the Open Group, the DIF is chartered to “provide testing and certification for directory servers and applications that use Open Directory Standards.” Microsoft Corporation is an active member of the DIF.

The Open Group LDAP Certifications

Recently released in 2003 are two directory Open Group/DIF test certifications: LDAP Ready and LDAP Certified. With an LDAP Certified directory, the vendor guarantees its LDAP directory server conforms to the LDAPv3 specifications published by the IETF. On the other hand, an LDAP Ready application warrants that it will interoperate with an LDAP Certified directory.

To become LDAP Certified, directory vendors typically submit a warranty of LDAPv3 conformance using the Open Group’s VSLDAP Test Suite as a compliance measuring tool. The warranty is intended to ensure conformity to industry standard specifications, conformity through a product’s lifecycle, and the quick remedy of any nonconformity. At the moment, no vendor has received either of these new certifications.

Setting an LDAP Compliance Baseline

According to the DIF, a baseline of LDAP compliance can be measured by verifying support of the following RFC components. By no means exhaustive, an LDAP compliance baseline can be supplemented by support of additional LDAP RFCs, features, or extensions.

RFC 

Feature 

Description 

2251 

Abandon 

The server accepts abandon requests and performs the requested abandon operations. 

2251 

Add 

The server accepts add requests and performs the requested add operations.

2251 

Anonymous Bind over SSL

The server accepts a simple bind over SSL where the password is null and treats the client as anonymously authenticated. 

2251 

Authenticated Simple Bind

The server accepts a simple bind request with a password and authenticates the client by that password.

2251 

BER 

The server correctly encodes and decodes protocol elements using ASN.1 BER as required by LDAP. 

2251

Client Server Communication

The client transmits an operation request to the server, which performs the operation on the directory and returns results/errors.

2251 

Compare 

The server accepts compare requests and performs the requested compare operations. 

2251 

Data Model 

Each entry must have an objectClass attribute. The attribute specifies the object classes of an entry, which along with system and user schema determine permitted attributes of entries.

2251 

Delete 

The server accepts delete requests and performs the requested delete operations. 

2251 

Modify (Add, Delete, Replace)

The server accepts modify requests and performs the operations, including additions, deletions, and replacements.

2251 

ModifyDN – Move Leaf Entry to New Parent 

The server accepts modify DN requests to move leaf entries to new parents and performs the leaf move operations.

2251 

ModifyDN – Move Renamed Leaf Entry to New Parent

The server accepts modify DN requests to rename leaf entries and move them to new parents and performs the requested leaf rename and move operations. 

2251 

ModifyDN – Rename a Leaf Entry

The server accepts modify DN requests to rename leaf entries and performs the requested leaf rename operations.

2251 

Search 

The server accepts search requests and performs the requested search operations. 

2251 

Simple Bind with Password over SSL

The server accepts a simple bind request over SSL with a password and authenticates the client by that password.

2251 

Simple Common Elements

The server correctly encodes, decodes, and processes the simple common elements of LDAPMessage envelope PDUs. 

2251

TCP as the Transporting Protocol

A server implements mapping of LDAP over TCP and provides a protocol listener IP port 389. 

2251

2252 

Definition of Attributes

The server’s entries have attributes in accordance with the X.500 model. The server supports entries with attributes.

2251

2252 

Definition of Object Classes

The server associates entries with object classes in accordance with the X.500 model. 

2251

2253 

Distinguished Name 

The server correctly encodes and decodes protocol representations of distinguished names.

2251

2829 

Anonymous Simple Bind

The server accepts a simple bind request where the password is null and treats the client as anonymously authenticated. 

2252 

Definition of Matching Rules

The server supports matching rules in accordance with the X.500 model.

2253 

Parsing 

The server correctly parses string representations of distinguished names. 

2253 

Relationship with LDAP v2

The server accepts but does not generate certain protocol constructs that are legal in LDAP v2 but not in LDAP v3. 

2253 

Relative Distinguished Name

The server correctly encodes and decodes protocol representations of relative distinguished names. 

2829

2830 

SSL over TCP as the Transporting Protocol 

A server implements mapping of LDAP over SSL over TCP and provides a protocol listener IP port 636.

 

 

Active Directory’s LDAP Compliance

Windows 2000 Server

With the release of Windows 2000 Server, Microsoft made its first introduction of Active Directory®directory service. With a foundation stemming from proven Microsoft technologies, Active Directory’s origins lie within the original 1996 release of Microsoft Exchange Server 4.0. Active Directory, however, was a significant paradigm shift from the familiar Windows NT 4.0 identity store. By envisioning a directory capable of extending beyond a typical NOS store, Microsoft’s first implementation of Active Directory became a viable repository for many forms of enterprise data.

To allow Active Directory to gain popularity as an enterprise repository, Microsoft had to make an early commitment to LDAP. The Windows 2000 implementation of Active Directory is an LDAP-compliant directory supporting the core LDAP RFCs available at the time of its release. Currently, Windows 2000 Active Directory reaches LDAP compliance through support of the following RFCs.

RFC 

Core LDAP RFCs 

RFC Status 

2251

 

Lightweight Directory Access Protocol (v3)

 

Proposed 

2252

Lightweight Directory Access Protocol (v3): Attribute Syntax Definitions 

Proposed 

2253 

Lightweight Directory Access Protocol (v3): UTF-8 String Representation of Distinguished Names 

Proposed 

2254 

The String Representation of LDAP Search Filters 

Proposed 

2255 

The LDAP URL Format

Proposed 

2256 

A Summary of the X.500(96) User Schema for use with LDAPv3 

Proposed 

2829 

Authentication Methods for LDAP 

Proposed 

2830 

Lightweight Directory Access Protocol (v3): Extension for Transport Layer Security 

Proposed 

RFC 

Additional LDAP RFC Support 

RFC Status 

2696 

LDAP Control Extension for Simple Paged Results Manipulation 

Proposed 

2247 

Using Domains in LDAP/X.500 Distinguished Names 

Proposed 

 

Windows Server 2003

Building on the foundation established in Windows 2000 Server, the Active Directory service in Windows Server 2003 extends beyond the baseline of LDAP compliance into one of the most comprehensive directory servers offering a wide range of LDAP support. Accordingly, the Windows Server 2003 Active Directory service introduces a number of new LDAP capabilities targeted for IT professionals and application developers. Some of the latest LDAP features include:

  • Dynamic Entries – Active Directory can store dynamic entries allowing the directory to assign Time-To-Live (TTL) values to determine automatic entry deletion.
  • Transport Layer Security (TLS) – Connections to Active Directory over LDAP can now be protected using the TLS security protocol.
  • Digest Authentication Mechanism – Connections to Active Directory over LDAP can now be authenticated using the DIGEST-MD5 Simple Authentication and Security Layer (SASL) authentication mechanism. The Windows Digest Security Support Provider (SSP) provides an interface for using Digest Authentication as an SASL mechanism.
  • Virtual List Views (VLV) – When an LDAP query has a large result set, it is inefficient for a client application to pull down the entire set from the server. VLV allows a client application to “window” through a large result set without having to transfer the entire set from the server.
  • Concurrent LDAP Binds – Although not specifically defined by the IETF, Concurrent LDAP Bind provides the ability to perform multiple LDAP binds on one connection for the purpose of authenticating users.

Although the LDAP compliance guidelines proposed by the Directory Interoperability Forum signify conformance at a base level, organizations demand directory servers capable of providing robust network and application services. This is one of the driving forces behind Microsoft’s support of virtually all IETF- recognized LDAP components. Although LDAP compliance should always be considered a work in progress until full IETF standardization, Microsoft’s current LDAP compliance in Windows Server 2003 includes support of the following RFCs.

RFC 

Core LDAP Requirements – RFC 3377 

RFC Status 

Timeline 

2251 

 

Lightweight Directory Access Protocol (v3)

 

Proposed 

Windows 2000 

2252 

Lightweight Directory Access Protocol (v3): Attribute Syntax Definitions 

Proposed 

Windows 2000 

2253 

Lightweight Directory Access Protocol (v3): UTF-8 String Representation of Distinguished Names 

Proposed 

Windows 2000 

2254 

The String Representation of LDAP Search Filters 

Proposed 

Windows 2000 

2255 

The LDAP URL Format

Proposed 

Windows 2000 

2256 

A Summary of the X.500(96) User Schema for use with LDAPv3 

Proposed 

Windows 2000 

2829 

Authentication Methods for LDAP 

Proposed 

Windows 2000 

2830 

Lightweight Directory Access Protocol (v3): Extension for Transport Layer Security 

Proposed 

Windows 2000 

RFC 

Additional LDAP RFC Support 

RFC Status 

Timeline 

2696 

LDAP Control Extension for Simple Paged Results Manipulation 

Proposed 

Windows 2000 

2247 

Using Domains in LDAP/X.500 Distinguished Names 

Proposed 

Windows 2000 

2589 

LDAP Protocol (v3): Extensions for Dynamic Directory Services 

Proposed 

Windows Server 2003 

2798 

Definition of the inetOrgPerson LDAP Object Class

Proposed 

Windows Server 2003 

2831 

Using Digest Authentication as an SASL Mechanism 

Proposed 

Windows Server 2003 

2891 

LDAP Control Extension for Server Side Sorting of Search Results 

Proposed 

Windows Server 2003 

 

Compliance Misconceptions

Since the release of Windows 2000 Server Active Directory, many assertions have been made about its LDAPv3 compliance. As noted previously, Active Directory and Microsoft Exchange Server have supported a baseline of LDAP compliance since inception. Furthermore, Active Directory extends this baseline by supporting a majority of the LDAP Informational and Proposed RFCs. Although Microsoft’s commitment to building one of the most comprehensive and interoperable directories in the marketplace is obvious, there are still common compliance misconceptions worth discussing.

inetOrgPerson

One of the most widely publicized Windows 2000 Server LDAP non-compliance claims pivots on the support of RFC 2798 and inetOrgPerson. RFC 2798 was submitted as an Informational RFC in April 2000 defining the inetOrgPerson LDAP object class. The object class definition is considered an extension to LDAPv3 and, as such, is not included in the RFC 3377 LDAPv3 specifications.

Nevertheless, the timing of the Windows 2000 Server release in February 2000 and the introduction of RFC 2798 in April of that same year barred the inclusion of the inetOrgPerson Object Class within the Windows 2000 Server Active Directory schema. However, any Windows 2000 Active Directory implementation can easily accommodate inetOrgPerson through Active Directory’s extensible schema.

In response to customer requests for an inetOrgPerson object class for Windows 2000, Microsoft released the Windows 2000 inetOrgPerson Kit. The kit, which is freely available, derives inetOrgPerson from the user class within Active Directory’s schema. In addition to defining the inetOrgPerson, the kit delivers basic administrative control over the class object. See the “Additional Resources” section for information on obtaining the Microsoft Windows 2000 inetOrgPerson Kit.

As noted previously, Windows Server 2003’s Active Directory schema includes the full definition and provides complete manageability of RFC 2798 and the inetOrgPerson LDAP Object Class.

Native LDAP Calls

Another common misconception is that Active Directory does not support native LDAP calls, thereby forcing developers to use Active Directory Services Interface (ADSI) for access to the directory. In fact, Active Directory fully supports native LDAP calls through support of the LDAP API (RFC 1823) — it is actually the primary access method for the majority of Window’s directory operations. Microsoft merely provides ADSI as an abstraction layer to simplify directory development and promote directory interoperability.

It is important to note that Active Directory does have some functionality that is not exposed by the LDAP protocol since the functionality extends beyond the LDAP model. For instance, LDAP applications bind to specific directory servers via the server’s DNS name whereas Active Directory applications have the option of employing a distributed locator service to find a nearby replica of a given directory partition. This option does not force ISVs to rewrite their applications since they can bind to a specific LDAP server as usual. However, if an ISV wants to use the Active Directory distributed locator service to reduce the customer cost of managing an application, they can call the DsGetDcName API and then use LDAP. Conversely, the ISV may use ADSI, which abstracts away both DsGetDcName and LDAP. The choice is entirely up to the ISV.

Active Directory fully supports writing to the Server Principal Name via the Windows Server 2003 and Windows 2000 Server LDAP APIs.

 

Directory Interoperability

It is not uncommon to find a variety of directories—from resource to application-specific—deployed within the enterprise today. However, multiple directories can create complex challenges for the users, administrators, and developers of any organization. From multiple user logon requirements and management complexities to developers needing to support multiple versions of their applications, directory complexities may continue without a common management and development framework. Microsoft has introduced several interoperability tools to help customers reduce the complexities of using and managing heterogeneous directory servers.

LDAP API

An LDAP API is available through the Microsoft Developer Network (MSDN) Platform SDK. It provides native LDAP functionality for client applications to search for and retrieve information from an LDAP-compliant directory server. Additional functions, such as modifying directory entries or access control to allow clients to authenticate themselves, are also available from within the LDAP API. Actually, the API supports all LDAP compliance specific functions identified in the previous sections to assist in development efforts.

Active Directory Services Interface

ADSI is a set of Component Object Model (COM) programming interfaces that make it easy for customers and independent software vendors (ISVs) to build applications that register with, access, and manage multiple directory services with a single set of well-defined interfaces. ADSI is an open API.

One of the most familiar open programming APIs is Open Data Base Connectivity (ODBC). ODBC provides open interfaces for relational databases, thus allowing developers to write applications and tools that work with any database that supports ODBC. Because of the thriving ODBC development community, every major relational database now supports ODBC. ADSI is equivalent to ODBC for directory services.

ADSI provides developers with access to multiple directory service providers through a set of interfaces. Applications written to ADSI work with any directory service that offers an ADSI provider, thus addressing some of the complexities outlined previously. By abstracting the capabilities of directory services from different network providers, ADSI presents a single set of directory service interfaces for managing network resources.

For example, ADSI defines a naming convention that can uniquely identify an ADSI object in an LDAP-compliant or similar directory. These names are called AdsPath strings. In the following code example, AdsPath contains the LDAP provider’s moniker and the provider’s specific path to identify an organizational unit in the Sample Domain:

LDAP://OU=Sales, DC=Sample, DC=COM

As an interoperability tool, ADSI is:

  • Open – Any directory vendor can implement an ADSI provider; Microsoft has included an LDAP ADSI provider.
  • Directory Service Independent – Administrative applications are not bound to a vendor’s directory service as applications can be developed without understanding of vendor-specific directory APIs.
  • Multiple Development Language Capable – COM applications can be written in languages such as Microsoft® Visual Basic®, Microsoft Visual Basic Scripting Edition (VBScript), and C/C++.
  • Simple Programming Model – ADSI consists of a small, easy-to-learn set of interfaces.
  • Scriptable – Any automation-compatible language can be used to develop directory service applications. Administrators and developers can use the tools they already know.
  • Functionally Rich – ISVs and sophisticated end users can develop serious applications using the same ADSI models used for simple scripted administrative applications.
  • Extensible – Directory providers, ISVs, and end users can extend ADSI with new objects and functions to add value or meet unique needs.
  • OLE DB Aware – ADSI provides an OLE DB interface so that programmers familiar with database programming through OLE DB can use it effectively.

Development Environments

ADSI client applications can be written in many languages. For most administrative tasks, ADSI defines interfaces and objects accessible from automation-compliant languages like Microsoft® Visual Basic®, Microsoft Visual Basic Scripting Edition (VBScript), Active Server Pages (ASP), and Java to the more performance and efficiency-conscious languages such as C and C++. A good foundation in COM programming is useful to the ADSI programmer.

Active Directory Application Mode

Active Directory Application Mode (ADAM) is a new capability of Microsoft Active Directory that addresses certain deployment scenarios related to directory-enabled applications. ADAM runs as a non-operating system service and, as such, does not require deployment on a domain controller. Running as a non-operating system service means that multiple instances of ADAM can run concurrently on a single server, with each instance being independently configurable.

While the vision of a single enterprise directory is still unfulfilled, ADAM represents a significant breakthrough in directory services technology that separates typical Network Operating System (NOS) functionality from a native LDAP directory.

Directory Services Markup Language

The Directory Services Markup Language (DSML) provides a means of representing directory structural information and directory operations as an XML document. The intent of DSML is to allow XML-based enterprise applications to leverage profile and resource information from a directory in their native environment. DSML allows XML and directories to work together and provides a common ground for all XML-based applications to make better use of directories.

DSML Services for Windows extends the power of the Active Directory service. Because DSML Services for Windows uses open standards such as HTTP, XML, and SOAP, a greater level of interoperability is possible. For example, in addition to the already standard Lightweight Directory Access Protocol (LDAP), many devices and other platforms have other alternatives to communicate with Active Directory. This provides a number of key benefits for IT administrators and independent software vendors (ISVs), who can now have even more open-standard choices to access Active Directory.

DSML Services for Windows supports DSML version 2 (DSMLv2). DSMLv2 is a standard approved by the Organization for the Advancement of Structural Information Standards (OASIS) and backed by many directory services vendors. Supporting the DSMLv2 specification allows better interoperability with other directory services vendors who are also supporting this standard. For information about the DSMLv2 specification and schema, see the OASIS Web site.

Microsoft Identity Integration Server 2003, Enterprise Edition

For organizations that must support heterogeneous directories, Microsoft Identity Integration Server (MIIS) 2003, Enterprise Edition, enables the integration and management of identity information across multiple repositories, systems, and platforms. Although the administration of these multiple repositories often leads to time-consuming and redundant efforts, it also causes frustration for users who may need to remember multiple IDs and passwords for varying applications or systems. The larger the organization, the greater the potential for multiple repositories and the effort required to keep them current.

MIIS augments Active Directory by providing broad interoperability capabilities, including integration with a wide range of identity repositories; provisioning and synchronizing identity information across multiple stores; and brokering changes to identity information by automatically detecting updates and sharing the changes across systems.

 

Additional Resources

See the following resources for further information:

Lightweight Directory Access Protocol Version 3

The IETF LDAPv3 Working Group:

The LDAPv3 Working Group archived newsgroup:

RFC 3377, the current definition of LDAPv3:

Open Group and the Directory Interoperability Forum

The Open Group’s VSLDAP compliance testing suite overview:

The Directory Interoperability Forum (DIF):

Developing with Active Directory Services Interface

Active Directory Services Interface developer’s reference:

Miscellaneous

The Microsoft Active Directory Web site:

Active Directory Application Mode (ADAM):

Directory Services Markup Language (DSML):

Microsoft Identity Integration Server 2003, Enterprise Edition:

The Microsoft Windows 2000 inetOrgPerson Kit:

LDAP reference for developers on MSDN:

LDAP API reference for developers on MSDN:

 

 

For the latest information about Windows Server 2003, see the Windows Server 2003 Web site at http://www.microsoft.com/windowsserver2003.

Identity and Access

Posted on by escapebusinesssolutions in Uncategorized

 


Identity and Access

Windows Server 2012

Table of contents

Identity and access enhancements in Windows Server 2012………… 5

Protecting digital assets with previous versions of Windows Server ……………………………………………. 5

Protecting digital assets with Windows Server 2012 …………………………………………………………………………. 6

Dynamic Access Control ………………………………………………………………………. 7

Classification …………………………………………………………………………………………………………………………………………………… 8

Control access ………………………………………………………………………………………………………………………………………………… 8

The structure of central access policies ……………………………………………………………………………………………………………… 9

Central access policies and file servers ……………………………………………………………………………………………………………. 10

Policy staging ……………………………………………………………………………………………………………………………………………………….. 11

Access-denied remediation ………………………………………………………………………………………………………………………………. 11

Security auditing ………………………………………………………………………………………………………………………………………….. 13

Protection ………………………………………………………………………………………………………………………………………………………. 14

Active Directory Domain Services ……………………………………………………..16

Simplified deployment ………………………………………………………………………………………………………………………………. 16

Deployment with cloning …………………………………………………………………………………………………………………………. 17

Safer virtualization of domain controllers ……………………………………………………………………………………………. 17

Windows PowerShell script generation ……………………………………………………………………………………………….. 18

Active Directory for client activation ……………………………………………………………………………………………………… 19

Group-managed service accounts …………………………………………………………………………………………………………. 19

DirectAccess and remote access ……………………………………………………….20

Integrated remote access …………………………………………………………………………………………………………………………. 21

Cross-premises connectivity ……………………………………………………………………………………………………………………. 23

Improved management experience ……………………………………………………………………………………………………… 24

Easier deployment ………………………………………………………………………………………………………………………………………. 24

Improved deployment scenarios ……………………………………………………………………………………………………………. 25

Windows Server 2012: Identity and Access

2

Windows Server 2012: Identity and Access

3

Scalability improvements ………………………………………………………………………………………………………………………….. 25

Summary ……………………………………………………………………………………………….26

List of charts, tables, and figures ………………………………………………………..27

Windows Server 2012: Identity and Access

4

Copyright information

© 2012 Microsoft Corporation. All rights reserved. This document is provided “as-is.” Information and views expressed in this document, including URL and other Internet website references, may change without notice. You bear the risk of using it. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. You may modify this document for your internal, reference purposes.

Windows Server 2012: Identity and Access

5

Identity and access enhancements in Windows Server 2012

Today’s organizations need the flexibility to respond rapidly to new opportunities. They also need to give workers access to data and information—across varied networks, devices, and applications—while still keeping costs down. Innovations that meet these needs—such as virtualization, multitenancy, and cloud-based applications—help organizations maximize existing infrastructure investments, while exploring new services, improving management, and increasing availability.

While some factors—such as hybrid cloud implementations, a mobile workforce, and increased work with third-party business partners—add flexibility and reduce costs, they also lead to a more porous network perimeter. When organizations move more and more resources into the cloud, and grant network access to mobile workers and business partners outside the firewall, managing security, identity, and access control becomes a greater challenge. Adding to this challenge are increasingly stringent regulatory requirements, such as Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule and Sarbanes-Oxley Act of 2002 (SOX), both of which increase the cost of compliance.

New and enhanced capabilities in Windows Server 2012 help organizations meet these challenges by making it easier and less costly to ensure secure access to valuable digital assets and comply with regulations. These identity and access improvements include:

Dynamic Access Control. A new feature that enables automated information governance on file servers for compliance with business and regulatory requirements.

Active Directory Domain Services. Storing directory data and managing communication between users and domains is improved by making it easier to deploy and virtualize domain services both locally and remotely, as well as simplifying management tasks.

DirectAccess. Always-available connectivity to the corporate network now includes simplified deployment, a streamlined management experience, and improved scalability and performance.

This paper provides an introduction to these Windows Server 2012 identity and access technologies for IT professionals.

Protecting digital assets with previous versions of Windows Server

A major obstacle faced by IT professionals today is the sheer volume of repetitive work that needs to be done, particularly when large deployments of physical or virtual desktops are involved. Manual work is not only time-consuming, but it makes systems less secure by introducing many opportunities for errors and misconfiguration. In previous versions of Windows Server, Microsoft helped reduce this type of work.

For example, Windows Server 2008 R2 included the following improvements:

Windows Server 2012: Identity and Access

6

File Classification Infrastructure. This new infrastructure for classifying files by tagging them enabled IT professionals to more easily manage unstructured data in files based on their organizational value.

Active Directory Domain Services. As part of the Information Protection Solution, Active Directory Domain Services was improved to make domain controllers easier to deploy, both on-premises and in the cloud.

System Management and Security. The Windows PowerShell 2.0 command-line interface enabled IT professionals to automate many common tasks involved in deploying and managing desktops.

Protecting digital assets with Windows Server 2012

With Windows Server 2012, Microsoft builds on these previous improvements by making it even easier to configure, manage, and monitor users, resources, and devices to improve security and automate auditing. In addition, Windows Server 2012 includes the following new and enhanced identity, access and data protection features:

Dynamic Access Control gives you the ability to automatically control and audit access to files in file shares across your organization based on the characteristics of both the files and the users requesting access to them. It uses claims to achieve this high degree of access control specificity. Claims, contained in security tokens, consist of assertions about a user or device, such as name or type, department, or security clearance, for example. You can employ user claims, device claims, and file classification tags to centrally control and audit access to files, as well as use Rights Management Services (RMS) to protect information in files across your organization.

Active Directory Domain Services is important in new hybrid cloud infrastructures because it supports the increased need for security, compliance, and access control. Furthermore, unlike many competing cloud services, which require users to have a separate set of credentials hosted with the provider, Active Directory Domain Services provides continuity between your on-premises and cloud resources so that users only need a single set of credentials no matter where the resources are located. A new deployment wizard and support for cloning virtual domain controllers makes Active Directory Domain Services easier to virtualize and simpler to deploy, both locally and remotely. The introduction of Active Directory Domain Services and Windows PowerShell 3.0 integration, and the ability to capture and record command-line syntax as you perform tasks in the Service Manager interface, improves automation of manual tasks. Other new features include desktop activation support using Active Directory Domain Services and Group Managed Service Accounts.

DirectAccess allows nearly any user who has an Internet connection to more securely access corporate resources, such as email servers, shared folders, or internal websites, with the experience of being easily connected to the corporate network. Windows Server 2012 offers a new, unified management experience, allowing administrators to configure DirectAccess and legacy virtual private network (VPN) connections from one location. Other enhancements simplify deployment, and improve performance and scalability.

The remaining sections in this paper describe these new features in more detail.

Windows Server 2012: Identity and Access

7

Dynamic Access Control

Dynamic Access Control in Windows Server 2012 gives IT professionals new ways to control access to file data and monitor regulatory compliance. It provides next-generation authorization and auditing controls, along with classification capabilities that let you apply information governance to the unstructured data on file servers.

Until now, file security was handled at the file and folder level. IT professionals had little control over the way security was handled by users day to day. However, by using Dynamic Access Control, you can restrict access to sensitive files regardless of user actions by establishing and enforcing file security policy at the domain level, which is then enforced across all Windows Server 2012 file servers. For instance, if a development engineer accidentally posts confidential files to a publicly shared folder, those files can still be protected from access by unauthorized users.

In addition, security auditing is now more powerful than ever, and audit tools make it easier to prove compliance with regulatory standards, such as the requirement that access to Health and Biomedical Information (HBI) is appropriately guarded and regularly monitored.

Windows Server 2012 provides the following new and enhanced ways to control access to your files while providing authorized users the resources they need:

Classify. Automatic and manual file classification using an improved file classification infrastructure. There are several methods to manually or automatically apply classification tags to files on file servers across the organization.

Control. Central access control for information governance. You can control access to classified files by applying central access policies (CAPs). CAPs is a new feature of Active Directory that enables you to define and enforce very specific requirements for granting access to classified files. For example, you can define which users can have access to files that contain health information within the organization by using claims that might include employment status (such as full-time or contractor) or access method (such as managed computer or guest). You can even require two-factor authentication, such as that provided by smartcards. Central access control functionality includes the ability to provide automated assisted access-denied remediation when users have problems gaining access to files and shares.

Audit. File access auditing for forensic analysis and compliance. You can audit access to files by using central audit policies, for example, to identify who gained (or tried to gain) access to highly sensitive information.

Protect. Classification-based encryption. You can apply protection by using automatic RMS encryption for sensitive documents. For example, you can configure Dynamic Access Control to automatically apply RMS protection to all documents that contain HIPAA information. This feature requires a previously provisioned RMS environment.

Dynamic Access Control can provide these classification, control, audit, and protection capabilities because it is built on top of the following technologies:

 A new Windows authorization and audit engine that can process conditional expressions and central policies

 Kerberos support for user claims and device claims within Active Directory Domain Services

Windows Server 2012: Identity and Access

8

Improvements to the File Classification Infrastructure

 RMS extensibility support so that partners can provide solutions that encrypt third-party files

You can use the Dynamic Access Control application programming interface (API) to extend these technologies and create custom classification tools, audit software, and more.

Classification

The first step in establishing file access policies that are more secure is to identify the files, and then classify them by applying tags to group files based on the information they contain. In Windows Server 2012, files are tagged in one of four ways:

By location. When a file is stored on a file server, it inherits the tags from its parent folder. Folder tags are specified by the folder owner.

Manually. Users and administrators can manually apply tags through the Windows 8 operating system File Explorer interface, or use data entry applications to apply them.

Automatically. Automatic classification processes in Windows Server 2012 can automatically tag files, depending on the content of the file. This method is useful for applying tags to large numbers of files.

By application API. Applications can use APIs to tag files that they manage. For example, tags can be specified by line-of-business (LOB) applications that store information on file servers, or by data management applications.

Control access

Controlling access to files is enforced by central access policies—sets of authorization policies that you centrally manage in Active Directory Domain Services, and deploy to file servers using Group Policy. You can use CAPs to comply with both organizational and regulatory requirements. CAPs help you to create more complete access policies by pairing information about files in the form of file tags with user and device claims.

In earlier versions of Windows Server, claims were used only by Active Directory Federation Services to authorize users in one domain to use applications in different, federated domains based on attributes submitted to Active Directory Federation Services. In Dynamic Access Control, the functionality of claims is essentially the same. In both cases, a claim consists of one or more statements (for example, name, identity, key, group, privilege, or capability) made about a user or device. These statements are contained in a security token that is issued and signed by a trusted partner or entity (such as Active Directory Domain Services) and used for authorizing that user or device to access a resource. You can create claim properties, either manually by using the Active Directory management tools or by using an identity management tool. In the case of Dynamic Access Control, the token is issued by Active Directory Domain Services and the resource to be accessed is a file.

In Dynamic Access Control, claims can be combined into logical policies that enable fine-grained control over arbitrarily-defined subsets of files. The following are two examples of situations where you might want to apply such policies:

 To obtain access to high-business-impact (HBI) information, a user might be required to be a full-time employee. In this scenario, you would have to do the following:

o Identify and tag the files that contain HBI information.

Windows Server 2012: Identity and Access

9

o Identify the full-time employees in your organization.

o Create a central access policy that applies to all files that contain HBI on all file servers across the organization.

 To enforce an organization-wide requirement to restrict access to personally identifiable information (PII) in files so that only the file owner and members of the human resources (HR) department are allowed to view it, you might implement a policy that applies to all PII files independent of their location. In this scenario, you would have to do the following:

o Identify and classify (tag) the files that contain PII.

o Identify the group of HR members that are allowed to view PII information.

o Create a CAP that applies to all files that contain PII on all file servers across the organization.

The motivation to deploy and enforce an authorization policy can arise for different reasons and from multiple levels of the organization. The following are some examples:

Organization-wide authorization policy. Most commonly initiated from the information security office, this type of authorization policy arises from compliance or another high-level requirement that is relevant across the organization. For example, HBI files should be accessed by full-time employees only.

Departmental authorization policy. Various departments in an organization may have special data-handling requirements that they want to enforce. For example, the finance department might want to limit access to finance servers for the finance employees.

Specific data-management policy. This type of policy usually arises from compliance and organizational requirements for protecting information that is being managed, such as to prevent modification or deletion of files that are under retention or files that are under electronic discovery (eDiscovery).

Need-to-know policy. This type of policy is typically used in conjunction with the policy types mentioned earlier. The following are two examples:

o Vendors should be able to access and edit only those files that relate to a project that they are working on.

o In financial institutions, information barriers are important so that analysts do not access brokerage information and brokers do not access analysis information.

The structure of central access policies

CAPs stored in Active Directory Domain Services act as security umbrellas that an organization applies across its file servers. These policies supplement—but do not replace—the local access policy, or discretionary access control list (DACL), applied to files and folders. For example, if a local DACL allows access to a specific user, but a CAP that is applied to the file denies access to the same user, the user cannot access the file. The reverse also applies: if a CAP allows access to a user but the local DACL denies it, the user cannot access the file. File access is only possible when permitted by both local DACLs and CAPs.

A CAP can contain many rules—each of which are evaluated and deployed as part of an overall CAP. Each rule contained in the CAP has the following logical parts:

Applicability. This is a condition that defines which files the rule applies to. For example, the condition can define all files tagged as containing personal information.

Windows Server 2012: Identity and Access

10

Access conditions. This is a list of one or more access control entries (ACEs) that define who can access the data, such as allow read and write access if the user has a high clearance level and their device is a managed device.

Figure 1 shows the components of a CAP rule, and how they can be combined to create very explicit data access policies.

Figure 1: CAP components

Central access policies and file servers

Figure 2 shows the interrelationships between Active Directory Domain Services, where CAPs, user claims, and property definitions are defined and stored; the file server, where these policies are applied; and the user, who is trying to gain access to a file on the file server.

Figure 2: CAP structure

Figure 3 shows how you can combine different rules (blue boxes) into a CAP (green boxes) that can then be applied to shares on file servers across the organization.

Windows Server 2012: Identity and Access

11

Figure 3: Combining multiple policies into policy lists and applying them to resources

Policy staging

When you want to change a policy, Microsoft Windows 2012 lets you test a proposed policy that runs parallel to the current policy so that you can identify the consequences of the new policy without enforcing it. This feature, which is known as policy staging, lets you measure the effects of a new policy in the production environment.

When policy staging is enabled, Windows Server 2012 continues to use the current policy to authorize user access to files. However, if the access allowed by the proposed policy differs from that of the current policy, the system logs an event with the details. You can use the logged events to determine whether the policy has to be changed or is ready to be deployed.

Access-denied remediation

Of course, denying access is only part of an effective central access control strategy, and sometimes access must be granted after at first being denied. Today, when access is denied, the user does not receive additional information on how to get access. This causes a lot of pain for both users and help desk or IT administrators. To mitigate this problem, assisted access-denied remediation in Windows Server 2012 enables you to provide the user with additional information and the opportunity to send an access request email message to the appropriate owner. Access-denied remediation reduces the need for manual intervention by providing the following three different processes for granting users access to resources:

Self-remediation. If users can determine what the issue is and correct the problem so that they can get the requested access, the impact on the organization is low, and minimal special exceptions are needed in the organization policy. Windows Server 2012 helps you to author a general access-denied message to help users self-remediate when access is denied. This message can include URLs to direct the users to self-remediation websites provided by the organization.

Windows Server 2012: Identity and Access

12

Remediation by the file owner. Windows Server 2012 enables you to create a distribution list of file or folder owners so that users can directly connect with them to request access. This resembles the Microsoft SharePoint model, where the share owner receives a user’s request for access to a file. Remediation can range from adding user rights to the appropriate file or folder to editing share permissions. For example, if a local DACL on a file allows access to a specific user but a CAP restricts access to the same user, the user will be unable to gain access to the file. In Windows Server 2012 when a user requests access to a file or a folder, an email message with the request details is sent to the file owner. When additional help is required, the file owner can in turn forward this information to the appropriate IT administrator.

Remediation by help desk and file server administrators. When the user cannot self-remediate an access problem and the file owner cannot help, the issue can be corrected manually by the help desk or file server administrator. This is the most costly and time-consuming remediation. Windows Server 2012 provides a UI to view the effective permissions for users on a file or a folder so that it is easier to troubleshoot access issues.

Figure 4 shows the series of events involved in access-denied remediation.

Figure 4: Access-denied remediation

Access denied remediation provides a user access to a file when it has been initially denied:

Windows Server 2012: Identity and Access

13

1. The user attempts to read a file.

2. The server returns an “access denied” error message because the user has not been assigned the appropriate claims.

3. On a computer running Windows 8, Windows retrieves the access information from the File Server Resource Manager on the file server, and presents a message with the access remediation options, which may include a link for requesting access.

4. When the user has satisfied the access requirements (for example, by signing a non-disclosure agreement, or providing other authentication) the user’s claims are updated.

5. The user can access the file.

 

Security auditing

Security auditing of file access is one of the most powerful tools to help maintain the security of an organization. A key goal of security auditing is regulatory compliance. For example, industry standards such as SOX, HIPAA, and Payment Card Industry (PCI) require organizations to follow a strict set of rules related to data security and privacy. Security audits help establish the presence or absence of such policies and thereby prove compliance or noncompliance with these standards. Additionally, security audits help detect anomalous behavior, identify and reduce gaps in security policy, and deter irresponsible behavior by creating a trail of user activity that can be used for forensic analysis. Audit policy requirements typically arise from three levels:

Information security. File access audit trails are frequently used for forensic analysis and intrusion detection. The ability to monitor specified events regarding access to high-value information lets organizations significantly improve their response time and investigation accuracy.

Organizational policy. For example, organizations regulated by PCI standards can have a central policy to monitor access to all files that are marked as containing credit card information and PII, or organizations may want to monitor all unauthorized attempts to view information about their projects.

Departmental policy. For example, the finance department may require that the ability to modify certain finance documents (for example, a quarterly earnings report) be restricted to the finance department, and thus want to monitor all other attempts to change these documents. Additionally, the compliance department may want to monitor all changes to central policies and policy constructs such as user, computer, and resource attributes.

One of the biggest considerations for security audits is the cost of collecting, storing, and analyzing audit events. If the audit policies are too broad, the volume of audit events that are collected increases, making it more time-consuming and expensive to identify the most important audit events. However, if the audit policies are too narrow, you risk missing important events.

With Windows Server 2012, you can author audit policies by combining claims and resource properties (file tags). This leads to audit policies that are richer, more selective, and easier to manage by reducing the number of potential audit events to those most relevant to your auditing requirements. It enables scenarios that until now were either impossible, or too difficult to implement. The following are examples of such audit policies:

Windows Server 2012: Identity and Access

14

Audit everyone who does not have a high security clearance and yet tries to access an HBI document. In this example, “high security clearance” is a claim, and “HBI” is a file tag. The audit policy triggers an event when a user who does not have a high security clearance tries to gain access to an HBI document.

 Audit all vendors when they try to access documents related to projects that they are not working on. In this example, the user’s claims include an employment status of “vendor” as well as a list of projects the user is authorized to work on. In addition, each file in the organization to which a vendor could potentially have access was tagged to identify its associated project. The audit policy triggers an event if a vendor tries to access a project file, and none of the projects in the vendor’s claim list matches the file’s project tag.

To view and query audit events you can use familiar tools such as Event Viewer on the local server or Microsoft System Center Operations Manager Audit Collection Service across multiple servers. The Dynamic Access Control API also provides support for integrating DAC audit events into third- party audit software consoles. These tools help you to answer such questions as, “Who is accessing my HBI data?” or “Was there an unauthorized attempt to access sensitive data?”

Figure 5 shows the file-access auditing workflow and the interrelationships between Active Directory, where claim types and resource properties are created; Group Policy, where the audit policies are defined and stored; the file server, where policies and resource properties are applied as file tags; and the user, who is trying to access information on the file server.

Figure 5: Central auditing workflow

Protection

Protecting sensitive information involves reducing risk for the organization. Various regulations, such as HIPAA or Payment Card Industry Data Security Standard (PCI-DSS), require encryption of information, and there are many business reasons to also encrypt sensitive information. However, encryption is expensive and can adversely affect productivity. Therefore, organizations usually have different approaches and priorities for it.

Windows Server 2012: Identity and Access

15

To support this scenario, Windows Server 2012 lets you automatically encrypt sensitive Microsoft Office files based on their classification. This is done through automatic file management of tasks that are running on the server, and that start RMS protection for sensitive Microsoft Office documents a few seconds after the file is identified as being a sensitive file on the file server (continuous file management tasks).

RMS encryption provides another layer of protection for files. If a person with access to a sensitive file inadvertently sends that file out through email, the file is still protected by the RMS encryption. Nearly any user who wants to gain access to the file must first authenticate to an RMS server to receive the decryption key. This process is illustrated in Figure 6.

Figure 6: Classification-based RMS protection

Dynamic Access Control allows sensitive information to be automatically protected using Active Directory RMS:

1. A rule is created to automatically apply RMS protection to nearly any file that contains the word “confidential.”

2. A user creates a file with the word “confidential” in the text and saves it.

3. The RMS Dynamic Access Control classification engine, following rules set in the CAP, discovers the document with the word “confidential,” and initiates RMS protection accordingly.

4. The RMS template and encryption are applied to the document on the file server, and it is classified and encrypted.

 

Note that support for third-party file formats is available through third-party vendors. Also be aware that if a file is RMS-protected, data management features such as search- or content-based classification are no longer available for that file.

Windows Server 2012: Identity and Access

16

Active Directory Domain Services

Active Directory has been at the center of IT infrastructure for more than 10 years, and its features, adoption, and business value have grown with each new release. Today, most Active Directory infrastructure remains on-premises, but the trend toward cloud computing is creating the need to deploy Active Directory in the cloud as well.

New hybrid infrastructures are emerging, and Active Directory Domain Services must support the needs of new and unique deployment models that include services hosted entirely in the cloud, services that consist of both cloud and on-premises components, and services that remain exclusively on-premises. These new hybrid models further increase the importance of security and compliance, and compound the already complex and time-consuming exercise of ensuring that access to information and services is appropriately audited, and accurately expresses the intent of the organization.

Active Directory Domain Services in Windows Server 2012 addresses these emerging needs with features that help you more quickly and easily deploy domain controllers both on-premises and in the cloud, audit and authorize access to files, and perform administrative tasks at scale—either locally or remotely—through consistent graphical and scripted management interfaces. Active Directory Domain Services in Windows Server 2012 improvements include the following:

 Simpler on-premises deployment, which replaces DCpromo with a new, streamlined domain controller configuration wizard that is integrated with Server Manager and built on Windows PowerShell 3.0.

 More rapid deployment of virtual domain controllers through cloning.

 Better support for public and private cloud implementations through safer virtualization of domain controllers.

 A consistent graphical and scripted management interface that enables you to perform tasks in the Active Directory Administrative Center and automatically generate the syntax required to fully automate the task in Windows PowerShell 3.0.

 Functionality that uses Active Directory to simplify client activations.

 Group Managed Services Accounts for groups of servers such as server clusters that share their identity and service principal name.

Using these features, you can effectively and efficiently deploy and manage Active Directory Domain Services over multiple servers, locally and around the globe.

Simplified deployment

The new Active Directory Domain Services configuration wizard in Windows Server 2012 integrates all the required steps to deploy new domain controllers into a single graphical interface. It requires only one organization-level credential, and can prepare the forest or domain by remotely targeting the appropriate operations master role holders. It conducts extensive prerequisite validation tests that minimize the opportunity for errors that might have otherwise blocked or slowed the installation. The wizard is built on

Windows Server 2012: Identity and Access

17

Windows PowerShell 3.0, and is integrated with Server Manager. It can configure multiple servers and remotely deploy domain controllers, resulting in a deployment experience that is simpler, more consistent, and less time-consuming.

The Active Directory Domain Services configuration wizard includes the following features:

Adprep integration into the Active Directory Domain Services deployment process. This reduces the time that is required to deploy Active Directory Domain Services, and reduces the chances for errors that might block domain controller promotion.

Remote execution against multiple servers. This greatly reduces the probability of administrative errors and the overall time required for deployment, especially when you deploy multiple domain controllers across global countries/regions and domains.

Prerequisite validation. This identifies potential errors before the deployment begins. You can correct error conditions before they occur without the concerns that result from a partially complete upgrade.

Configuration pages grouped in a sequence that mirror the requirements of the most common promotion options, with related options grouped in fewer wizard pages. This provides better context for making installation choices, and reduces the number of steps and time that is required to complete domain controller installation.

Options that were specified in the wizard are exported into a Windows PowerShell script. This simplifies the process of automating later Active Directory Domain Services installations through automatically generated Windows PowerShell scripts.

Deployment with cloning

With earlier versions of Windows Server, administrators found that deploying virtualized replica domain controllers was as labor-intensive as deploying physical domain controllers. In theory, this should not be the case because virtualization brings the possibility of cloning domain controllers, instead of performing all deployment steps separately for each one. Domain controllers within the same domain/forest are nearly identical, except for name, IP address, and so on. Therefore virtualization should be fairly easy. With earlier versions of Windows Server, however, deployment still involved many (redundant) steps.

With Windows Server 2012, you can deploy replica virtual domain controllers by “cloning” existing virtual domain controllers. This significantly reduces the number of steps and time involved by eliminating repetitive deployment tasks and also lets you fully deploy additional domain controllers that are authorized and configured for cloning by the Active Directory domain administrator.

Safer virtualization of domain controllers

Active Directory Domain Services has been successfully virtualized for several years, but features present in most hypervisors can invalidate strong assumptions made by the Active Directory replication algorithms—primarily, the assumption that the logical clocks used by domain controllers to determine relative levels of convergence only go forward in time. Windows Server 2012 includes improvements that enable virtual domain controllers to detect when snapshots are applied to a virtual machine or a virtual machine is copied, causing the domain controller clock to go backward in time.

This new functionality is made possible by a virtual domain controller that uses a unique ID exposed by the hypervisor, called the virtual machine GenerationID. The virtual machine GenerationID changes when

Windows Server 2012: Identity and Access

18

the virtual machine experiences an event that affects its position in time. The virtual machine GenerationID is exposed to the virtual machine’s address space within its Basic Input Output System (BIOS), and is made available to its operating system and applications through a Windows Server 2012 driver.

During startup, and before completing any transactions, a Windows Server 2012 virtual domain controller compares the current value of the virtual machine GenerationID against the value that it stored in the directory. A mismatch is interpreted as a “rollback” event, and the domain controller uses safeguards in Active Directory Domain Services that are new to Windows Server 2012. The safeguards enable the virtual domain controller to converge with other domain controllers and also prevent it from creating duplicate security principals.

For Windows Server 2012 virtual domain controllers to gain this extra level of protection, the virtual domain controller must be hosted on a virtual machine GenerationID–aware hypervisor such as Windows Server 2012 Hyper-V.

Windows PowerShell script generation

The Windows PowerShell cmdlets for Active Directory are a set of tools that help you to manipulate and query Active Directory Domain Services by using Windows PowerShell commands, and to create scripts that automate common administrative tasks. The Active Directory Administrative Center uses these cmdlets to query and modify Active Directory Domain Services according to the actions that are performed within the Active Directory Administrative Center UI.

In Windows Server 2012, the Windows PowerShell History viewer in the Active Directory Administrative Center, as shown in the following figure, lets an administrator view the Windows PowerShell commands as they execute in real time. For example, when you create a new fine-grained password policy, the Active Directory Administrative Center displays the equivalent Windows PowerShell commands in the Windows PowerShell History viewer task pane. You can then use those commands to automate the process by creating a Windows PowerShell script.

Figure 7: Windows Server 2012 Windows PowerShell History viewer

By combining scripts with scheduled tasks, you can automate everyday administrative duties that were previously completed manually. Because the cmdlets and required syntax are created for you, very little experience with Windows PowerShell is required. Because the Windows PowerShell commands are the same as the ones executed by the Active Directory Administrative Center, they should replicate their original function exactly.

Windows Server 2012: Identity and Access

19

Active Directory for client activation

Client licensing is an additional labor-intensive task that can be eased with the improved Active Directory functionality in Windows Server 2012. With earlier versions of Windows Server, volume licensing for Windows and Office required Key Management Service (KMS) servers. These entail several drawbacks:

 They require remote procedure call (RPC) network traffic, which some organizations want to disable.

 Additional training is necessary.

 The turnkey solution only covers approximately 90 percent of deployments.

 There is no graphical administration console, so the process is more complex than it needs to be.

 KMS does not support any kind of authentication, because the Microsoft Software License terms prohibit the customer from connecting the KMS server to any external network.

 Access to the service means that anyone can be activated.

This situation is improved in Windows Server 2012 because it helps leverage your existing Active Directory infrastructure to help you activate clients. No additional computers are required, and no RPC is needed. The activation uses Lightweight Directory Access Protocol (LDAP) exclusively and includes support for read-only domain controllers (RODCs).

In this activation process, the only data written back to the directory is what’s required for the installation and service. Activating the initial customer-specific volume license key (CSVLK) requires the following:

 One-time contact with Microsoft Activation Services over the Internet (identical to retail activation).

 A key entered using volume activation server role or the command line.

 Repetition of the activation process for additional forests (by default, up to six times).

Another benefit of Active Directory integration is that the activation object is maintained in the configuration partition. This represents proof-of-purchase, and means that the activated computers can be a member of any domain in the forest. And, perhaps most important, with Active Directory activation integration, all computers that are running Windows 8 will automatically activate. This represents a significant workflow improvement over earlier versions of Windows Server, and is achieved by using additional resources that are provided in Active Directory.

Group-managed service accounts

Managed service accounts (MSAs) were a new type of account introduced in Windows Server 2008 R2 and Windows 7 to enhance the service isolation and manageability of network applications such as Microsoft SQL Server and Exchange Server. They eliminate the need for an administrator to manually administer the service principal name (SPN) and credentials for domain-level service accounts.

Until now, however, this feature has not been available for server groups, such as clusters, that share their identity and service principal name. This creates a problem for IT administrators.

When a client connects to a service hosted on a server farm using network load balancing (NLB) or some other method where all the servers appear to be the same service to the client, authentication protocols supporting mutual authentication, such as Kerberos, cannot be used unless all the instances of the

Windows Server 2012: Identity and Access

20

services use the same principal (which means that they use the same passwords/keys to prove their identity). Service administrators find managing this to be difficult.

When a client connects to a shared service it cannot know in advance which instance it will connect to, so the authentication must succeed regardless of the host. This requires that each instance of the server use the same security principal. Today, services have four principals to choose from, each with their own issue: computer, virtual, managed service, or user.

Computer, MSA, or virtual accounts cannot be shared across multiple systems. This only leaves the option of using a user account for services on server farms. Because user accounts do not have password management, each organization then has to create a solution to update keys for the service in Active Directory, and distribute the keys to all instances of the services. This is expensive and problematic.

By creating a group MSA, services or service administrators do not have to manage password synchronization between service instances. The group MSA supports credential reset, hosts that are kept offline for some time, and seamless management of member host group management for all instances of a service.

 You can deploy single-identity server farms/clusters on Windows Server 2012 to which domain clients can authenticate without knowing which instance of a server farm/cluster they are connecting.

 You can configure services by using Service Control Manager to use a shared domain identity that automatically manages passwords.

 As soon as the group MSA is created, a domain administrator can delegate management of the group MSA to a service administrator.

 You can deploy single identity server farms/clusters on Windows Server 2012 servers running Windows 8 for identities in mixed-mode domains.

DirectAccess and remote access

Windows Server 2012 provides an integrated remote access solution that is easier to deploy and manage when compared to earlier versions that relied on multiple tools and consoles. Employees can gain access to corporate network resources while they work remotely, and IT administrators can manage corporate computers in Active Directory that are located outside the internal network. Windows Server 2012 accomplishes this by integrating two existing remote access technologies: DirectAccess for automatic, transparent connectivity, and traditional virtual private networks (VPNs) for compatibility.

DirectAccess was introduced in Windows 7 and Windows Server 2008 R2 to help remote users to more securely access shared resources, websites, and applications on an internal network without connecting to a VPN. DirectAccess establishes bidirectional connectivity with an organization’s corporate network every time a DirectAccess-enabled computer is connected to the Internet. Users never have to think about connecting to the corporate network, and IT administrators can manage remote computers outside the office, even when the computers are not connected to the VPN. Windows Server 2012 continues to offer

Windows Server 2012: Identity and Access

21

this transparent connection to the corporate network, with improvements around deployment, management, performance, and scalability. Remote access improvements in Windows Server 2012 include:

Integrated remote access: DirectAccess and VPN can be configured together in the Remote Access Management console by using a single wizard. The new role allows easier migration of Windows 7 Routing and Remote Access service (RRAS) and DirectAccess deployments.

Cross-premises connectivity. Windows Server 2012 provides a highly cloud-optimized operating system. VPN site-to-site functionality in remote access provides cross-premises connectivity between enterprises and hosting service providers, including Azure.

Improved management experience: By using the new Remote Access Management console, you can configure, manage, and monitor multiple DirectAccess and VPN remote access servers in a single location. The console provides a dashboard that allows you to view information about server and client activity.

Simplified deployment: In simple deployments, you can configure DirectAccess without being required to set up a certificate infrastructure. DirectAccess clients can now authenticate themselves by using only Active Directory credentials; no computer certificate is required.

New deployment scenarios. Remote access in Windows Server 2012 includes integrated deployment for several scenarios that required manual configuration in Windows Server 2008 R2. These include force tunneling (which sends all traffic through the DirectAccess connection), Network Access Protection (NAP) compliance, support for locating the nearest remote access server from DirectAccess clients in different geographical locations, and deploying DirectAccess for only remote management.

Improved scalability: Remote access in Windows Server 2012 offers several scalability improvements than support more users while providing better performance and lower costs. These include support for network load balancing, better performance in virtualized environments, and underlying platform improvements.

Integrated remote access

With DirectAccess, users who have an Internet connection can more securely access corporate resources, such as email servers, shared folders, or internal websites, with the experience of being easily connected to the corporate network.

DirectAccess transparently connects client computers to the internal network whenever the computer connects to the Internet, even before the user logs on, as shown in the following figure. This transparent, automatic connectivity means that access is provided without additional steps or configuration required by the user.

Windows Server 2012: Identity and Access

22

Figure 8: DirectAccess connection architecture

DirectAccess also lets you easily monitor connections and remotely manage DirectAccess client computers on the Internet.

At the same time, RRAS provides traditional client VPN connectivity for unmanaged client computers, such as computers running client operating systems earlier than Windows 7. In addition, RRAS site-to-site VPN provides connectivity between VPN servers, as shown in the following figure.

Figure 9: VPN connection architecture

The remote access server role in Windows Server 2012 integrates DirectAccess and RRAS VPN. You can configure DirectAccess and VPNs together in the Remote Access Management console by using a single wizard. You also can configure other RRAS features by using the legacy RRAS management console. The new role allows easier migration of Windows 7 RRAS and DirectAccess deployments, and it provides new features and improvements.

Windows Server 2012: Identity and Access

23

Windows Server 2012

Cross-premises connectivity

Windows Server 2012 provides an operating system that is highly optimized for the cloud. VPN site-to-site functionality in remote access provides cross-premises connectivity between enterprises and hosting service providers. Cross-premises connectivity enables organizations to connect to private subnetworks in a hosted cloud network. It also enables connectivity between geographically separate enterprise locations.

With cross-premises connectivity, you can use existing networking equipment to connect to hosting providers by using the industry standard Internet Key Exchange version 2 (IKEv2) and IP security (IPsec) protocol.

Figure 10 demonstrates how these two organizations implement cross-premises deployments by using Windows Server 2012.

Figure 10: Example of a cross-premises deployment

Windows Server 2012: Identity and Access

24

The following steps describe the procedures that Contoso and Woodgrove use for the cross-premises deployment shown in Figure 10:

1. Contoso.com and Woodgrove.com offload some of their enterprise infrastructure in a hosted cloud.

2. The hosting provider provides private clouds for each organization.

3. In the hosted cloud, virtual machines running Windows Server 2012 are configured as remote access servers running site-to-site VPN.

4. In each hosted private cloud, a cluster of two or more remote access servers is deployed to provide continuous availability and failover.

5. Contoso.com has two branch office locations. In each location, a Windows Server 2012 remote access server is deployed to provide a cross-premises connectivity solution to the hosted cloud and between the branch offices.

6. The Contoso.com branch office computers running the unified remote access server role in Windows Server 2012 are also configured as DirectAccess servers in a multisite deployment. DirectAccess clients can access any resource in the Contoso.com public cloud or Contoso.com branch offices from nearly any location on the Internet.

7. Woodgrove.com can use existing routers to connect to the hosted cloud because cross-premises functionality in Windows Server 2012 complies with IKEv2 and IPsec standards.

 

Improved management experience

By using the new Remote Access Management console, you can configure, manage, and monitor multiple DirectAccess and VPN remote access servers in a single location. The console provides a dashboard that allows you to view information about server and client activity. You can also generate reports for additional, more detailed information. Operations status provides comprehensive monitoring information about specific server components. Event logs and tracing help diagnose specific issues. By using client monitoring, you can see detailed views of connected users and computers, and you can even monitor which resources the clients are accessing. Accounting data can be logged to a local database or a Remote Authentication Dial-In User Service (RADIUS) server.

In addition to the Remote Access Management console, you can use Windows PowerShell command-line interface tools and automated scripts for remote access setup, configuration, management, monitoring, and troubleshooting.

On client computers, users can access the Network Connectivity Assistant application, integrated with Windows Network Connection Manager, to see a concise view of the DirectAccess connection status and links to corporate help resources, diagnostics tools, and troubleshooting information. Users can also enter one-time password (OTP) credentials if OTP authentication for DirectAccess is configured.

Easier deployment

The enhanced installation and configuration design in Windows Server 2012 allows you to set up a working deployment without changing your internal networking infrastructure. In simple deployments, you can configure DirectAccess without setting up a certificate infrastructure. DirectAccess clients can now authenticate themselves by using only Active Directory credentials; no computer certificate is required. In

Windows Server 2012: Identity and Access

25

addition, you can choose to use a self-signed certificate created automatically by DirectAccess for IP-HTTPS and for authentication of the network location server.

To further simplify deployment, DirectAccess in Windows Server 2012 supports access to internal servers that are running IPv4 only. An IPv6 infrastructure is not required for DirectAccess deployment.

Improved deployment scenarios

The remote access server role in Windows Server 2012 includes additional enhancements, including integrated deployment for several scenarios.

 With Windows Server 2012, you can now configure a DirectAccess server with two network adapters at the network edge or behind an edge device, or with a single network adapter running behind a firewall or network address translation device. The ability to use a single adapter removes the requirement to have dedicated public IPv4 addresses for DirectAccess deployment. With this configuration, clients connect to the DirectAccess server by using IP-HTTPS.

 In Windows Server 2012, you can configure remote access servers in a multisite deployment that allows users in dispersed geographical locations to connect to the multisite entry point closest to them. You can distribute and balance traffic across the multisite deployment by using an external global load balancer. To support fault tolerance, redundancy, and scalability, DirectAccess servers can now be deployed in a cluster configuration that uses Windows load balancer or an external hardware load balancer.

 DirectAccess in Windows Server 2012 adds support for two-factor authentication that uses an one-time password (OTP). For two-factor smart card authentication; Windows Server 2012 supports the use of Trusted Platform Module (TPM)-based virtual smart card capabilities that are available in Windows 8. The TPM of client computers can act as a virtual smart card for two-factor authentication, which reduces the overhead and costs incurred in smart card deployment.

 Windows Server 2012 also introduces the ability of computers to join an Active Directory domain and receive domain settings remotely via the Internet. By using this capability, you will find that deployment of new computers in remote offices and provisioning of client settings to DirectAccess clients is easier. You can configure client computers running Windows 8, Windows 7, and Windows Server 2008 R2 as DirectAccess clients. Clients running Windows 8 have access to all DirectAccess features, and they have an improved experience when connecting from behind a proxy server that requires authentication. Clients not running Windows 8 have the following limitations:

o They must download and install the DirectAccess Connectivity Assistant tool.

o They require a computer certificate for authentication.

o In a multisite deployment, they must be configured to always connect through the same entry point.

Scalability improvements

Remote access offers several scalability improvements, including support for more users with better performance and lower costs:

Windows Server 2012: Identity and Access

26

You can cluster multiple remote access servers for load balancing, continuous availability, and failover. Cluster traffic can be load-balanced by using NLB or a third-party load balancer. Servers can be added to or removed from the cluster with few interruptions to the connections in progress.

 The remote access server role takes advantage of Single Root I/O Virtualization (SR-IOV) for improved I/O performance when running on a virtual machine. In addition, remote access improves the overall scalability of the server host with support for IPsec hardware offload capabilities, available on many server interface cards that perform packet encryption and decryption in hardware.

 Optimization improvements in IP-HTTPS use the encryption that IPsec provides. This optimization, combined with the removal of the Secure Sockets Layer (SSL) encryption requirement, increases scalability and performance.

With the new DirectAccess and remote access enhancements, you can easily provide more secure remote access connections for your users, as well as log reports for monitoring and troubleshooting those connections. The new features in Windows Server 2012 support deployments in dispersed geographical locations, improved scalability with continuous availability, and improved performance in virtualized environments.

Summary

Identity and access control are two areas that require critical attention from IT professionals, particularly when you move to virtualized and private or public cloud environments. Windows Server 2012 makes these tasks easier by offering simple but powerful new and enhanced features to provide intelligent, auditable security; easier deployment and management of Active Directory Domain Services; and secure, always-on connectivity to the corporate resources.

For more information, see the Windows Server 2012 website at: http://www.microsoft.com/windowsserver2012/.

Windows Server 2012: Identity and Access

27

List of charts, tables, and figures

Figure 1: CAP components ………………………………………………………………………………………………………………10

Figure 2: CAP structure ……………………………………………………………………………………………………………………10

Figure 3: Combining multiple policies into policy lists and applying them to resources ……………………..11

Figure 4: Access-denied remediation ……………………………………………………………………………………………….12

Figure 5: Central auditing workflow …………………………………………………………………………………………………14

Figure 6: Classification-based RMS protection ………………………………………………………………………………….15

Figure 7: Windows Server 2012 Windows PowerShell History viewer ………………………………………………18

Figure 8: DirectAccess connection architecture ………………………………………………………………………………..22

Figure 9: VPN connection architecture …………………………………………………………………………………………….22

Figure 10: Example of a cross-premises deployment ………………………………………………………………………..23

Microsoft Press eBook Introducing Microsoft SQL Server 2012

Posted on by escapebusinesssolutions in Uncategorized

 

Contents

 

Introduction. xv

 

PART 1 DATABASE ADMINISTRATION

 

Chapter 1 SQL Server 2012 Editions and Engine Enhancements 3

 

SQL Server 2012 Enhancements for Database Administrators. 4

 

Availability Enhancements. 4

 

Scalability and Performance Enhancements. 6

 

Manageability Enhancements. 7

 

Security Enhancements. 10

 

Programmability Enhancements. 11

 

SQL Server 2012 Editions. 12

 

Enterprise Edition. 12

 

Standard Edition. 13

 

Business Intelligence Edition. 14

 

Specialized Editions. 15

 

SQL Server 2012 Licensing Overview. 15

 

Hardware and Software Requirements. 16

 

Installation, Upgrade, and Migration Strategies. 17

 

The In-Place Upgrade. 17

 

Side-by-Side Migration. 19

 

What do you think of this book? We want to hear from you!

 

Microsoft is interested in hearing your feedback so we can continually improve our books and learning

resources for you. To participate in a brief online survey, please visit:

 

microsoft.com/learning/booksurvey

 

 

 

viii Contents

 

Chapter 2 High-Availability and Disaster-Recovery

Enhancements 21

 

SQL Server AlwaysOn: A Flexible and Integrated Solution. 21

 

AlwaysOn Availability Groups . 23

 

Understanding Concepts and Terminology . 24

 

Configuring Availability Groups . 29

 

Monitoring Availability Groups with the Dashboard. 31

 

Active Secondaries. 32

 

Read-Only Access to Secondary Replicas . 33

 

Backups on Secondary . 33

 

AlwaysOn Failover Cluster Instances. 34

 

Support for Deploying SQL Server 2012 on Windows Server Core . 36

 

SQL Server 2012 Prerequisites for Server Core. 37

 

SQL Server Features Supported on Server Core. 38

 

SQL Server on Server Core Installation Alternatives . 38

 

Additional High-Availability and Disaster-Recovery Enhancements. 39

 

Support for Server Message Block . 39

 

Database Recovery Advisor. 39

 

Online Operations. 40

 

Rolling Upgrade and Patch Management. 40

 

Chapter 3 Performance and Scalability 41

 

Columnstore Index Overview. 41

 

Columnstore Index Fundamentals and Architecture. 42

 

How Is Data Stored When Using a Columnstore Index?. 42

 

How Do Columnstore Indexes Significantly Improve the

Speed of Queries? . 44

 

Columnstore Index Storage Organization. 45

 

Columnstore Index Support and SQL Server 2012. 46

 

Columnstore Index Restrictions. 46

 

Columnstore Index Design Considerations and Loading Data. 47

 

When to Build a Columnstore Index. 47

 

When Not to Build a Columnstore Index. 48

 

Loading New Data. 48

 

 

 

Beyond-Relational Example . 76

 

FILESTREAM Enhancements. 76

 

FileTable . 77

 

FileTable Prerequisites. 78

 

Creating a FileTable. 80

 

Managing FileTable. 81

 

Full-Text Search. 81

 

Statistical Semantic Search. 82

 

Configuring Semantic Search. 83

 

Semantic Search Examples. 85

 

Spatial Enhancements . 86

 

Spatial Data Scenarios . 86

 

Spatial Data Features Supported in SQL Server . 86

 

Spatial Type Improvements. 87

 

Additional Spatial Improvements. 89

 

Extended Events. 90

 

PART 2 BUSINESS INTELLIGENCE DEVELOPMENT

 

Chapter 6 Integration Services 93

 

Developer Experience . 93

 

Add New Project Dialog Box. 93

 

General Interface Changes. 95

 

Getting Started Window. 96

 

SSIS Toolbox. 97

 

Shared Connection Managers. 98

 

Scripting Engine. 99

 

Expression Indicators. 100

 

Undo and Redo . 100

 

Package Sort By Name. 100

 

Status Indicators. 101

 

Control Flow. 101

 

Expression Task. 101

 

Execute Package Task. 102

 

 

 

Data Flow . 103

 

Sources and Destinations. 103

 

Transformations. 106

 

Column References. 108

 

Collapsible Grouping. 109

 

Data Viewer. 110

 

Change Data Capture Support. 111

 

CDC Control Flow. 112

 

CDC Data Flow. 113

 

Flexible Package Design. 114

 

Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115

 

Expressions. 115

 

Deployment Models. 116

 

Supported Deployment Models. 116

 

Project Deployment Model Features. 118

 

Project Deployment Workflow . 119

 

Parameters. 122

 

Project Parameters. 123

 

Package Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124

 

Parameter Usage. 124

 

Post-Deployment Parameter Values. 125

 

Integration Services Catalog. 128

 

Catalog Creation . 128

 

Catalog Properties. 129

 

Environment Objects. 132

 

Administration. 135

 

Validation. 135

 

Package Execution. 135

 

Logging and Troubleshooting Tools. 137

 

Security . 139

 

Package File Format. 139

 

 

 

Miscellaneous Changes. 197

 

SharePoint Integration . 197

 

Metadata. 197

 

Bulk Updates and Export . 197

 

Transactions . 198

 

Windows PowerShell. 198

 

Chapter 9 Analysis Services and PowerPivot 199

 

Analysis Services. 199

 

Server Modes. 199

 

Analysis Services Projects. 201

 

Tabular Modeling. 203

 

Multidimensional Model Storage. 215

 

Server Management . 215

 

Programmability. 217

 

PowerPivot for Excel. 218

 

Installation and Upgrade . 218

 

Usability. 218

 

Model Enhancements. 221

 

DAX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222

 

PowerPivot for SharePoint. 224

 

Installation and Configuration. 224

 

Management. 224

 

Chapter 10 Reporting Services 229

 

New Renderers . 229

 

Excel 2010 Renderer . 229

 

Word 2010 Renderer. 230

 

SharePoint Shared Service Architecture. 230

 

Feature Support by SharePoint Edition. 230

 

Shared Service Architecture Benefits. 231

 

Service Application Configuration . 231

 

 

 

Power View. 232

 

Data Sources. 233

 

Power View Design Environment . 234

 

Data Visualization . 237

 

Sort Order. 241

 

Multiple Views . 241

 

Highlighted Values. 242

 

Filters. 243

 

Display Modes . 245

 

PowerPoint Export. 246

 

Data Alerts. 246

 

Data Alert Designer. 246

 

Alerting Service . 248

 

Data Alert Manager. 249

 

Alerting Configuration . 250

 

Index 251

 

What do you think of this book? We want to hear from you!

 

Microsoft is interested in hearing your feedback so we can continually improve our books and learning

resources for you. To participate in a brief online survey, please visit:

 

microsoft.com/learning/booksurvey

 

 

 

3

 

CHAP TE R 1

 

SQL Server 2012 Editions and

Engine Enhancements

 

SQL Server 2012 is Microsoft’s latest cloud-ready information platform. Organizations can use SQL

Server 2012 to efficiently protect, unlock, and scale the power of their data across the desktop,

mobile device, datacenter, and either a private or public cloud. Building on the success of the SQL

Server 2008 R2 release, SQL Server 2012 has made a strong impact on organizations worldwide with

its significant capabilities. It provides organizations with mission-critical performance and availability,

as well as the potential to unlock breakthrough insights with pervasive data discovery across the

organization. Finally, SQL Server 2012 delivers a variety of hybrid solutions you can choose from. For

example, an organization can develop and deploy applications and database solutions on traditional

nonvirtualized environments, on appliances, and in on-premises private clouds or off-premises public

clouds. Moreover, these solutions can easily integrate with one another, offering a fully integrated

hybrid solution. Figure 1-1 illustrates the Cloud Ready Information Platform ecosystem.

 

Hybrid IT

Traditional

Nonvirtualized

Private

Cloud

On-premises cloud

Public

Cloud

Off-premises cloud

Nonvirtualized

applications

Pooled (Virtualized)

Elastic

Self-service

Usage-based

Pooled (Virtualized)

Elastic

Self-service

Usage-based

Managed Services

 

FIGURE 1-1 SQL Server 2012, cloud-ready information platform

 

To prepare readers for SQL Server 2012, this chapter examines the new SQL Server 2012 features,

capabilities, and editions from a database administrator’s perspective. It also discusses SQL Server

2012 hardware and software requirements and installation strategies.

 

 

 

4 PART 1 Database Administration

 

SQL Server 2012 Enhancements for Database Administrators

 

Now more than ever, organizations require a trusted, cost-effective, and scalable database platform

that offers mission-critical confidence, breakthrough insights, and flexible cloud-based offerings.

These organizations face ever-changing business conditions in the global economy and challenges

such as IT budget constraints, the need to stay competitive by obtaining business insights, and

the ability to use the right information at the right time. In addition, organizations must always be

adjusting because new and important trends are regularly changing the way software is developed

and deployed. Some of these new trends include data explosion (enormous increases in data usage),

consumerization IT, big data (large data sets), and private and public cloud deployments.

 

Microsoft has made major investments in the SQL Server 2012 product as a whole; however, the

new features and breakthrough capabilities that should interest database administrators (DBAs) are

divided in the chapter into the following categories: Availability, Manageability, Programmability,

Scalability and Performance, and Security. The upcoming sections introduce some of the new features

and capabilities; however, other chapters in this book conduct a deeper explanation of the major

technology investments.

 

Availability Enhancements

 

A tremendous amount of high-availability enhancements were added to SQL Server 2012, which is

sure to increase both the confidence organizations have in their databases and the maximum uptime

for those databases. SQL Server 2012 continues to deliver database mirroring, log shipping, and replication.

However, it now also offers a new brand of technologies for achieving both high availability

and disaster recovery known as AlwaysOn. Let’s quickly review the new high-availability enhancement

AlwaysOn:

 

¦ ¦ AlwaysOn Availability Groups For DBAs, AlwaysOn Availability Groups is probably the

most highly anticipated feature related to the Database Engine for DBAs. This new capability

protects databases and allows for multiple databases to fail over as a single unit. Better data

redundancy and protection is achieved because the solution supports up to four secondary

replicas. Of these four secondary replicas, up to two secondaries can be configured as synchronous

secondaries to ensure the copies are up to date. The secondary replicas can reside

within a datacenter for achieving high availability within a site or across datacenters for disaster

recovery. In addition, AlwaysOn Availability Groups provide a higher return on investment

because hardware utilization is increased as the secondaries are active, readable, and can be

leveraged to offload backups, reporting, and ad hoc queries from the primary replica. The

solution is tightly integrated into SQL Server Management Studio, is straightforward to deploy,

and supports either shared storage or local storage.

 

 

Figure 1-2 illustrates an organization with a global presence achieving both high availability

and disaster recovery for mission-critical databases using AlwaysOn Availability Groups. In

addition,

the secondary replicas are being used to offload reporting and backups.

 

 

 

CHAPTER 1 SQL Server 2012 Editions and Engine Enhancements 5

 

70%

50%

25%

15%

70%

50%

25%

15%

Primary

Datacenter

Replica2

Replica3

Reports

Backups

Reports

Backups

Secondary

Datacenter

Replica4

Synchronous Data Movement

Asynchronous Data Movement

A

A

Secondary Replica

Primary Replica

A

A

A

A

Replica1

 

FIGURE 1-2 AlwaysOn Availability Groups for an organization with a global presence

 

¦ ¦ AlwaysOn Failover Cluster Instances (FCI) AlwaysOn Failover Cluster Instances provides

superior instance-level protection using Windows Server Failover Clustering and shared

storage.

However, with SQL Server 2012 there are a tremendous number of enhancements to

improve availability and reliability. First, FCI now provides support for multi-subnet failover

clusters. These subnets, where the FCI nodes reside, can be located in the same datacenter

or in geographically dispersed sites. Second, local storage can be leveraged for the TempDB

database.

Third, faster startup and recovery times are achieved after a failover transpires.

Finally, improved cluster health-detection policies can be leveraged, offering a stronger and

more flexible failover.

¦ ¦ Support for Windows Server Core Installing SQL Server 2012 on Windows Server Core

is now supported. Windows Server Core is a scaled-down edition of the Windows operating

system and requires approximately 50 to 60 percent fewer reboots when patching servers.

 

 

 

 

This translates to greater SQL Server uptime and increased security. Server Core deployment

options using Windows Server 2008 R2 SP1 and higher are required. Chapter 2, “High-Availability

and Disaster-Recovery Options,” discusses deploying SQL Server 2012 on Server

Core, including the features supported.

¦ ¦ Recovery Advisor A new visual timeline has been introduced in SQL Server Management

Studio to simplify the database restore process. As illustrated in Figure 1-3, the scroll bar

beneath

the timeline can be used to specify backups to restore a database to a point in time.

 

 

 

 

FIGURE 1-3 Recovery Advisor visual timeline

 

Note For detailed information about the AlwaysOn technologies and other high-availability

enhancements, be sure to read Chapter 2.

 

Scalability and Performance Enhancements

 

The SQL Server product group has made sizable investments in improving scalability and

performance

associated with the SQL Server Database Engine. Some of the main enhancements that

allow organizations to improve their SQL Server workloads include the following:

 

¦ ¦ Columnstore Indexes More and more organizations have a requirement to deliver

breakthrough

and predictable performance on large data sets to stay competitive. SQL

Server 2012 introduces a new in-memory, columnstore index built directly in the relational

engine. Together with advanced query-processing enhancements, these technologies provide

blazing-fast performance and improve queries associated with data warehouse workloads

by 10 to 100 times. In some cases, customers have experienced a 400 percent improvement

in performance. For more information on this new capability for data warehouse workloads,

review Chapter 3, “Blazing-Fast Query Performance with Columnstore Indexes.”

 

 

 

 

¦ ¦ Partition Support Increased To dramatically boost scalability and performance associated

with large tables and data warehouses, SQL Server 2012 now supports up to 15,000 partitions

per table by default. This is a significant increase from the previous version of SQL Server,

which was limited to 1000 partitions by default. This new expanded support also helps enable

large sliding-window scenarios for data warehouse maintenance.

¦ ¦ Online Index Create, Rebuild, and Drop Many organizations running mission-critical

workloads use online indexing to ensure their business environment does not experience

downtime during routine index maintenance. With SQL Server 2012, indexes containing

varchar(max), nvarchar(max), and varbinary(max) columns can now be created, rebuilt, and

dropped as an online operation. This is vital for organizations that require maximum uptime

and concurrent user activity during index operations.

¦ ¦ Achieve Maximum Scalability with Windows Server 2008 R2 Windows Server 2008 R2

is built to achieve unprecedented workload size, dynamic scalability, and across-the-board

availability and reliability. As a result, SQL Server 2012 can achieve maximum scalability when

running on Windows Server 2008 R2 because it supports up to 256 logical processors and

2 terabytes of memory in a single operating system instance.

 

 

Manageability Enhancements

 

SQL Server deployments are growing more numerous and more common in organizations. This fact

demands that all database administrators be prepared by having the appropriate tools to successfully

manage their SQL Server infrastructure. Recall that the previous releases of SQL Server included

many new features tailored toward manageability. For example, database administrators could easily

leverage Policy Based Management, Resource Governor, Data Collector, Data-tier applications, and

Utility Control Point. Note that the product group responsible for manageability never stopped

investing in manageability. With SQL Server 2012, they unveiled additional investments in SQL Server

tools and monitoring features. The following list articulates the manageability enhancements in SQL

Server 2012:

 

¦ ¦ SQL Server Management Studio With SQL Server 2012, IntelliSense and Transact-SQL

debugging

have been enhanced to bolster the development experience in SQL Server Management

Studio.

¦ ¦ IntelliSense Enhancements A completion list will now suggest string matches based on

partial words, whereas in the past it typically made recommendations based on the first character.

¦ ¦ A new Insert Snippet menu This new feature is illustrated in Figure 1-4. It offers developers

a categorized list of snippets to choose from to streamline code. The snippet picket tooltip can

be launched by pressing CTRL+K, pressing CTRL+X, or selecting it from the Edit menu.

¦ ¦ Transact-SQL Debugger This feature introduces the potential to debug Transact-SQL

scripts on instances of SQL Server 2005 Service Pack 2 (SP2) or later and enhances breakpoint functionality.

 

 

 

 

 

 

FIGURE 1-4 Leveraging the Transact-SQL code snippet template as a starting point when writing new

Transact-SQL statements in the SQL Server Database Engine Query Editor

 

¦ ¦ Resource Governor Enhancements Many organizations currently leverage Resource

Governor to gain predictable performance and improve their management of SQL Server

workloads and resources by implementing limits on resource consumption based on incoming

requests. In the past few years, customers have also been requesting additional improvements

to the Resource Governor feature. Customers wanted to increase the maximum number of

resource pools and support large-scale, multitenant database solutions with a higher level of

isolation between workloads. They also wanted predictable chargeback and vertical isolation

of machine resources.

 

 

The SQL Server product group responsible for the Resource Governor feature introduced

new capabilities to address the requests of its customers and the SQL Server community. To

begin, support for larger scale multitenancy can now be achieved on a single instance of SQL

Server because the number of resource pools Resource Governor supports increased from 20

to 64. In addition, a maximum cap for CPU usage has been introduced to enable predictable

chargeback

and isolation on the CPU. Finally, resource pools can be affinitized to an individual

schedule or a group of schedules for vertical isolation of machine resources.

 

A new Dynamic Management View (DMV) called sys.dm_resource_governor_resource_pool_

affinity improves database administrators’ success in tracking resource pool affinity.

 

Let’s review an example of some of the new Resource Governor features in action. In the

following

example, resource pool Pool25 is altered to be affinitized to six schedulers (8, 12,

13, 14, 15, and 16), and it’s guaranteed a minimum 5 percent of the CPU capacity of those

schedulers.

It can receive no more than 80 percent of the capacity of those schedulers. When

there is contention for CPU bandwidth,

the maximum average CPU bandwidth that will be

allocated is 40 percent.

 

 

 

ALTER RESOURCE POOL Pool25

WITH(

MIN_CPU_PERCENT = 5,

MAX_CPU_PERCENT = 40,

CAP_CPU_PERCENT = 80,

AFFINITY SCHEDULER = (8, 12 TO 16),

MIN_MEMORY_PERCENT = 5,

MAX_MEMORY_PERCENT = 15,

);

 

¦ ¦ Contained Databases Authentication associated with database portability was a challenge

in the previous versions of SQL Server. This was the result of users in a database being associated

with logins on the source instance of SQL Server. If the database ever moved to another

instance of SQL Server, the risk was that the login might not exist. With the introduction of

contained databases in SQL Server 2012, users are authenticated directly into a user database

without the dependency of logins in the Database Engine. This feature facilitates better

portability of user databases among servers because contained databases have no external

dependencies.

¦ ¦ Tight Integration with SQL Azure A new Deploy Database To SQL Azure wizard, pictured

in Figure 1-5, is integrated in the SQL Server Database Engine to help organizations deploy

an on-premise database to SQL Azure. Furthermore, new scenarios can be enabled with SQL

Azure Data Sync, which is a cloud service that provides bidirectional data synchronization

between databases across the datacenter and cloud.

 

 

 

 

FIGURE 1-5 Deploying a database to SQL Azure with the Deploy Database Wizard

 

 

 

¦ ¦ Startup Options Relocated Within SQL Server Configuration Manager, a new Startup

Parameters

tab was introduced for better manageability of the parameters required for

startup. A DBA can now easily specify startup parameters compared to previous versions of

SQL Server, which at times was a tedious task. The Startup Parameters tab can be invoked

by right-clicking a SQL Server instance name in SQL Server Configuration Manager and then

selecting Properties.

¦ ¦ Data-Tier Application (DAC) Enhancements SQL Server 2008 R2 introduced the concept

of data-tier applications. A data-tier application is a single unit of deployment containing

all of the database’s schema, dependent objects, and deployment requirements used by

an application.

SQL Server 2012 introduces a few enhancements to DAC. With the new SQL

Server, DAC upgrades are performed in an in-place fashion compared to the previous side-byside

upgrade process we’ve all grown accustomed to over the years. Moreover, DACs can be

deployed, imported and exported more easily across premises and public cloud environments,

such as SQL Azure. Finally, data-tier applications now support many more objects compared to

the previous SQL Server release.

 

 

Security Enhancements

 

It has been approximately 10 years since Microsoft initiated its trustworthy computing initiative. Since

then, SQL Server has had the best track record with the least amount of vulnerabilities and exposures

among the major database players in the industry. The graph shown in Figure 1-6 is from the National

Institute of Standards and Technology (Source: ITIC 2011: SQL Server Delivers Industry-Leading Security).

It shows common vulnerabilities and exposures reported from January 2002 to June 2010.

 

Oracle

0

50

100

150

200

250

300

350

DB2 MySQL SQL Server

 

FIGURE 1-6 Common vulnerabilities and exposures reported to NIST from January 2002 to January 2010

 

With SQL Server 2012, the product continues to expand on this solid foundation to deliver

enhanced

security and compliance within the database platform. For detailed information

of all the security enhancements associated with the Database Engine, review Chapter 4,

 

 

 

”

Security Enhancements.”

For now, here is a snapshot of some of the new enterprise-ready security

capabilities and controls that enable organizations to meet strict compliance policies and regulations:

 

¦ ¦ User-defined server roles for easier separation of duties

¦ ¦ Audit enhancements to improve compliance and resiliency

¦ ¦ Simplified security management, with a default schema for groups

¦ ¦ Contained Database Authentication, which provides database authentication that uses

self-

contained access information without the need for server logins

¦ ¦ SharePoint and Active Directory security models for higher data security in end-user reports

 

 

Programmability Enhancements

 

There has also been a tremendous investment in SQL Server 2012 regarding programmability.

Specifically,

there is support for “beyond relational” elements such as XML, Spatial, Documents, Digital

Media, Scientific Records, factoids, and other unstructured data types. Why such investments?

 

Organizations have demanded they be given a way to reduce the costs associated with managing

both structured and nonstructured data. They wanted to simplify the development of applications

over all data, and they wanted the management and search capabilities for all data improved. Take a

minute to review some of the SQL Server 2012 investments that positively impact programmability.

For more information associated with programmability and beyond relational elements, please review

Chapter 5, “Programmability and Beyond-Relational Enhancements.”

 

¦ ¦ FileTable Applications typically store data within a relational database engine; however, a

myriad of applications also maintain the data in unstructured formats, such as documents,

media files, and XML. Unstructured data usually resides on a file server and not directly in

a relational database such as SQL Server. As you can imagine, it becomes challenging for

organizations

to not only manage their structured and unstructured data across these disparate

systems, but to also keep them in sync. FileTable, a new capability in SQL Server 2012,

addresses these challenges. It builds on FILESTREAM technology that was first introduced

with SQL Server 2008. FileTable offers organizations Windows file namespace support and

application

compatibility with the file data stored in SQL Server. As an added bonus, when

applications

are allowed to integrate storage and data management within SQL Server, fulltext

and semantic search is achievable over unstructured and structured data.

¦ ¦ Statistical Semantic Search By introducing new semantic search functionality, SQL Server

2012 allows organizations to achieve deeper insight into unstructured data stored within the

Database Engine. Three new Transact-SQL rowset functions were introduced to query not only

the words in a document, but also the meaning of the document.

¦ ¦ Full-Text Search Enhancements Full-text search in SQL Server 2012 offers better query

performance and scale. It also introduces property-scoped searching functionality, which

allows

organizations the ability to search properties such as Author and Title without the need

 

 

 

 

Specialized Editions

 

Above and beyond the three main editions discussed earlier, SQL Server 2012 continues to deliver

specialized editions for organizations that have a unique set of requirements. Some examples include

the following:

 

¦ ¦ Developer The Developer edition includes all of the features and functionality found in the

Enterprise edition; however, it is meant strictly for the purpose of development, testing, and

demonstration. Note that you can transition a SQL Server Developer installation directly into

production by upgrading it to SQL Server 2012 Enterprise without reinstallation.

¦ ¦ Web Available at a much more affordable price than the Enterprise and Standard editions,

SQL Server 2012 Web is focused on service providers hosting Internet-facing web services

environments. Unlike the Express edition, this edition doesn’t have database size restrictions,

it supports four processors, and supports up to 64 GB of memory. SQL Server 2012 Web does

not offer the same premium features found in Enterprise and Standard editions, but it still

remains the ideal platform for hosting websites and web applications.

¦ ¦ Express This free edition is the best entry-level alternative for independent software

vendors, nonprofessional developers, and hobbyists building client applications. Individuals

learning about databases or learning how to build client applications will find that this edition

meets all their needs. This edition, in a nutshell, is limited to one processor and 1 GB of

memory, and it can have a maximum database size of 10 GB. Also, Express is integrated with

Microsoft Visual Studio.

 

 

Note Review “Features Supported by the Editions of SQL Server 2012” at

http://msdn.microsoft.com/en-us/library/cc645993(v=sql.110).aspx and

http://www.microsoft.com/sqlserver/en/us/future-editions/sql2012-editions.aspx for a

complete

comparison of the key capabilities of the different editions of SQL Server 2012.

 

SQL Server 2012 Licensing Overview

 

The licensing models affiliated with SQL Server 2012 have been both simplified to better align to

customer solutions and optimized for virtualization and cloud deployments. Organizations should

process knowledge of the information that follows. With SQL Server 2012, the licensing for computing

power is core-based and the Business Intelligence and Standard editions are available under

the Server + Client Access License (CAL) model. In addition, organizations can save on cloud-based

computing costs by licensing individual database virtual machines. Because each customer environment

is unique, we will not have the opportunity to provide an overview of how the license changes

affect your environment. For more information on the licensing changes and how they impact your

organization,

please contact your Microsoft representative or partner.

 

 

 

Software Component

 

Requirements

 

Windows PowerShell

 

Windows PowerShell 2.0

 

SQL Server support tools and

software

 

SQL Server 2012 – SQL Server Native Client

 

SQL Server 2012 – SQL Server Setup Support Files

 

Minimum: Windows Installer 4.5

 

Internet Explorer

 

Minimum: Windows Internet Explorer 7 or later version

 

Virtualization

 

Windows Server 2008 SP2 running Hyper-V role

 

or

 

Windows Server 2008 R2 SP1 running Hyper-V role

 

 

 

 

 

Note The server hardware has supported both 32-bit and 64-bit processors for several

years; however, Windows Server 2008 R2 is 64-bit only. Take this into serious consideration

when planning SQL Server 2012 deployments.

 

Installation, Upgrade, and Migration Strategies

 

Like its predecessors, SQL Server 2012 is available in both 32-bit and 64-bit editions. Both can be

installed

with either the SQL Server Installation Wizard through a command prompt or with Sysprep

for automated deployments with minimal administrator intervention. As mentioned earlier in the

chapter, SQL Server 2012 can now be installed on the Server Core, which is an installation option of

Windows Server 2008 R2 SP1 or later. Finally, database administrators also have the option to upgrade

an existing installation of SQL Server or conduct a side-by-side migration when installing SQL Server

2012. The following sections elaborate on the different strategies.

 

The In-Place Upgrade

 

An in-place upgrade is the upgrade of an existing SQL Server installation to SQL Server 2012.

When an in-place upgrade is conducted, the SQL Server 2012 setup program replaces the previous

SQL Server binaries with the new SQL Server 2012 binaries on the existing machine. SQL Server data is

automatically converted from the previous version to SQL Server 2012. This means data does not have

to be copied or migrated. In the example in Figure 1-8, a database administrator is conducting an

in-place upgrade on a SQL Server 2008 instance running on Server 1. When the upgrade is complete,

Server 1 still exists, but the SQL Server 2008 instance and all of its data is upgraded to SQL Server 2012.

 

Note SQL Server 2005 with SP4, SQL Server 2008 with SP2, and SQL Server 2008 R2 with

SP1 are all supported for an in-place upgrade to SQL Server 2012. Unfortunately, earlier

versions such as SQL Server 2000, SQL Server 7.0, and SQL Server 6.5 cannot be upgraded

to SQL Server 2012.

 

 

 

Side-by-Side Migration Pros and Cons

 

The greatest benefit of a side-by-side migration over an in-place upgrade is the opportunity to build

out a new database infrastructure on SQL Server 2012 and avoid potential migration issues that can

occur with an in-place upgrade. The side-by-side migration also provides more granular control

over the upgrade process because you can migrate databases and components independent of one

another. In addition, the legacy instance remains online during the migration process. All of these

advantages result in a more powerful server. Moreover, when two instances are running in parallel,

additional testing and verification can be conducted. Performing a rollback is also easy if a problem

arises during the migration.

 

However, there are disadvantages to the side-by-side strategy. Additional hardware might need to

be purchased. Applications might also need to be directed to the new SQL Server 2012 instance, and

it might not be a best practice for very large databases because of the duplicate amount of storage

required during the migration process.

 

SQL Server 2012 High-Level, Side-by-Side Strategy

 

The high-level, side-by-side migration strategy for upgrading to SQL Server 2012 consists of the following

steps:

 

1. Ensure the instance of SQL Server you plan to migrate meets the hardware and software

requirements for SQL Server 2012.

2. Review the deprecated and discontinued features in SQL Server 2012 by referring to

“Deprecated

Database Engine Features in SQL Server 2012″ at http://technet.microsoft.com

/en-us/library/ms143729(v=sql.110).aspx.

3. Although a legacy instance will not be upgraded to SQL Server 2012, it is still beneficial to

run the SQL Server 2012 Upgrade Advisor to ensure the data being migrated to the new SQL

Server 2012 is supported and there is no possibility of a break occurring after migration.

4. Procure the hardware, and install your operating system of choice. Windows Server 2012 is

recommended.

5. Install the SQL Server 2012 prerequisites and desired components.

6. Migrate objects from the legacy SQL Server to the new SQL Server 2012 database platform.

7. Point applications to the new SQL Server 2012 database platform.

8. Decommission legacy servers after the migration is complete.

 

 

 

 

21

 

CHAP TE R 2

 

High-Availability and Disaster-

Recovery Enhancements

 

Microsoft SQL Server 2012 delivers significant enhancements to well-known, critical capabilities

such as high availability (HA) and disaster recovery. These enhancements promise to assist

organizations

in achieving their highest level of confidence to date in their server environments.

Server

Core support,

breakthrough features such as AlwaysOn Availability Groups and active

secondaries,

and key improvements

to features such as failover clustering are improvements that

provide organizations a range of accommodating options to achieve maximum application availability

and data protection for SQL Server instances and databases within a datacenter and across

datacenters.

 

 

This chapter’s goal is to bring readers up to date with the high-availability and disaster-recovery

capabilities that are fully integrated into SQL Server 2012 as a result of Microsoft’s heavy investment

in AlwaysOn.

 

SQL Server AlwaysOn: A Flexible and Integrated Solution

 

Every organization’s success and service reputation is built on ensuring that its data is always

accessible

and protected. In the IT world, this means delivering a product that achieves the highest

level of availability and disaster recovery while minimizing data loss and downtime. With the

previous

versions of SQL Server, organizations achieved high availability and disaster recovery by

using technologies

such as failover clustering, database mirroring, log shipping, and peer-to-peer

replication.

Although organizations achieved great success with these solutions, they were tasked with

combining these native SQL Server technologies to achieve their business requirements related to

their Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

 

Figure 2-1 illustrates a common high-availability and disaster-recovery strategy used by

organizations

with the previous versions of SQL Server. This strategy includes failover clustering to

protect SQL Server instances within each datacenter, combined with asynchronous database mirroring

to provide disaster-recovery capabilities for mission-critical databases.

 

 

 

22 PART 1 Database Administration

 

Primary

Datacenter

Asynchronous Data Movement

with Database Mirroring

Secondary

Datacenter

SQL Server 2008 R2

Failover Cluster

SQL Server 2008 R2

Failover Cluster

 

FIGURE 2-1 Achieving high availability and disaster recovery by combining failover clustering with database

mirroring

in SQL Server 2008 R2

 

Likewise, for organizations that either required more than one secondary datacenter or that did

not have shared storage, their high-availability and disaster-recovery deployment incorporated

synchronous

database mirroring with a witness within the primary datacenter combined with log

shipping for moving data to multiple locations. This deployment strategy is illustrated in Figure 2-2.

 

Primary

Datacenter

Log Shipping

Disaster Recover

Datacenter 1

SQL Server 2008 R2

SQL Server 2008 R2

Database Mirroring

Witness

Disaster Recover

Datacenter 2

SQL Server 2008 R2

Synchronous Data Movement

with Database Mirroring

Log Shipping

Log Shipping

 

FIGURE 2-2 Achieving high availability and disaster recovery with database mirroring combined with log shipping

in SQL Server 2008 R2

 

 

 

CHAPTER 2 High-Availability and Disaster-Recovery Enhancements 23

 

Figures 2-1 and 2-2 both reveal successful solutions for achieving high availability and disaster

recovery. However, these solutions had some limitations, which warranted making changes. In addition,

with organizations constantly evolving, it was only a matter of time until they voiced their own

concerns and sent out a request for more options and changes.

 

One concern for many organizations was regarding database mirroring. Database mirroring is

a great way to protect databases; however, the solution is a one-to-one mapping, making multiple

secondaries unattainable. When confronted with this situation, many organizations reverted to log

shipping as a replacement for database mirroring because it supports multiple secondaries. Unfortunately,

organizations encountered limitations with log shipping because it did not provide zero data

loss or automatic failover capability. Concerns were also experienced by organizations working with

failover clustering because they felt that their shared-storage devices, such as a storage area network

(SAN), could be a single point of failure. Similarly, many organizations thought that from a cost

perspective their investments were not being used to their fullest potential. For example, the passive

servers in many of these solutions were idle. Finally, many organizations wanted to offload reporting

and maintenance tasks from the primary database servers, which was not an easy task to achieve.

 

SQL Server has evolved to answer many of these concerns, and this includes an integrated solution

called AlwaysOn. AlwaysOn Availability Groups and AlwaysOn Failover Cluster Instances are new

features,

introduced in SQL Server 2012, that are rich with options and promise the highest level of

availability and disaster recovery to its customers. At a high level, AlwaysOn Availability Groups is

used for database protection and offers multidatabase failover, multiple secondaries, active secondaries,

and integrated HA management. On the other hand, AlwaysOn Failover Cluster Instances is a

feature tailored to instance-level protection, multisite clustering, and consolidation, while consistently

providing flexible failover polices and improved diagnostics.

 

AlwaysOn Availability Groups

 

AlwaysOn Availability Groups provides an enterprise-level alternative to database mirroring, and it

gives organizations the ability to automatically or manually fail over a group of databases as a single

unit, with support for up to four secondaries. The solution provides zero-data-loss protection and is

flexible. It can be deployed on local storage or shared storage, and it supports both synchronous and

asynchronous data movement. The application failover is very fast, it supports automatic page repair,

and the secondary replicas can be leveraged to offload reporting and a number of maintenance

tasks, such as backups.

 

Take a look at Figure 2-3, which illustrates an AlwaysOn Availability Groups deployment strategy

that includes one primary replica and three secondary replicas.

 

 

 

70%

50%

25%

15%

70%

50%

25%

15%

Primary

Datacenter

Replica2

Replica3

Reports

Reports Backups

Backups

Secondary

Datacenter

Synchronous Data Movement

Asynchronous Data Movement

Replica4

A

A

A

A

A

Secondary Replica

Primary Replica

A

SQL Server

SQL Server

SQL Server

SQL Server

Replica1

 

FIGURE 2-3 Achieving high availability and disaster recovery with AlwaysOn Availability Groups

 

In this figure, synchronous data movement is used to provide high availability within the primary

datacenter and asynchronous data movement is used to provide disaster recovery. Moreover, secondary

replica 3 and replica 4 are employed to offload reports and backups from the primary replica.

 

It is now time to take a deeper dive into AlwaysOn Availability Groups through a review of the new

concepts and terminology associated with this breakthrough capability.

 

Understanding Concepts and Terminology

 

Availability groups are built on top of Windows Failover Clustering and support both shared and

nonshared

storage. Depending on an organization’s RPO and RTO requirements, availability groups

can use either an asynchronous-commit availability mode or a synchronous-commit availability

mode to move data between primary and secondary replicas. Availability groups include built-in

compression

and encryption as well as support for file-stream replication and auto page repair.

Failover between replicas is either automatic or manual.

 

When deploying AlwaysOn Availability Groups, your first step is to deploy a Windows Failover

Cluster. This is completed by using the Failover Cluster Manager Snap-in within Windows Server 2008

R2. Once the Windows Failover Cluster is formed, the remainder of the Availability Groups configurations

is completed in SQL Server Management Studio. When you use the Availability Groups wizards

 

 

 

to configure your availability groups, SQL Server Management Studio automatically creates the

appropriate

services, applications, and resources in Failover Cluster Manager; hence, the deployment

is much easier for database administrators who are not familiar with failover clustering.

 

Now that the fundamentals of the AlwaysOn Availability Groups have been laid down, the most

natural question that follows is how an organization’s operations are enhanced with this feature.

Unlike

database mirroring, which supports only one secondary, AlwaysOn Availability Groups supports

one primary replica and up to four secondary replicas. Availability groups can also contain more

than one availability database. Equally appealing is the fact you can host more than one availability

group within an implementation. As a result, it is possible to group databases with application

dependencies

together within an availability group and have all the availability databases seamlessly

fail over as a single cohesive unit as depicted in Figure 2-4.

 

 

 

FIGURE 2-4 Dedicated availability groups for Finance and HR availability databases

 

In addition, as shown in Figure 2-4, there is one primary replica and two secondary replicas with

two availability groups. One of these availability groups is called Finance, and it includes all the

Finance databases; the other availability group is called HR, and it includes all the Human Resources

databases. The Finance availability group can fail over independently of the HR availability group,

and unlike database mirroring, all availability databases within an availability group fail over as a

single unit. Moreover, organizations can improve their IT efficiency, increase performance, and reduce

 

 

 

total cost of ownership with better resource utilization of secondary/passive hardware because these

secondary replicas can be leveraged for backups and read-only operations such as reporting and

maintenance. This is covered in the “Active Secondaries” section later in this chapter.

 

Now that you have been introduced to some of the benefits AlwaysOn Availability Groups offers

for an organization, let’s take the time to get a stronger understanding of the AlwaysOn Availability

Groups concepts and how this new capability operates. The concepts covered include the following:

 

¦ ¦ Availability replica roles

¦ ¦ Data synchronization modes

¦ ¦ Failover modes

¦ ¦ Connection mode in secondaries

¦ ¦ Availability group listeners

 

 

Availability Replica Roles

 

Each AlwaysOn availability group comprises a set of two or more failover partners that are referred to

as availability replicas. The availability replicas can consist of either a primary role or a secondary role.

Note that there can be a maximum of four secondaries, and of these four secondaries only a maximum

of two secondaries can be configured to use the synchronous-commit availability mode.

 

The roles affiliated with the availability replicas in AlwaysOn Availability Groups follow the same

principles as the legendary Sith rule of two doctrines in the Star Wars saga. In Star Wars, there can be

only two Siths at one time, a master and an apprentice. Similarly, a SQL Server instance in an availability

group can be only a primary replica or a secondary replica. At no time can it be both because the

role swapping is controlled by Windows Server Failover Cluster (WSFC).

 

Each of the SQL Server instances in the availability group is hosted on either a SQL Server failover

cluster instance (FCI) or a stand-alone instance of SQL Server 2012. Each of these instances resides on

different nodes of a WSFC. WSFC is typically used for providing high availability and disaster recovery

for well-known Microsoft products. As such, availability groups use WSFC as the underlying mechanism

to provide internode health detection, failover coordination, primary health detection, and

distributed change notifications for the solution.

 

Each availability replica hosts a copy of the availability databases in the availability group. Because

there are multiple copies of the databases being hosted on each availability replica, there isn’t a

prerequisite for using shared storage like there was in the past when deploying traditional SQL Server

failover clusters. On the flip side, when using nonshared storage, an organization must realize that

storage requirements increase as the number of replicas it plans on hosting increases.

 

 

 

Data Synchronization Modes

 

To move data from the primary replica to the secondary replica, each mode uses either

synchronous-

commit availability mode or asynchronous-commit availability mode. Give

consideration

to the following items when selecting either option:

 

¦ ¦ When you use the synchronous-commit mode, a transaction is committed on both replicas

to guarantee transactional consistency. This, however, means increased latency. As such, this

option might not be appropriate for partners who don’t share a high-speed network or who

reside in different geographical locations.

¦ ¦ The asynchronous-commit mode commits transactions between partners without waiting for

the partner to write the log to disk. This maximizes performance between the application and

the primary replica and is well suited for disaster-recovery solutions.

 

 

Availability Groups Failover Modes

 

When configuring AlwaysOn availability groups, database administrators can choose from two

failover modes when swapping roles from the primary to the secondary replicas. For administrators

who are familiar with database mirroring, you’ll see that to obtain high availability and disaster

recovery the failover modes are very similar to the modes in database mirroring. These are the two

AlwaysOn failover modes available when using the New Availability Group Wizard:

 

¦ ¦ Automatic Failover This replica uses synchronous-commit availability mode, and it supports

both automatic failover and manual failover between the replica partners. A maximum of two

failover replica partners are supported when choosing automatic failover.

¦ ¦ Manual Failover This replica uses synchronous or asynchronous commit availability mode

and supports only manual failovers between the replica partners.

 

 

Connection Mode in Secondaries

 

As indicated earlier, each of the secondaries can be configured to support read-only access for

reporting

or other maintenance tasks, such as backups. During the final configuration stage of

AlwaysOn

availability groups, database administrators decide on the connection mode for the

secondary

replicas. There are three connection modes available:

 

¦ ¦ Disallow connections In the secondary role, this availability replica does not allow any connections.

¦ ¦ Allow only read-intent connections In the secondary role, this availability replica allows

only read-intent connections.

¦ ¦ Allow all connections In the secondary role, this availability replica allows all connections

for read access, including connections running with older clients.

 

 

 

 

Availability Group Listeners

 

The availability group listener provides a way of connecting to databases within an availability group

via a virtual network name that is bound to the primary replica. Applications can specify the network

name affiliated with the availability group listener in connection strings. After the availability group

fails over from the primary replica to a secondary replica, the network name directs connections to

the new primary replica. The availability group listener concept is similar to a Virtual SQL Server Name

when using failover clustering; however, with an availability group listener, there is a virtual network

name for each availability group, whereas with SQL Server failover clustering, there is one virtual

network name for the instance.

 

You can specify your availability group listener preferences when using the Create A New

Availability

Group Wizard in SQL Server Management Studio, or you can manually create or modify

an availability group listener after the availability group is created. Alternatively, you can use

Transact-

SQL to create or modify the listener too. Notice in Figure 2-5 that each availability group

listener requires a DNS name, an IP address, and a port such as 1433. Once the availability group

listener is created, a server name and an IP address cluster resource are automatically created within

Failover Cluster Manager. This is certainly a testimony to the availability group’s flexibility and tight

integration with SQL Server, because the majority of the configurations are done within SQL Server.

 

 

 

FIGURE 2-5 Specifying availability group listener properties

 

Be aware that there is a one-to-one mapping between availability group listeners and availability

groups. This means you can create one availability group listener for each availability group. However,

if more than one availability group exists within a replica, you can have more than one availability

group listener. For example, there are two availability groups shown in Figure 2-6: one is for

 

 

 

the Finance

availability databases, and the other is for the Accounting availability databases. Each

availability

group has its own availability group listener that clients and applications connect to.

 

 

 

FIGURE 2-6 Illustrating two availability group listeners within a replica

 

Configuring Availability Groups

 

When creating a new availability group, a database administrator needs to specify an availability

group name, such as AvailablityGroupFinance, and then select one or more databases to be part of

in the availability group. The next step involves first specifying one or more instances of SQL Server

to host secondary availability replicas, and then specifying your availability group listener preference.

The final step is selecting the data-synchronization preference and connection mode for the secondary

replicas. These configurations are conducted with the New Availability Group Wizard or with

Transact-SQL PowerShell scripts.

 

Prerequisites

 

To deploy AlwaysOn Availability Groups, the following prerequisites must be met:

 

¦ ¦ All computers running SQL Server, including the servers that will reside in the disaster-recovery

site, must reside in the same Windows-based domain.

¦ ¦ All SQL Server computers must participate in a single Windows Server failover cluster even if

the servers reside in multiple sites.

¦ ¦ All servers must partake in a Windows Server failover cluster.

¦ ¦ AlwaysOn Availability Groups must be enabled on each server.

¦ ¦ All the databases must be in full recovery mode.

 

 

 

 

¦ ¦ A full backup must be conducted on all databases before deployment.

¦ ¦ The server cannot host the Active Directory Domain Services role.

 

 

Deployment Examples

 

Figure 2-7 shows the Specify Replicas page you see when using the New Availability Group Wizard.

In this example, there are three SQL Server instances in the availability group called Finance:

SQL01\Instance01, SQL02\Instance01, and SQL03\Instance01. SQL01\Instance01 is configured as the

Primary replica, whereas SQL02\Instance01 and SQL03\Instance01 are configured as secondaries.

SQL01\Instance01 and SQL02\Instance01 support automatic failover with synchronous data movement,

whereas SQL-03\Instance01 uses asynchronous-commit availability mode and supports only

a forced failover. Finally, SQL01\Instance01 does not allow read-only connections to the secondary,

whereas SQL02\Instance01 and SQL03\Instance01 allow read-intent connections to the secondary. In

addition, for this example, SQL01\Instance01 and SQL02\Instance01 reside in a primary datacenter for

high availability within a site, and SQL03\Instance01 resides in the disaster recovery datacenter and

will be brought online manually in the event the primary datacenter becomes unavailable.

 

 

 

FIGURE 2-7 Specifying the SQL Server instances in the availability group

 

One thing becomes vividly clear from Figure 2-7 and the preceding example: there are many

different

deployment configurations available to satisfy any organization’s high-availability and

disaster-recovery requirements. See Figure 2-8 for the following additional deployment alternatives:

 

¦ ¦ Nonshared storage, local, regional, and geo target

¦ ¦ Multisite cluster with another cluster as the disaster recovery (DR) target

 

 

 

 

¦ ¦ Three-node cluster with similar DR target

¦ ¦ Secondary targets for backup, reporting, and DR

 

 

Nonshared Storage, Local,

Regional, and Geo Target

A

A

A

A

A

A

A A

A

A

A

A A A

A

A

A

A A

A

A

A A A

Multisite Cluster with Another

Cluster as Disaster Recovery Target

Three-Node Cluster with Similar

Disaster Recovery Target

Secondary Targets for Backup,

Reporting, and Disaster Recovery

 

FIGURE 2-8 Additional AlwaysOn deployment alternatives

 

Monitoring Availability Groups with the Dashboard

 

Administrators have an opportunity to leverage a new and remarkably intuitive manageability

dashboard in in SQL Server 2012 to monitor availability groups. The dashboard, shown in Figure 2-9,

reports the health and status associated with each instance and availability database in the availability

group. Moreover, the dashboard displays the specific replica role of each instance and provides synchronization

status. If there is an issue or if more information on a specific event is required, a database

administrator can click the availability group state, server instance name, or health status hyperlinks

for additional information. The dashboard is launched by right-clicking the Availability Groups

folder in the Object Explorer in SQL Server Management Studio and selecting Show Dashboard.

 

 

 

 

 

 

FIGURE 2-9 Monitoring availability groups with the new Availability Group dashboard

 

Active Secondaries

 

As indicated earlier, many organizations communicated to the SQL Server team their need to improve

IT efficiency by optimizing their existing hardware investments. Specifically, organizations hoped their

production systems for passive workloads could be used in some other capacity instead of remaining

in an idle state. These same organizations also wanted reporting and maintenance tasks offloaded

from production servers because these tasks negatively affected production workloads. With SQL

Server 2012, organizations can leverage the AlwaysOn Availability Group capability to configure

secondary replicas, also referred to as active secondaries, to provide read-only access to databases

affiliated with an availability group.

 

All read-only operations on the secondary replicas are supported by row versioning and are

automatically

mapped to snapshot isolation transaction level, which eliminates reader/writer

contention.

In addition, the data in the secondary replicas is near real time. In many circumstances,

data latency between the primary and secondary databases should be within seconds. Note that the

latency of log synchronization affects data freshness.

 

For organizations, active secondaries are synonymous with performance optimization on a primary

replica and increases to overall IT efficiency and hardware utilization.

 

 

 

Read-Only Access to Secondary Replicas

 

Recall that when configuring the connection mode for secondary replicas, you can choose Disallow

Connections, Allow Only Read-Intent Connections, and Allow All Connections. The Allow Only

Read-

Intent Connections and Allow All Connections options both provide read-only access to

secondary

replicas. The Disallow Connections alternative does not allow read-only access as implied

by its name.

 

Now let’s look at the major differences between Allow Only Read-Intent Connections and Allow

All Connections. The Allow Only Read-Intent Connections option allows connections to the databases

in the secondary replica when the Application Intent Connection property is set to Read-only in the

SQL Server native client. When using the Allow All Connections settings, all client connections are

allowed independent of the Application Intent property. What is the Application Intent property in

the connection string? The Application Intent property declares the application workload type when

connecting

to a server. The possible values are Read-only and Read Write. Commands that try to

create

or modify data on the secondary replica will fail.

 

Backups on Secondary

 

Backups of availability databases participating in availability groups can be conducted on any of the

replicas. Although backups are still supported on the primary replica, log backups can be conducted

on any of the secondaries. Note that this is independent of the replication commit mode being

used.synchronous-commit or asynchronous-commit. Log backups completed on all replicas form a

single log chain, as shown in Figure 2-10.

 

Primary Replica Backups

Backups Supported

If Desired

Secondary Replica 2 Backups

Step 2: Log Backup

LSN 41-60

Secondary Replica 1 Backups

Step 1: Log Backup

LSN 21-40

Secondary Replica 3 Backups

Step 3: Log Backup

LSN 61-80

 

FIGURE 2-10 Forming a single log chain by backing up the transaction logs on multiple secondary replicas

 

As a result, the transaction log backups do not all have to be performed on the same replica.

This in no way means that serious thought should not be given to the location of your backups. It is

recommended that you store all backups in a central location because all transaction log backups are

 

 

 

required to perform a restore in the event of a disaster. Therefore, if a server is no longer available

and it contained the backups, you will be negatively affected. In the event of a failure, use the new

Database Recovery Advisor Wizard; it provides many benefits when conducting restores. For example,

if you are performing backups on different secondaries, the wizard generates a visual image of a

chronological timeline by stitching together all of the log files based on the Log Sequence Number

(LSN).

 

AlwaysOn Failover Cluster Instances

 

You’ve seen the results of the development efforts in engineering the new AlwaysOn Availability

Groups capability for high availability and disaster recovery, and the creation of active secondaries.

Now you’ll explore the significant enhancements to traditional capabilities such as SQL Server failover

clustering that leverage shared storage. The following list itemizes some of the improvements that

will appeal to database administrators looking to gain high availability for their SQL Server instances.

Specifically, this section discusses the following features:

 

¦ ¦ Multisubnet Clustering This feature provides a disaster-recovery solution in addition to

high availability with new support for multisubnet failover clustering.

¦ ¦ Support for TempDB on Local Disk Another storage-level enhancement with failover

clustering is associated with TempDB. TempDB no longer has to reside on shared storage as

it did in previous versions of SQL Server. It is now supported on local disks, which results in

many practical benefits for organizations. For example, you can now offload TempDB I/O from

shared-storage devices (SSD) like a SAN and leverage fast SSD storage locally within the server

nodes to optimize TempDB workloads, which are typically random I/O.

¦ ¦ Flexible Failover Policy SQL Server 2012 introduces improved failure detection for the SQL

Server failover cluster instance by adding failure condition-level properties that allow you to

configure a more flexible failover policy.

 

 

Note AlwaysOn failover cluster instances can be combined with availability groups to offer

maximum SQL Server instance and database protection.

 

With the release of Windows Server 2008, new functionality enabled cluster nodes to be

connected

over different subnets without the need for a stretch virtual local area network (VLAN)

across networks. The nodes could reside on different subnets within a datacenter or in another

geographical

location, such as a disaster recovery site. This concept is commonly referred to as

multisite

clustering, multisubnet clustering, or stretch clustering. Unfortunately, the previous versions

of SQL Server could not take advantage of this Windows failover clustering feature. Organizations

that wanted to create either a multisite or multisubnet SQL Server failover cluster still had to create

a stretch VLAN to expose a single IP address for failover across sites. This was a complex and challenging

task for many organizations. This is no longer the case because SQL Server 2012 supports

 

 

 

multisubnet

and multisite clustering out of the box; therefore, the need for implementing stretch

VLAN technology no longer exists.

 

Figure 2-11 illustrates an example of a SQL Server multisubnet failover cluster between two

subnets

spanning two sites. Notice how each node affiliated with the multisubnet failover cluster

resides on a different subnet. Node 1 is located in Site 1 and resides on the 192.168.115.0/24 subnet,

whereas Node 2 is located in Site 2 and resides on the 192.168.116.0/24 subnet. The DNS IP address

associated with the virtual network name of the SQL Server cluster is automatically updated when a

failover from one subnet to another subnet occurs.

 

SQL Server 2012

Node 1

192.168.115.0/24 Subnet

SQL Server 2012

Node 2

192.168.116.0/24 Subnet

SQL Server Failover Cluster Instance

Node 1 – 192.168.115.5

Node 2 – 192.168.116.5

Data Replication Between Storage Systems

Failover

Site 1 Site 2

 

FIGURE 2-11 A multisubnet failover cluster instance example

 

For clients and applications to connect to the SQL Server failover cluster, they need two IP

addresses

registered to the SQL Server failover cluster resource name in WSFC. For example, imagine

your server name is SQLFCI01 and the IP addresses are 192.168.115.5 and 192.168.116.5. WSFC automatically

controls the failover and brings the appropriate IP address online depending on the node

that currently owns the SQL Server resource. Again, if Node 1 is affiliated with the 192.168.115.0/24

subnet and owns the SQL Server failover cluster, the IP address resource 192.168.115.6 is brought

online

as shown in Figure 2-12. Similarly, if a failover occurs and Node 2 owns the SQL Server

resource,

IP address resource 192.168.115.6 is taken offline and the IP address resource 192.168.116.6

is brought online.

 

 

 

 

 

FIGURE 2-12 Multiple IP addresses affiliated with a multisubnet failover cluster instance

 

Because there are multiple IP addresses affiliated with the SQL Server failover cluster instance

virtual name, the online address changes automatically when there is a failover. In addition,

Windows

failover cluster issues a DNS update immediately after the network name resource name

comes online.

The IP address change in DNS might not take effect on clients because of cache

settings; therefore, it is recommended that you minimize the client downtime by configuring the

HostRecordTTL

in DNS to 60 seconds. Consult with your DNS administrator before making any DNS

changes, because additional load requests could occur when tuning the TTL time with a host record.

 

Support for Deploying SQL Server 2012 on Windows Server

Core

 

Windows Server Core was originally introduced with Windows Server 2008 and saw significant

enhancements

with the release of Windows Server 2008 R2. For those who are unfamiliar with Server

Core, it is an installation option for the Windows Server 2008 and Windows Server 2008 R2 operating

systems. Because Server Core is a minimal deployment of Windows, it is much more secure because

its attack surface is greatly reduced. Server Core does not include a traditional Windows graphical

interface and, therefore, is managed via a command prompt or by remote administration tools.

 

Unfortunately, previous versions of SQL Server did not support the Server Core operating system,

but that has all changed. For the first time, Microsoft SQL Server 2012 supports Server Core installations

for organizations running Server Core based on Windows Server 2008 R2 with Service Pack 1 or

later.

 

Why is Server Core so important to SQL Server, and how does it positively affect availability?

When you are running SQL Server 2012 on Server Core, operating-system patching is drastically

reduced.by up to 60 percent. This translates to higher availability and a reduction in planned

downtime for any organization’s mission-critical databases and workloads. In addition, surface-area

 

 

 

attacks are greatly reduced and overall security of the database platform is strengthened, which again

translates to maximum availability and data protection.

 

When first introduced, Server Core required the use and knowledge of command-line syntax to

manage it. Most IT professionals at this time were accustomed to using a graphical user interface

(GUI) to manage and configure Windows, so they had a difficult time embracing Server Core. This

affected its popularity and, ultimately, its implementation. To ease these challenges, Microsoft created

SCONFIG, which is an out-of-the-box utility that was introduced with the release of Windows Server

2008 R2 to dramatically ease server configurations. To navigate through the SCONFIG options, you

only need to type one or more numbers to configure server properties, as displayed in Figure 2-13.

 

 

 

FIGURE 2-13 SCONFIG utility for configuring server properties in Server Core

 

The following sections articulate the SQL Server 2012 prerequisites for Server Core, SQL Server

features supported on Server Core, and the installation alternatives.

 

SQL Server 2012 Prerequisites for Server Core

 

Organizations installing SQL Server 2012 on Windows Server 2008 R2 Server Core must meet the

following

operating system, features, and components prerequisites.

 

The operating system requirements are as follows:

 

¦ ¦ Windows Server 2008 R2 SP1 64-bit x64 Data Center Server Core

¦ ¦ Windows Server 2008 R2 SP1 64-bit x64 Enterprise Server Core

¦ ¦ Windows Server 2008 R2 SP1 64-bit x64 Standard Server Core

¦ ¦ Windows Server 2008 R2 SP1 64-bit x64 Web Server Core

 

 

Here is the list of features and components:

 

¦ ¦ .NET Framework 2.0 SP2

¦ ¦ .NET Framework 3.5 SP1 Full Profile

¦ ¦ .NET Framework 4 Server Core Profile

 

 

 

 

¦ ¦ Windows Installer 4.5

¦ ¦ Windows PowerShell 2.0

 

 

Once you have all the prerequisites, it important to become familiar with the SQL Server

components

supported on Server Core.

 

SQL Server Features Supported on Server Core

 

There are numerous SQL Server features that are fully supported on Server Core. They include

Database

Engine Services, SQL Server Replication, Full Text Search, Analysis Services, Client Tools

Connectivity, and Integration Services. Likewise, Sever Core does not support many other features,

including

Reporting Services, Business Intelligence Development Studio, Client Tools Backward

Compatibility, Client Tools SDK, SQL Server Books Online, Distributed Replay Controller, SQL Client

Connectivity SDK, Master Data Services, and Data Quality Services. Some features such as Management

Tools – Basic, Management Tools – Complete, Distributed Replay Client, and Microsoft Sync

Framework are supported only remotely. Therefore, these features can be installed on editions of the

Windows operating system that are not Server Core, and then used to remotely connect to a SQL

Server instance running on Server Core. For a full list of supported and unsupported features review

the information at this link: http://msdn.microsoft.com/en-us/library/hh231669(SQL.110).aspx.

 

Note To leverage Server Core, you need to plan your SQL Server installation ahead of

time. Give yourself the opportunity to fully understand which SQL Server features are

required

to support your mission-critical workloads.

 

SQL Server on Server Core Installation Alternatives

 

The typical SQL Server Installation Setup Wizard is not supported when installing SQL Server 2012 on

Server Core. As a result, you need to automate the installation process by either using a commandline

installation, using a configuration file, or leveraging the DefaultSetup.ini methodology. Details

and examples for each of these methods can be found in Books Online: http://technet.microsoft.com

/en-us/library/ms144259(SQL.110).aspx.

 

Note When installing SQL Server 2012 on Server Core, ensure that you use Full Quiet

mode by using the /Q parameter or Quiet Simple mode by using the /QS parameter.

 

 

 

Additional High-Availability and Disaster-Recovery

Enhancements

 

This section summarizes some of the additional high-availability and disaster recovery enhancements

found in SQL Server 2012.

 

Support for Server Message Block

 

A common trend for organizations in recent years has been the movement toward consolidating

databases and applications onto fewer servers.specifically, hosting many instances of SQL Server

running on a failover cluster. When using failover clustering for consolidation, the previous versions of

SQL Server required a single drive letter for each SQL Server failover cluster instance. Because there

are only 23 drive letters available, without taking into account reservations, the maximum amount of

SQL Server instances supported on a single failover cluster was 23. Twenty-three instances sounds like

an ample amount; however, the drive letter limitation negatively affects organizations running powerful

servers that have the compute and memory resources to host more than 23 instances on a single

server. Going forward, SQL Server 2012 and failover clustering introduces support for Server Message

Block (SMB).

 

Note You might be thinking you can use mount points to alleviate the drive-letter pain

point. When working with previous versions of SQL Server, even with mount points, you

need at least one drive letter for each SQL Server failover cluster instance.

 

Some of the SQL Server 2012 benefits brought about by SMB are, of course, database-storage

consolidation and the potential to support more than 23 clustering instances in a single WSFC. To

take advantage of these features, the file servers must be running Windows Server 2008 or later

versions

of the operating system.

 

Database Recovery Advisor

 

The Database Recovery Advisor is a new feature aimed at optimizing the restore experience for

database

administrators conducting database recovery tasks. This tool includes a new timeline feature

that provides a visualization of the backup history, as shown in Figure 2-14.

 

 

 

 

 

FIGURE 2-14 Database Recovery Advisor backup and restore visual timeline

 

Online Operations

 

SQL Server 2012 also includes a few enhancements for online operation that reduce downtime during

planned maintenance operations. Line-of-business (LOB) re-indexing and adding columns with

defaults

are now supported.

 

Rolling Upgrade and Patch Management

 

All of the new AlwaysOn capabilities reduce application downtime to only a single manual failover

by supporting rolling upgrades and patching of SQL Server. This means a database administrator

can apply

a service pack or critical fix to the passive node or nodes if using a failover cluster or to

secondary

replicas if using availability groups. Once the installation is complete on all passive nodes

or secondaries, a database administrator can conduct a manual failover and then apply the service

pack or critical fix to the node in an FCI or replica. This rolling strategy also applies when upgrading

the database platform.

 

 

 

41

 

CHAP TE R 3

 

Performance and Scalability

 

Microsoft SQL Server 2012 introduces a new index type called columnstore. The columnstore

index feature was originally referred to as project Apollo during the development phases of SQL

Server 2012 and during the distribution of the Community Technology Preview (CTP) releases of the

product. This new index combined with the advanced query-processing enhancements offer blazingfast

performance optimizations for data-warehousing workloads and other similar queries. In many

cases, data-warehouse query performance has improved by tens to hundreds of times.

 

This chapter aims to teach, enlighten, and even dispel flawed beliefs about the columnstore index

so that database administrators can greatly increase query performance for their data warehouse

workloads. The questions this chapter focuses are the following:

 

¦ ¦ What is a columnstore index?

¦ ¦ How does a columnstore index drastically increase the speed of data warehouse queries?

¦ ¦ When should a database administrator build a columnstore index?

¦ ¦ Are there any well-established best practices for a columnstore index deployment?

 

 

Let’s look under the hood to see how organizations will benefit from significant data-warehouse

performance gains with the new, in-memory, columnstore index technology, which also helps with

managing increasing data volumes.

 

Columnstore Index Overview

 

Because of a proliferation of data being captured across devices, applications, and services,

organizations

today are tasked with storing massive amounts of data to successfully operate their

businesses. Using traditional tools to capture, manage, and process data within an acceptable time is

becoming increasingly challenging as the data users want to capture continues to grow. For example,

the volume of data is overwhelming the ability of data warehouses to execute queries in a timely

manner, and considerable time is spent tuning queries and designing and maintaining indexes to try

to get acceptable query performance. In many cases, so much time might have elapsed between the

time the query is launched and the time the result sets are returned that organizations have difficulty

recalling what the original request was about. Equally unproductive are the cases where the delay

causes the business opportunity to be lost.

 

 

 

42 PART 1 Database Administration

 

With the many issues organizations are facing as one of their primary concerns, the Query

Processing

and Storage teams from the SQL Server product group set to work on new technologies

that would allow very large data sets to be read quickly and accurately while transforming the data

into useful information and knowledge for organizations in a timely manner. The Query Processing

team reviewed academic research in columnstore data representations and analyzed improved

query-execution capabilities for data warehousing. In addition, they collaborated with the SQL Server

product group Analysis Services team to gain a stronger understanding about the other team’s work

with their columnstore implementation known as PowerPivot for SQL Server 2008 R2. Their research

and analysis led the Query Processing team to create the new columnstore index and query optimizations

based on vector-based execution capability, which significantly improves data-warehouse

query performance.

 

When developing the new columnstore index, the Query Processing team committed to a number

of goals. They aimed to ensure that end users who consume data had an interactive and positive

experience with all data sets, whether large or small, which meant the response time on data must be

swift. These strategies also apply to ad hoc and reporting queries. Moreover, database administrators

might even be able to reduce their needs for manually tuning queries, summary tables, indexed

views, and in some cases OLAP cubes. All these goals naturally impact total cost of ownership (TCO)

because hardware costs are lowered and fewer people are required to get a task accomplished.

 

Columnstore Index Fundamentals and Architecture

 

Before designing, implementing, or managing a columnstore index, it is beneficial to understand how

they work, how data is stored in a columnstore index, and what type of queries can benefit from a

columnstore index.

 

How Is Data Stored When Using a Columnstore Index?

 

With traditional tables (heaps) and indexes (B-trees), SQL Server stores data in pages in a row-based

fashion. This storage model is typically referred to as a row store. Using column stores is like turning

the traditional storage model 90 degrees, where all the values from a single column are stored contiguously

in a compressed form. The columnstore index stores each column in a separate set of disk

pages rather than storing multiple rows per page, which has been the traditional storage format. The

following examples illustrate the differences.

 

Let’s use a common table populated with employee data, as illustrated in Table 3-1, and then

evaluate the different ways data can be stored. This employee table includes typical data such as an

employee ID number, employee name, and the city and state the employee is located in.

 

 

 

CHAPTER 3 Performance and Scalability 43

 

TABLE 3-1 Traditional Table Containing Employee Data

 

EmployeeID

 

Name

 

City

 

State

 

1

 

Ross

 

San Francisco

 

CA

 

2

 

Sherry

 

New York

 

NY

 

3

 

Gus

 

Seattle

 

WA

 

4

 

Stan

 

San Jose

 

CA

 

5

 

Lijon

 

Sacramento

 

CA

 

 

 

 

 

Depending on the type of index chosen—traditional or columnstore—database administrators can

organize their data by row (as shown in Table 3-2) or by column (as shown in Table 3-3).

 

TABLE 3-2 Employee Data Stored in a Traditional “Row Store” Format

 

Row Store

 

1 Ross San Francisco CA

 

2 Sherry New York NY

 

3 Gus Seattle WA

 

4 Stan San Jose CA

 

5 Lijon Sacramento CA

 

 

 

 

 

TABLE 3-3 Employee Data Stored in the New Columnstore Format

 

Columnstore

 

1 2 3 4 5

 

Ross Sherry Gus Stan Lijon

 

San Francisco New York Seattle San Jose Sacramento

 

CA NY WA CA CA

 

 

 

 

 

As you can see, the major difference between the columnstore format in Table 3-3 and the row

store method in Table 3-2 is that a columnstore index groups and stores data for each column and

then joins all the columns to complete the whole index, whereas a traditional index groups and

stores data for each row and then joins all the rows to complete the whole index.

 

Now that you understand how data is stored when using columnstore indexes compared to

traditional

B-tree indexes, let’s take a look at how this new storage model and advanced query

optimizations

significantly speed up the retrieval of data. The next section describes three ways that

SQL Server columnstore indexes significantly improve the speed of queries.

 

 

 

How Do Columnstore Indexes Significantly Improve the Speed

of Queries?

 

The new columnstore storage model significantly improves data warehouse query speeds for

many reasons. First, data organized in a column shares many more similar characteristics than data

organized

across rows. As a result, a much higher level of compression can be achieved compared

to data organized across rows. Moreover, the columnstore index within SQL Server uses the VertiPaq

compression algorithm technology, which in SQL Server 2008 R2 was found only in Analysis Server

for PowerPivot. VertiPaq compression is far superior to the traditional row and page compression

used in the Database Engine, with compression rates of up to 15 to 1 having been achieved with

VertiPag. When data is compressed, queries require less IO because the amount of data transferred

from disk to memory is significantly reduced. Reducing IO when processing queries equates to faster

performance-response times. The advantages of columnstore do not come to an end here. With less

data transferred to memory, less space is required in memory to hold the working set affiliated with

the query.

 

Second, when a user runs a query using the columnstore index, SQL Server fetches data only for

the columns that are required for the query, as illustrated in Figure 3-1. In this example, there are 15

columns in the table; however, because the data required for the query resides in Column 7, Column

8, and Column 9, only these columns are retrieved.

 

 

C1 C2 C3 C4 C5 C6

C7 C8 C9

C10 C11 C12 C13 C14 C15

 

FIGURE 3-1 Improving performance and reducing IO by fetching only columns required for the query

 

Because data-warehouse queries typically touch only 10 to 15 percent of the columns in large fact

tables, fetching only selected columns translates into savings of approximately 85 to 90 percent of an

organization’s IO, which again increases performance speeds.

 

 

 

Batch-Mode Processing

 

Finally, an advanced technology for processing queries that use columnstore indexes speeds up

queries yet another way. Speaking about processes, it is a good time to dig a little deeper into how

queries are processed.

 

First, the data in the columns are processed in batches using a new, highly-efficient vector

technology

that works with columnstore indexes. Database administrators should take a moment

to review the query plan and notice the groups of operators that execute in batch mode. Note

that not all operators execute in batch mode; however, the most important ones affiliated with

data warehousing

do, such as Hash Join and Hash Aggregation. All of the algorithms have been

significantly

optimized to take advantage of modern hardware architecture, such as increased core

counts and additional RAM, thereby improving parallelism. All of these improvements affiliated

with the columnstore index contribute to better batch-mode processing than traditional row-mode processing.

 

 

Columnstore Index Storage Organization

 

Let’s examine how the storage associated with a columnstore index is organized. First I’ll define the

new storage concepts affiliated with columnstore indexes, such as a segment and a row group, and

then I’ll elucidate how these concepts relate to one another.

 

As illustrated in Figure 3-2, data in the columnstore index is broken up into segments. A segment

contains data from one column for a set of up to about 1 million rows. Segments for the same set

of rows comprise a row group. Instead of storing the data page by page, SQL Server stores the row

group as a unit. Each segment is internally stored in a separate Large Object (LOB). Therefore, when

SQL Server reads the data, the unit reading from disk consists of a segment and the segment is a unit

of transfer between the disk and memory.

 

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15

Segment

Row Group

 

FIGURE 3-2 How a columnstore index stores data

 

 

 

Columnstore Index Support and SQL Server 2012

 

Columnstore indexes and batch-query execution mode are deeply integrated in SQL Server 2012, and

they work in conjunction with many of the Database Engine features found in SQL Server 2012. For

example, database administrators can implement a columnstore index on a table and still successfully

use AlwaysOn Availability Groups (AG), AlwaysOn failover cluster instances (FCI), database mirroring,

log shipping, and SQL Server Management Studio administration tools. Here are the common business

data types supported by columnstore indexes:

 

¦ ¦ char and varchar

¦ ¦ All Integer types (int, bigint, smallint, and tinyint)

¦ ¦ real and float

¦ ¦ string

¦ ¦ money and small money

¦ ¦ All date, time, and DateTime types, with one exception (datetimeoffset with precision greater

than 2)

¦ ¦ Decimal and numeric with precision less than or equal to 18 (that is, with less than or exactly

18 digits)

¦ ¦ Only one columnstore index can be created per table.

 

 

Columnstore Index Restrictions

 

Although columnstore indexes work with the majority of the data types, components, and features

found in SQL Server 2012, columnstore indexes have the following restrictions and cannot be

leveraged

in the following situations:

 

¦ ¦ You can enable PAGE or ROW compression on the base table, but you cannot enable PAGE or

ROW compression on the columnstore index.

¦ ¦ Tables and columns cannot participate in a replication topology.

¦ ¦ Tables and columns using Change Data Capture are unable to participate in a columnstore

index.

¦ ¦ Create Index: You cannot create a columnstore index on the following data types:

• decimal greater than 18 digits

• binary and varbinary

• BLOB

• CLR

• (n)varchar(max)

 

 

 

 

 

 

 

• uniqueidentifier

• datetimeoffset with precision greater than 2

 

 

 

¦ ¦ Table Maintenance: If a columnstore index exists, you can read the table but you cannot

directly update it. This is because columnstore indexes are designed for data-warehouse

workloads

that are typically read based. Rest assured that there is no need to agonize. The

upcoming “Columnstore Index Design Considerations and Loading Data” section articulates

strategies on how to load new data when using columnstore indexes.

¦ ¦ Process Queries: You can process all read-only T-SQL queries using the columnstore index, but

because batch processing works only with certain operators, you will see that some queries

are accelerated more than others.

¦ ¦ A column that contains filestream data cannot participate in a columnstore index.

¦ ¦ INSERT, UPDATE, DELETE, and MERGE statements are not allowed on tables using columnstore

indexes.

¦ ¦ More than 1024 columns are not supported when creating a columnstore index.

¦ ¦ Only nonclustered columnstore indexes are allowed. Filtered columnstore indexes are not allowed.

¦ ¦ Computed and sparse columns cannot be part of a columnstore index.

¦ ¦ A columnstore index cannot be created on an indexed view.

 

 

Columnstore Index Design Considerations and Loading Data

 

When working with columnstore indexes, some queries are accelerated much more than others.

Therefore, to optimize query performance, it is important to understand when to build a columnstore

index and when not to build a columnstore index. The next sections cover when to build columnstore

indexes, design considerations, and how to load data when using a columnstore index.

 

When to Build a Columnstore Index

 

The following list describes when database administrators should use a columnstore index to optimize

query performance:

 

¦ ¦ When workloads are mostly read based—specifically, data warehouse workloads.

¦ ¦ Your workflow permits partitioning (or a drop-rebuild index strategy) to handle new data.

Most commonly, this is associated with periodic maintenance windows when indexes can be

rebuilt or when staging tables are switching into empty partitions of existing tables.

¦ ¦ If most queries fit a star join pattern or entail scanning and aggregating large amounts

of data.

 

 

 

 

¦ ¦ If updates occur and most updates append new data, which can be loaded using staging

tables and partition switching.

 

 

You should use columnstore indexes when building the following types of tables:

 

¦ ¦ Large fact tables

¦ ¦ Large (millions of rows) dimension tables

 

 

When Not to Build a Columnstore Index

 

Database administrators might encounter situations when the performance benefits achieved by

using traditional B-tree indexes on their tables are greater than the benefits of using a columnstore

index. The following list describes some of these situations:

 

¦ ¦ Data in your table constantly requires updating.

¦ ¦ Partition switching or rebuilding an index does not meet the workflow requirements of your

business.

¦ ¦ You encounter frequent small look-up queries. Note, however, that a columnstore index might

still benefit you in this situation. As such, you can implement a columnstore index without any

repercussions because the query optimizer should be able to determine when to use the traditional

B-tree index instead of the columnstore index. This strategy assumes you have updated

statistics.

¦ ¦ You test columnstore indexes on your workload and do not see any benefit.

 

 

Loading New Data

 

As mentioned in earlier sections, tables with a columnstore index cannot be updated directly.

However,

there are three alternatives for loading data into a table with a columnstore index:

 

¦ ¦ Disable the Columnstore Index This procedure consists of the following three steps:

1. First disable the columnstore index.

2. Update the data.

3. Rebuild the index when the updates are complete.

 

 

 

 

 

Database administrators should ensure a maintenance window exists when leveraging this

strategy. The time affiliated with the maintenance window will vary per customers workload.

Therefore test within a prototype environment to determine time required.

 

¦ ¦ Leverage Partitioning and Partition Switching Partitioning data enables database

administrators

to manage and access subsets of their data quickly and efficiently while

maintaining

the integrity of the entire data collection. Partition switching also allows database

administrators to quickly and efficiently transfer subsets of their data by assigning a table as

a partition to an already existing partitioned table, switching a partition from one partitioned

 

 

 

 

table to another, or reassigning a partition to form a single table. Partition switching is fully

supported with a columnstore index and is a practical way for updating data.

 

 

To use partitioning to load data, follow these steps:

 

1. Ensure you have an empty partition to accept the new data.

2. Load data into an empty staging table.

3. Switch the staging table (containing the newly loaded data) into the empty partition.

 

 

To use partitioning to update existing data, use the following steps:

 

1. Determine which partition contains the data to be modified.

2. Switch the partition into an empty staging table.

3. Disable the columnstore index on the staging table.

4. Update the data.

5. Rebuild the columnstore index on the staging table.

6. Switch the staging table back into the original partition (which was left empty when the

partition was switched into the staging table).

 

 

¦ ¦ Union All Database administrators can load data by storing their main data in a fact table

that has a columnstore. Next, create a secondary table to add or update data. Finally, leverage

a UNION ALL query so that it returns all of the data between the large fact table with the

columnstore index and smaller updateable tables. Periodically load the table from the secondary

table into the main table by using partition switching or by disabling and rebuilding the

columnstore index. Note that some queries using the UNION ALL strategy might not be as fast

as when all the data is in a single table.

 

 

Creating a Columnstore Index

 

Creating a columnstore index is very similar to creating any other traditional SQL Server indexes. You

can use the graphical user interface in SQL Server Management Studio or Transact-SQL. Many individuals

prefer to use the graphical user interface because they want to avoid typing all of the column

names, which is what they have to do when creating the index with Transact-SQL.

 

A few questions arise frequently when creating a columnstore index. Database administrators

often want to know the following:

 

¦ ¦ Which columns should be included in the columnstore index?

¦ ¦ Is it possible to create a clustered columnstore index?

 

 

When creating a columnstore index, a database administrator should typically include all of the

supported columnstore index columns associated with the table. Note that you don’t need to include

 

 

 

all of the columns. The answer to the second question is “No.” All columnstore indexes must be

nonclustered;

therefore, a clustered columnstore index is not allowed.

 

The next sections explain the steps for creating a columnstore index using either of the two

methods

mentioned: SQL Server Management Studio and Transact-SQL.

 

Creating a Columnstore Index by Using SQL Server Management

Studio

 

Here are the steps for creating a columnstore index using SQL Server Management Studio (SSMS):

 

1. In SQL Server Management Studio, use Object Explorer to connect to an instance of the SQL

Server Database Engine.

2. In Object Explorer, expand the instance of SQL Server, expand Databases, expand a database,

and expand a table in which you would like to create a new columnstore index.

3. Expand the table, right-click the Index folder, choose New Index, and then click Non-Clustered

Columnstore Index.

4. On the General tab, in the Index name box, type a name for the new index, and then click

Add.

5. In the Select Columns dialog box, select the columns to participate in the columnstore index

and then click OK.

6. If desired, configure the settings on the Options, Storage, and Extended Properties pages. If

you want to maintain the defaults, click OK to create the index, as illustrated in Figure 3-3.

 

 

 

 

FIGURE 3-3 Creating a nonclustered columnstore index with SSMS

 

 

 

Creating a Columnstore Index Using Transact-SQL

 

As mentioned earlier, you can use Transact-SQL to create a columnstore index instead of using the

graphical user interface in SQL Server Management Studio. The following example illustrates the

syntax

for creating a columnstore index with Transact-SQL:

 

CREATE [ NONCLUSTERED ] COLUMNSTORE INDEX index_name

ON <object> ( column [ ,…n ] )

[ WITH ( <column_index_option> [ ,…n ] ) ]

[ ON {

{ partition_scheme_name ( column_name ) }

| filegroup_name

| “default”

}

]

[ ; ]

<object> ::=

{

[database_name. [schema_name ] . | schema_name . ]

table_name

{

<column_index_option> ::=

{

DROP_EXISTING = { ON | OFF }

| MAXDOP = max_degree_of_parallelism

}

 

The following bullets explain the arguments affiliated with the Transact-SQL syntax to create the

nonclustered columnstore index:

 

¦ ¦ NONCLUSTERED This argument indicates that this index is a secondary representation of the

data.

¦ ¦ COLUMNSTORE This argument indicates that the index that will be created is a columnstore

index.

¦ ¦ index_name This is where you specify the name of the columnstore index to be created.

Index names must be unique within a table or view but do not have to be unique within a

database.

¦ ¦ column This refers to the column or columns to be added to the index. As a reminder, a

columnstore index is limited to 1024 columns.

¦ ¦ ON partition_scheme_name(column_name) Specifies the partition scheme that defines the

file groups on which the partitions of a partitioned index are mapped. The column_name

specifies the column against which a partitioned index will be partitioned. This column must

match the data type, length, and precision of the argument of the partition function that the

partition_scheme_name is using: partition_scheme_name or filegroup.. If these are not specified

and the table is partitioned, the index is placed in the same partition scheme using the same

partitioning column as the underlying table.

 

 

 

 

 

 

FIGURE 3-5 Reviewing the Columnstore Index Scan results

 

Using Hints with a Columnstore Index

 

Finally, if you believe the query can benefit from a columnstore index and the query execution plan is

not leveraging it, you can force the query to use a columnstore index. You do this by using the WITH

(INDEX(<indexname>)) hint where the <indexname> argument is the name of the columnstore index

you want to force.

 

The following example illustrates a query with an index hint forcing the use of a columnstore index:

 

SELECT DISTINCT (SalesTerritoryKey)

FROM dbo.FactResellerSales WITH (INDEX (Non-ClusteredColumnStoreIndexSalesTerritory)

GO

 

The next example illustrates a query with an index hint forcing the use of a different index, such

as a traditional clustered B-tree index over a columnstore index. For example, let’s say there are two

indexes on this table called SalesTerritoryKey, a clustered index called ClusteredIndexSalesTerritory,

and a nonclustered columnstore index called Non-ClusteredColumnStoreIndexSalesTerritory.

Instead

of using the columnstore index, the hint forces the query to use the clustered index known as

ClusteredIndexSalesTerritory:

 

 

 

 

93

 

CHAP TE R 6

 

Integration Services

 

Since its initial release in Microsoft SQL Server 2005, Integration Services has had incremental

changes in each subsequent version of the product. However, those changes were trivial in comparison

to the number of enhancements, performance improvements, and new features introduced

in SQL Server 2012 Integration Services. This product overhaul affects every aspect of Integration

Services, from development to deployment to administration.

 

Developer Experience

 

The first change that you notice as you create a new Integration Services project is that Business

Intelligence Development Studio (BIDS) is now a Microsoft Visual Studio 2010 shell called SQL Server

Data Tools (SSDT). The Visual Studio environment alone introduces some slight user-interface changes

from the previous version of BIDS. However, several more significant interface changes of note are

specific to SQL Server Integration Services (SSIS). These enhancements to the interface help you to

learn about the package-development process if you are new to Integration Services, and they enable

you to develop packages more easily if you already have experience with Integration Services. If you

are already an Integration Services veteran, you will also notice the enhanced appearance of tasks and

data flow components with rounded edges and new icons.

 

Add New Project Dialog Box

 

To start working with Integration Services in SSDT, you create a new project by following the same

steps you use to perform the same task in earlier releases of Integration Services. From the File menu,

point to New, and then select Project. The Add New Project dialog box displays. In the Installed

Templates

list, you can select the type of Business Intelligence template you want to use and then

view only the templates related to your selection, as shown in Figure 6-1. When you select a template,

a description of the template displays on the right side of the dialog box.

 

 

 

94 PART 2 Business Intelligence Development

 

 

 

FIGURE 6-1 New Project dialog box displaying installed templates

 

There are two templates available for Integration Services projects:

 

¦ ¦ Integration Services Project You use this template to start development with a blank

package

to which you add tasks and arrange those tasks into workflows. This template type

was available in previous versions of Integration Services.

¦ ¦ Integration Services Import Project Wizard You use this wizard to import a project from

the Integration Services catalog or from a project deployment file. (You learn more about

project deployment files in the “Deployment Models” section of this chapter.) This option is

useful when you want to use an existing project as a starting point for a new project, or when

you need to make changes to an existing project.

 

 

Note The Integration Services Connections Project template from previous versions is no

longer available.

 

 

 

CHAPTER 6 Integration Services 95

 

General Interface Changes

 

After creating a new package, several changes are visible in the package-designer interface, as you

can see in Figure 6-2:

 

¦ ¦ SSIS Toolbox You now work with the SSIS Toolbox to add tasks and data flow components

to a package, rather than with the Visual Studio toolbox that you used in earlier versions of

Integration Services. You learn more about this new toolbox in the “SSIS Toolbox” section of

this chapter.

¦ ¦ Parameters The package designer includes a new tab to open the Parameters window for

a package. Parameters allow you to specify run-time values for package, container, and task

properties or for variables, as you learn in the “Parameters” section of this chapter.

¦ ¦ Variables button This new button on the package designer toolbar provides quick access

to the Variables window. You can also continue to open the window from the SSIS menu or by

right-clicking the package designer and selecting the Variables command.

¦ ¦ SSIS Toolbox button This button is also new in the package-designer interface and allows

you to open the SSIS Toolbox when it is not visible. As an alternative, you can open the SSIS

Toolbox from the SSIS menu or by right-clicking the package designer and selecting the SSIS

Toolbox command.

¦ ¦ Getting Started This new window displays below the Solution Explorer window and

provides

access to links to videos and samples you can use to learn how to work with

Integration

Services. This window includes the Always Show In New Project check box, which

you can clear if you prefer not to view the window after creating a new project. You learn

more about using this window in the next section, “Getting Started Window.”

¦ ¦ Zoom control Both the control flow and data flow design surface now include a zoom

control

in the lower-right corner of the workspace. You can zoom in or out to a maximum size

of 500 percent of the normal view or to a minimum size of 10 percent, respectively. As part

of the zoom control, a button allows you to resize the view of the design surface to fit the

window.

 

 

 

 

SSIS Toolbox

Zoom Control Getting Started

Parameters

Variables

Button

SSIS Toolbox

Button Parameters

 

FIGURE 6-2 Package-designer interface changes

 

Getting Started Window

 

As explained in the previous section, the Getting Started window is new to the latest version of

Integration

Services. Its purpose is to provide resources to new developers. It will display automatically

when you create a new project unless you clear the check box at the bottom of the window. You

must use the Close button in the upper-right corner of the window to remove it from view. Should

you want to access the window later, you can choose Getting Started on the SSIS menu or right-click

the design surface and select Getting Started.

 

In the Getting Started window, you find several links to videos and Integration Services samples.

To use the links in this window, you must have Internet access. By default, the following topics are available:

 

 

¦ ¦ Designing and Tuning for Performance Your SSIS Packages in the Enterprise This

link provides access to a series of videos created by the SQL Server Customer Advisory Team

 

 

 

 

(SQLCAT) that explain how to monitor package performance and techniques to apply during

package development to improve performance.

¦ ¦ Parameterizing the Execute SQL Task in SSIS This link opens a page from which you can

access a brief video explaining how to work with parameterized SQL statements in Integration

Services.

¦ ¦ SQL Server Integration Services Product Samples You can use this link to access the

product samples available on Codeplex, Microsoft’s open-source project-hosting site. By

studying the package samples available for download, you can learn how to work with various

control flow tasks or data flow components.

 

 

Note Although the videos and samples accessible through these links were developed for

previous versions of Integration Services, the principles remain applicable to the latest version.

When opening a sample project in SSDT, you will be prompted to convert the project.

 

You can customize the Getting Started window by adding your own links to the SampleSites.xml

file located in the Program Files (x86)\Microsoft SQL Server\110\DTS\Binn folder.

 

SSIS Toolbox

 

Another new window for the package designer is the SSIS Toolbox. Not only has the overall interface

been improved, but you will find there is also added functionality for arranging items in the toolbox.

 

Interface Improvement

 

The first thing you notice in the SSIS Toolbox is the updated icons for most items. Furthermore, the

SSIS Toolbox includes a description for the item that is currently selected, allowing you to see what it

does without needing to add it first to the design surface. You can continue to use drag-and-drop to

place items on the design surface, or you can double-click the item. However, the new behavior when

you double-click is to add the item to the container that is currently selected, which is a welcome

time-saver for the development process. If no container is selected, the item is added directly to the

design surface.

 

Item Arrangement

 

At the top of the SSIS Toolbox, you will see two new categories, Favorites and Common, as shown

in Figure 6-3. All categories are populated with items by default, but you can move items into

another

category at any time. To do this, right-click the item and select Move To Favorites or Move

To Common.

If you are working with control flow items, you have Move To Other Tasks as another

choice, but if you are working with data flow items, you can choose Move To Other Sources, Move To

Other Transforms, or Move To Other Destinations. You will not see the option to move an item to the

category in which it already exists, nor are you able to use drag-and-drop to move items manually.

If you decide to start over and return the items to their original locations, select Restore Toolbox

Defaults.

 

 

 

 

 

FIGURE 6-3 SSIS Toolbox for control flow and data flow

 

Shared Connection Managers

 

If you look carefully at the Solution Explorer window, you will notice that the Data Sources and Data

Source Views folders are missing, and have been replaced by a new file and a new folder. The new

file is Project.params, which is used for package parameters and is discussed in the “Package Parameters”

section of this chapter. The Connections Managers folder is the new container for connection

managers

that you want to share among multiple packages.

 

Note If you create a Cache Connection Manager, Integration Services shares the in-memory

cache with child packages using the same cache as the parent package. This

feature

is valuable for optimizing repeated lookups to the same source across multiple

packages.

 

 

 

To create a shared connection manager, follow these steps:

 

1. Right-click the Connections Managers folder, and select New Connection Manager.

2. In the Add SSIS Connection Manager dialog box, select the desired connection-manager type

and then click the Add button.

3. Supply the required information in the editor for the selected connection-manager type, and

then click OK until all dialog boxes are closed.

 

 

A file with the CONMGR file extension displays in the Solution Explorer window within the

Connections

Managers folder. In addition, the file also appears in the Connections Managers tray

in the package designer in each package contained in the same project. It displays with a (project)

prefix to differentiate it from package connections. If you select the connection manager associated

with one package and change its properties, the change affects the connection manager in all other packages.

 

 

If you change your mind about using a shared connection manager, you can convert it to a

package

connection. To do this, right-click the connection manager in the Connection Managers

tray, and select Convert To Package Connection. The conversion removes the CONMGR file from

the Connections

Manager folder in Solution Explorer and from all other packages. Only the package

in which you execute the conversion contains the connection. Similarly, you can convert a package

connection to a shared connection manager by right-clicking the connection manager in Solution

Explorer and selecting Convert To Project Connection.

 

Scripting Engine

 

The scripting engine in SSIS is an upgrade to Visual Studio Tools for Applications (VSTA) 3.0 and

includes support for the Microsoft .NET Framework 4.0. When you edit a script task in the control

flow or a script component in the data flow, the VSTA integrated development environment (IDE)

continues

to open in a separate window, but now it uses a Visual Studio 2010 shell. A significant

improvement

to the scripting engine is the ability to use the VSTA debug features with a Script

component

in the data flow.

 

Note As with debugging the Script task in the control flow, you must set the

Run64BitRunTime project property to False when you are debugging on a 64-bit computer.

 

 

 

 

Expression Indicators

 

The use of expressions in Integration Services allows you, as a developer, to create a flexible package.

Behavior can change at run-time based on the current evaluation of the expression. For example, a

common reason to use expressions with a connection manager is to dynamically change connection

strings to accommodate the movement of a package from one environment to another, such as from

development to production. However, earlier versions of Integration Services did not provide an easy

way to determine whether a connection manager relies on an expression. In the latest version, an

extra icon appears beside the connection manager icon as a visual cue that the connection manager

uses expressions, as you can see in Figure 6-4.

 

 

 

FIGURE 6-4 A visual cue that the connection manager uses an expression

 

This type of expression indicator also appears with other package objects. If you add an expression

to a variable or a task, the expression indicator will appear on that object.

 

Undo and Redo

 

A minor feature, but one you will likely appreciate greatly, is the newly added ability to use Undo and

Redo while developing packages in SSDT. You can now make edits in either the control flow or data

flow designer surface, and you can use Undo to reverse a change or Redo to restore a change you

had just reversed. This capability also works in the Variables window, and on the Event Handlers and

Parameters tabs. You can also use Undo and Redo when working with project parameters.

 

To use Undo and Redo, click the respective buttons in the standard toolbar. You can also use Ctrl+Z

and Ctrl+Y, respectively. Yet another option is to access these commands on the Edit menu.

 

Note The Undo and Redo actions will not work with changes you make to the SSIS

Toolbox, nor will they work with shared connection managers.

 

Package Sort By Name

 

As you add multiple packages to a project, you might find it useful to see the list of packages in

Solution

Explorer display in alphabetical order. In previous versions of Integration Services, the only

way to re-sort the packages was to close the project and then reopen it. Now you can easily sort the

list of packages without closing the project by right-clicking the SSIS Packages folder and selecting

Sort By Name.

 

 

 

Status Indicators

 

After executing a package, the status of each item in the control flow and the data flow displays in the

package designer. In previous versions of Integration Services, the entire item was filled with green to

indicate success or red to indicate failure. However, for people who are color-blind, this use of color

was not helpful for assessing the outcome of package execution. Consequently, the user interface

now displays icons in the upper-right corner of each item to indicate success or failure, as shown in

Figure 6-5.

 

 

 

FIGURE 6-5 Item status indicators appear in the upper-right corner

 

Control Flow

 

Apart from the general enhancements to the package-designer interface, there are three notable

updates for the control flow. The Expression Task is a new item available to easily evaluate an expression

during the package workflow. In addition, the Execute Package Task has some changes to make it

easier to configure the relationship between a parent package and child package. Another new item

is the Change Data Capture Task, which we discuss in the “Change Data Capture Support” section of

this chapter.

 

Expression Task

 

Many of the developer experience enhancements in Integration Services affect both control flow and

data flow, but there is one new feature that is exclusive to control flow. The Expression Task is a new

item available in the SSIS Toolbox when the control flow tab is in focus. The purpose of this task is to

make it easier to assign a dynamic value to a variable.

 

Rather than use a Script Task to construct a variable value at runtime, you can now add an

Expression

Task to the workflow and use the SQL Server Integration Services Expression Language.

When you edit the task, the Expression Builder opens. You start by referencing the variable and

including

the equals sign (=) as an assignment operator. Then provide a valid expression that resolves

to a single value with the correct data type for the selected variable. Figure 6-6 illustrates an example

of a variable assignment in an Expression Task.

 

 

 

 

 

FIGURE 6-6 Variable assignment in an Expression Task

 

Note The Expression Builder is an interface commonly used with other tasks and data flow

components. Notice in Figure 6-6 that the list on the left side of the dialog box includes

both variables and parameters. In addition, system variables are now accessible from a

separate folder rather than listed together with user variables.

 

Execute Package Task

 

The Execute Package Task has been updated to include a new property, ReferenceType, which appears

on the Package page of the Execute Package Task Editor. You use this property to specify the

location

of the package to execute. If you select External Reference, you configure the path to the

child package

just as you do in earlier versions of Integration Services. If you instead select Project

Reference,

you then choose the child package from the drop-down list.

 

In addition, the Execute Package Task Editor has a new page for parameter bindings, as shown

in Figure 6-7. You use this page to map a parameter from the child package to a parameter value or

variable value in the parent package.

 

 

 

 

 

FIGURE 6-7 Parameter bindings between a parent package and a child package

 

Data Flow

 

The data flow also has some significant updates. It has some new items, such as the Source and

Destination

assistants and the DQS Cleansing transformation, and there are some improved items

such as the Merge and Merge Join transformation. In addition, there are several new data flow

components

resulting from a partnership between Microsoft and Attunity for use when accessing

Open Database Connectivity (ODBC) connections and processing change data capture logs. We

describe the change data capture components in the “Change Data Capture Support” section of this

chapter. Some user interface changes have also been made to simplify the process and help you get

your job done faster when designing the data flow.

 

Sources and Destinations

 

Let’s start exploring the changes in the data flow by looking at sources and destinations.

 

Source and Destination Assistants

 

The Source Assistant and Destination Assistant are two new items available by default in the Favorites

folder of the SSIS Toolbox when working with the data flow designer. These assistants help you easily

create a source or a destination and its corresponding connection manager.

 

 

 

To create a SQL Server source in a data flow task, perform the following steps:

 

1. Add the Source Assistant to the data flow design surface by using drag-and-drop or by

double-clicking the item in the SSIS Toolbox, which opens the Source Assistant – Add New

Source dialog box as shown here:

 

 

 

 

Note Clear the Show Only Installed Source Types check box to display the

additional

available source types that require installation of one of the following

client providers: DB2, SAP BI, Sybase, or Teradata.

 

2. In the Select Connection Managers list, select an existing connection manager or select New

to create a new connection manager, and click OK.

3. If you selected the option to create a new connection manager, specify the server name,

authentication method, and database for your source data in the Connection Manager dialog

box, and click OK.

 

 

The new data source appears on the data flow design surface, and the connection manager

appears in the Connection Managers tray. You next need to edit the data source to configure

the data-access mode, columns, and error output.

 

ODBC Source and Destination

 

The ODBC Source and ODBC Destination components, shown in Figure 6-8, are new to Integration

Services in this release and are based on technology licensed by Attunity to Microsoft. Configuration

of these components is similar to that of OLE DB sources and destinations. The ODBC Source supports

Table Name and SQL Command as data-access modes, whereas data-access modes for the ODBC

Destination are Table Name – Batch and Table Name – Row By Row.

 

 

 

 

 

FIGURE 6-8 ODBC Source and ODBC Destination data flow components

 

Flat File Source

 

You use the Flat File source to extract data from a CSV or TXT file, but there were some data formats

that this source did not previously support without requiring additional steps in the extraction

process. For example, you could not easily use the Flat File source with a file containing a variable

number of columns. Another problem was the inability to use a character that was designated as a

qualifier as a literal value inside a string. The current version of Integration Services addresses both of

these problems.

 

¦ ¦ Variable columns A file layout with a variable number of columns is also known as a

ragged-right delimited file. Although Integration Services supports a ragged-right format,

a problem arises when one or more of the rightmost columns do not have values and the

column delimiters for the empty columns are omitted from the file. This situation commonly

occurs when the flat file contains data of mixed granularity, such as header and detail transaction

records. Although a row delimiter exists on each row, Integration Services ignored the row

delimiter and included data from the next row until it processed data for each expected column.

Now the Flat File source correctly recognizes the row delimiter and handles the missing

columns as NULL values.

 

 

Note If you expect data in a ragged-right format to include a column delimiter

for each missing column, you can disable the new processing behavior by changing

the AlwaysCheckForRowDelimiters property of the Flat File connection

manager to False.

 

¦ ¦ Embedded qualifiers Another challenge with the Flat File source in previous versions of Integration

Services was the use of a qualifier character inside a string encapsulated within qualifiers.

For example, consider a flat file that contains the names of businesses. If a single quote

is used as a text qualifier but also appears within the string as a literal value, the common

practice is to use another single quote as an escape character, as shown here.

 

 

ID,BusinessName

404,’Margie”s Travel’

406, ‘Kickstand Sellers’

 

 

 

In the first data row in this example, previous versions of Integration Services would fail to

interpret the second apostrophe in the BusinessName string as an escape character, and

instead would process it as the closing text qualifier for the column. As a result, processing of

the flat file returned an error because the next character in the row is not a column delimiter.

This problem is now resolved in the current version of Integration Services with no additional

configuration required for the Flat File source.

 

Transformations

 

Next we turn our attention to transformations.

 

Pivot Transformation

 

The user interface of the Pivot transformation in previous versions of Integration Services was a

generic editor for transformations, and it was not intuitive for converting input rows into a set of

columns for each row. The new custom interface, shown in Figure 6-9, provides distinctly named fields

and includes descriptions describing how each field is used as input or output for the pivot operation.

 

 

 

FIGURE 6-9 Pivot transformation editor for converting a set of rows into columns of a single row

 

Row Count Transformation

 

Another transformation having a generic editor in previous versions is the Row Count transformation.

The sole purpose of this transformation is to update a variable with the number of rows passing

through the transformation. The new editor makes it very easy to change the one property for this

transformation that requires configuration, as shown in Figure 6-10.

 

 

 

 

 

FIGURE 6-10 Row Count transformation editor for storing the current row count in a variable

 

Merge and Merge Join Transformations

 

Both the Merge transformation and the Merge Join transformation allow you to collect data from two

inputs and produce a single output of combined results. In earlier versions of Integration Services,

these transformations could result in excessive memory consumption by Integration Services when

data arrives from each input at different rates of speed. The current version of Integration Services

better accommodates this situation by introducing a mechanism for these two transformations to

better manage memory pressure in this situation. This memory-management mechanism operates

automatically, with no additional configuration of the transformation necessary.

 

Note If you develop custom data flow components for use in the data flow and if

these components accept multiple inputs, you can use new methods in the

Microsoft.SqlServer.Dts.Pipeline namespace to provide similar memory pressure

management

to your custom components. You can learn more about implementing these

methods by reading “Developing Data Flow Components with Multiple Inputs,” located at

http://msdn.microsoft.com/en-us/library/ff877983(v=sql.110).aspx.

 

DQS Cleansing Transformation

 

The DQS Cleansing transformation is a new data flow component you use in conjunction with Data

Quality Services (DQS). Its purpose is to help you improve the quality of data by using rules that

are established for the applicable knowledge domain. You can create rules to test data for common

misspellings

in a text field or to ensure that the column length conforms to a standard specification.

 

To configure the transformation, you select a data-quality-field schema that contains the rules to

apply and then select the input columns in the data flow to evaluate. In addition, you configure error

handling. However, before you can use the DQS Cleansing transformation, you must first install and

configure DQS on a server and create a knowledge base that stores information used to detect data

anomalies and to correct invalid data, which deserves a dedicated chapter. We explain not only how

DQS works and how to get started with DQS, but also how to use the DQS Cleansing transformation

in Chapter 7, “Data Quality Services.”

 

 

 

Column References

 

The pipeline architecture of the data flow requires precise mapping between input columns and

output

columns of each data flow component that is part of a Data Flow Task. The typical workflow

during data flow development is to begin with one or more sources, and then proceed with the

addition

of new components in succession until the pipeline is complete. As you plug each subsequent

component into the pipeline, the package designer configures the new component’s input

columns to match the data type properties and other properties of the associated output columns

from the preceding

component. This collection of columns and related property data is also known as

metadata.

 

If you later break the path between components to add another transformation to pipeline,

the metadata in some parts of the pipeline could change because the added component can add

columns, remove columns, or change column properties (such as convert a data type). In previous

versions of Integration Services, an error would display in the data flow designer whenever metadata

became invalid. On opening a downstream component, the Restore Invalid Column References editor

displayed to help you correct the column mapping, but the steps to perform in this editor were not

always intuitive. In addition, because of each data flow component’s dependency on access to metadata,

it was often not possible to edit the component without first attaching it to an existing component

in the pipeline.

 

Components Without Column References

 

Integration Services now makes it easier to work with disconnected components. If you attempt to

edit a transformation or destination that is not connected to a preceding component, a warning message

box displays: “This component has no available input columns. Do you want to continue editing

the available properties of this component?”

 

After you click Yes, the component’s editor displays and you can configure the component as

needed. However, the lack of input columns means that you will not be able to fully configure the

component using the basic editor. If the component has an advanced editor, you can manually add

input columns and then complete the component configuration. However, it is usually easier to use

the interface to establish the metadata than to create it manually.

 

Resolve References Editor

 

The current version of Integration Services also makes it easier to manage the pipeline metadata if

you need to add or remove components to an existing data flow. The data flow designer displays an

error indicator next to any path that contains unmapped columns. If you right-click the path between

components, you can select Resolve References to open a new editor that allows you to map the

output

columns to input columns by using a graphical interface, as shown in Figure 6-11.

 

 

 

 

 

FIGURE 6-11 Resolve References editor for mapping output to input columns

 

In the Resolve References editor, you can drag a column from the Unmapped Output Columns list

and add it to the Source list in the Mapped Columns area. Similarly, you can drag a column from the

Unmapped Input Columns area to the Destination list to link the output and input columns. Another

option is to simply type or paste in the names of the columns to map.

 

Tip When you have a long list of columns in any of the four groups in the editor, you

can type a string in the filter box below the list to view only those columns matching the

criteria

you specify. For example, if your input columns are based on data extracted from

the Sales.SalesOrderDetail table in the AdventureWorks2008R2 database, you can type

unit in the filter box to view only the UnitPrice and UnitPriceDiscount columns.

 

You can also manually delete a mapping by clicking the Delete Row button to the right of each

mapping. After you have completed the mapping process, you can quickly delete any remaining

unmapped input columns by selecting the Delete Unmapped Input Columns check box at the bottom

of the editor. By eliminating unmapped input columns, you reduce the component’s memory

requirements

during package execution.

 

Collapsible Grouping

 

Sometimes the data flow contains too many components to see at one time in the package designer,

depending on your screen size and resolution. Now you can consolidate data flow components into

groups and expand or collapse the groups. A group in the data flow is similar in concept to a se

 

 

 

quence container in the control flow, although you cannot use the group to configure a common

property for all components that it contains, nor can you use it to set boundaries for a transaction or

to set scope for a variable.

 

To create a group, follow these steps:

 

1. On the data flow design surface, use your mouse to draw a box around the components that

you want to combine as a group. If you prefer, you can click each component while pressing

the Ctrl key.

2. Right-click one of the selected components, and select Group. A group containing the

components

displays in the package designer, as shown here:

 

 

 

 

3. Click the arrow at the top right of the Group label to collapse the group.

 

 

Data Viewer

 

The only data viewer now available in Integration Services is the grid view. The histogram, scatter plot,

and chart views have been removed.

 

To use the data viewer, follow these steps:

 

1. Right-click the path, and select Enable Data Viewer. All columns in the pipeline are automatically

included.

2. If instead you want to display a subset of columns, right-click the new Data Viewer icon (a

magnifying glass) on the data flow design surface, and select Edit.

3. In the Data Flow Path Editor, select Data Viewer in the list on the left.

4. Move columns from the Displayed Columns list to the Unused Columns list as applicable

(shown next), and click OK.

 

 

 

 

 

 

Change Data Capture Support

 

Change data capture (CDC) is a feature introduced in the SQL Server 2008 database engine. When

you configure a database for change data capture, the Database Engine stores information about

insert,

update, and delete operations on tables you are tracking in corresponding change data

capture

tables. One purpose for tracking changes in separate tables is to perform extract, transform,

and load (ETL) operations without adversely impacting the source table.

 

In the previous two versions of Integration Services, there were multiple steps required to develop

packages that retrieve data from change data capture tables and load the results into destination.

To expand Integration Services’ data-integration capabilities in SQL Server 2012 by supporting

change data capture for SQL Server, Microsoft partnered with Attunity, a provider of real-time data-integration

software. As a result, new change data capture components are available for use in the

control flow and data flow, simplifying the process of package development for change data capture.

 

Note Change data capture support in Integration Services is available only in Enterprise,

Developer, and Evaluation editions. To learn more about the change data capture feature

in the Database Engine, see “Basics of Change Data Capture” at http://msdn.microsoft.com

/en-us/library/cc645937(SQL.110).aspx.

 

 

 

CDC Control Flow

 

There are two types of packages you must develop to manage change data processing with

Integration

Services: an initial load package for one-time execution, and a trickle-feed package for

ongoing execution on a scheduled basis. You use the same components in each of these packages,

but you configure the control flow differently. In each package type, you include a package variable

with a string data type for use by the CDC components to reflect the current state of processing.

 

As shown in Figure 6-12, you begin the control flow with a CDC Control Task to mark the start of

an initial load or to establish the Log Sequence Number (LSN) range to process during a trickle-feed

package execution. You then add a Data Flow Task that contains CDC components to perform the

processing of the initial load or changed data. (We describe the components to use in this Data Flow

Task later in this section.) Then you complete the control flow with another CDC Control Task to mark

the end of the initial load or the successful processing of the LSN range for a trickle-feed package.

 

 

 

FIGURE 6-12 Trickle-feed control flow for change data capture processing

 

Figure 6-13 shows the configuration of the CDC Control Task for the beginning of a trickle-feed

package. You use an ADO.NET Connection Manager to define the connection to a SQL Server database

for which change data capture is enabled. You also specify a CDC control operation and the

name of the CDC state variable. Optionally, you can use a table to persist the CDC state. If you do not

use a table for state persistency, you must include logic in the package to write the state to a persistent

store when change data processing completes and to read the state before beginning the next

execution of change data processing.

 

 

 

 

 

FIGURE 6-13 CDC Control Task Editor for retrieving the current LSN range for change data to process

 

CDC Data Flow

 

To process changed data, you begin a Data Flow Task with a CDC Source and a CDC Splitter, as

shown in Figure 6-14. The CDC Source extracts the changed data according to the specifications

defined by the CDC Control Task, and then the CDC Splitter evaluates each row to determine whether

the changed data is a result of an insert, update, or delete operation. Then you add data flow

components

to the each output of the CDC Splitter for downstream processing.

 

 

 

FIGURE 6-14 CDC Data Flow Task for processing changed data

 

 

 

CDC Source

 

In the CDC Source editor (shown in Figure 6-15), you specify an ADO.NET connection manager for the

database and select a table and a corresponding capture instance. Both the database and table must

be configured for change data capture in SQL Server. You also select a processing mode to control

whether to process all change data or net changes only. The CDC state variable must match the

variable you define in the CDC Control Task that executes prior to the Data Flow Task containing the

CDC Source. Last, you can optionally select the Include Reprocessing Indicator Column check box to

identify reprocessed rows for separate handling of error conditions.

 

 

 

FIGURE 6-15 CDC Source editor for extracting change data from a CDC-enabled table

 

CDC Splitter

 

The CDC Splitter uses the value of the _$operation column to determine the type of change

associated

with each incoming row and assigns the row to the applicable output: InsertOutput,

UpdateOutput,

or DeleteOutput. You do not configure this transformation. Instead, you add downstream

data flow components to manage the processing of each output separately.

 

Flexible Package Design

 

During the initial development stages of a package, you might find it easiest to work with hard-coded

values in properties and expressions to ensure that your logic is correct. However, for maximum

flexibility,

you should use variables. In this section, we review the enhancements for variables and

expressions—the cornerstones of flexible package design.

 

 

 

Variables

 

A common problem for developers when adding a variable to a package has been the scope

assignment.

If you inadvertently select a task in the control flow designer and then add a new variable

in the Variables window, the variable is created within the scope of that task and cannot be changed.

In these cases, you were required to delete the variable, clear the task selection on the design surface,

and then add the variable again within the scope of the package.

 

Integration Services now creates new variables with scope set to the package by default. To change

the variable scope, follow these steps:

 

1. In the Variables window, select the variable to change and then click the Move Variable button

in the Variables toolbar (the second button from the left), as shown here:

 

 

 

 

2. In the Select New Scope dialog box, select the executable to have scope—the package, an

event handler, container, or task—as shown here, and click OK:

 

 

 

 

Expressions

 

The expression enhancements in this release address a problem with expression size limitations and

introduce new functions in the SQL Server Integration Services Expression Language.

 

 

 

Expression Result Length

 

Prior to the current version of Integration Services, if an expression result had a data type of DT_WSTR

or DT_STR, any characters above a 4000-character limit would be truncated. Furthermore, if an

expression

contained an intermediate step that evaluated a result exceeding this 4000-character limit,

the intermediate result would similarly be truncated. This limitation is now removed.

 

New Functions

 

The SQL Server Integration Services Expression Language now has four new functions:

 

¦ ¦ LEFT You can now more easily return the leftmost portion of a string rather than use the

SUBSTRING function:

 

 

LEFT(character_expression,number)

 

¦ ¦ REPLACENULL You can use this function to replace NULL values in the first argument with

the expression specified in the second expression:

 

 

REPLACENULL(expression, expression)

 

¦ ¦ TOKEN This function allows you to return a substring by using delimiters to separate a string

into tokens and then specifying which occurrence to return:

 

 

TOKEN(character_expression, delimiter_string, occurrence)

 

¦ ¦ TOKENCOUNT This function uses delimiters to separate a string into tokens and then

returns

the count of tokens found within the string:

 

 

TOKENCOUNT(character_expression, delimiter_string)

 

Deployment Models

 

Up to now in this chapter, we have explored the changes to the package-development process in

SSDT, which have been substantial. Another major change to Integration Services is the concept of

deployment models.

 

Supported Deployment Models

 

The latest version of Integration Services supports two deployment models:

 

¦ ¦ Package deployment model The package deployment model is the deployment model

used in previous versions of Integration Services, in which the unit of deployment is an

individual

package stored as a DTSX file. A package can be deployed to the file system or to

the MSDB database in a SQL Server database instance. Although packages can be deployed

as a group and dependencies can exist between packages, there is no unifying object in

Integration

Services that identifies a set of related packages deployed using the package

 

 

 

 

model. To modify properties of package tasks at runtime, which is important when running

a package in different environments such as development or production, you use configurations

saved as DTSCONFIG files on the file system. You use either the DTExec or the DTExecUI

utilities

to execute a package on the Integration Services server, providing arguments on the

command line or in the graphical interface when you want to override package property values

at run time manually or by using configurations.

¦ ¦ Project deployment model With this deployment model, the unit of deployment is a

project, stored as an ISPAC file, which in turn is a collection of packages and parameters. You

deploy the project to the Integration Services catalog, which we describe in a separate section

of this chapter. Instead of configurations, you use parameters (as described later in the

“Parameters” section) to assign values to package properties at runtime. Before executing

a package, you create an execution object in the catalog and, optionally, assign parameter

values or environment references to the execution object. When ready, you start the execution

object by using a graphical interface in SQL Server Management Studio by executing a stored

procedure or by running managed code.

 

 

In addition to the characteristics just described, there are additional differences between

the package

deployment model and the project deployment model. Table 6-1 compares these differences.

 

 

TABLE 6-1 Deployment Model Comparison

 

Characteristic

 

Package Deployment Model

 

Project Deployment Model

 

Unit of deployment

 

Package

 

Project

 

Deployment location

 

File system or MSDB database

 

Integration Services catalog

 

Run-time property value assignment

 

 

Configurations

 

Parameters

 

Environment-specific values

for use in property values

 

Configurations

 

Environment variables

 

Package validation

 

Just before execution using:

 

• DTExec

 

• Managed code

 

Independent of execution using:

 

• SQL Server Management Studio

interface

 

 

• Stored procedure

 

• Managed code

 

Package execution

 

DTExec

 

DTExecUI

 

SQL Server Management Studio

interface

 

 

Stored procedure

 

Managed code

 

Logging

 

Configure log provider or

implement custom logging

 

No configuration required

 

Scheduling

 

SQL Server Agent job

 

SQL Server Agent job

 

CLR integration

 

Not required

 

Required

 

 

 

 

 

 

 

When you create a new project in SSDT, the project is by default established as a project

deployment

model. You can use the Convert To Package Deployment Model command on the Project

menu (or choose it from the context menu when you right-click the project in Solution Explorer)

to switch to the package deployment model. The conversion works only if your project is compatible

with the package deployment model. For example, it cannot use features that are exclusive to

the project deployment model, such as parameters. After conversion, Solution Explorer displays

an additional

label after the project name to indicate the project is now configured as a package

deployment

model, as shown in Figure 6-16. Notice that the Parameters folder is no longer available

in the project, while the Data Sources folder is now available in the project.

 

 

 

FIGURE 6-16 Package deployment model

 

Tip You can reverse the process by using the Project menu, or the project’s context

menu in Solution Explorer, to convert a package deployment model project to a project deployment

model.

 

Project Deployment Model Features

 

In this section, we provide an overview of the project deployment model features to help you

understand

how you use these features in combination to manage deployed projects. Later in this

chapter, we explain each of these features in more detail and provide links to additional information

available online.

 

Although you can continue to work with the package deployment model if you prefer, the primary

advantage of using the new project deployment model is the improvement in package management

across multiple environments. For example, a package is commonly developed on one server,

tested on a separate server, and eventually implemented on a production server. With the package

deployment model, you can use a variety of techniques to provide connection strings for the correct

environment at runtime, each of which requires you to create at least one configuration file and,

optionally, maintain SQL Server tables or environment variables. Although this approach is flexible,

it can also be confusing and prone to error. The project deployment model continues to separate

run-time values from the packages, but it uses object collections in the Integration Services catalog to

store these values and to define relationships between packages and these object collections, known

as parameters, environments, and environment variables.

 

 

 

¦ ¦ Catalog The catalog is a dedicated database that stores packages and related configuration

information accessed at package runtime. You can manage package configuration and execution

by using the catalog’s stored procedures and views or by using the graphical interface in

SQL Server Management Studio.

¦ ¦ Parameters As Table 6-1 shows, the project deployment model relies on parameters to

change task properties during package execution. Parameters can be created within a project

scope or within a package scope. When you create parameters within a project scope, you

apply a common set of parameter values across all the packages contained in the project. You

can then use parameters in expressions or tasks, much the same way that you use variables.

¦ ¦ Environments Each environment is a container of variables you associate with a package at

runtime. You can create multiple environments to use with a single package, but the package

can use variables from only one environment during execution. For example, you can create

environments for development, test, and production, and then execute a package using one

of the applicable environments.

¦ ¦ Environment variables An environment variable contains a literal value that Integration

Services assigns to a parameter during package execution. After deploying a project, you can

associate a parameter with an environment variable. The value of the environment variable

resolves during package execution.

 

 

Project Deployment Workflow

 

The project deployment workflow includes not only the process of converting design-time objects in

SSDT into database objects stored in the Integration Services catalog, but also the process of retrieving

database objects from the catalog to update a package design or to use an existing package as a

template for a new package. To add a project to the catalog or to retrieve a project from the catalog,

you use a project-deployment file that has an ISPAC file extension. There are four stages of the

project

deployment workflow in which the ISPAC file plays a role: build, deploy, import, and convert.

In this section, we review each of these stages.

 

Build

 

When you use the project deployment model for packages, you use SSDT to develop one or more

packages as part of an Integration Services project. In preparation for deployment to the catalog,

which serves as a centralized repository for packages and related objects, you build the Integration

Services project in SSDT to produce an ISPAC file. The ISPAC file is the project deployment file that

contains project information, all packages in the Integration Services project, and parameters.

 

Before performing the build, there are two additional tasks that might be necessary:

 

¦ ¦ Identify entry-point package If one of the packages in the project is the package that

triggers the execution of the other packages in the project, directly or indirectly, you should

flag that package as an entry-point package. You can do this by right-clicking the package in

 

 

 

 

Solution Explorer and selecting Entry-Point Package. An administrator uses this flag to identify

the package to start when a package contains multiple projects.

¦ ¦ Create project and package parameters You use project-level or package-level

parameters

to provide values for use in tasks or expressions at runtime, which you learn

more about how to do later in this chapter in the “Parameters” section. In SSDT, you assign

parameter

values to use as a default. You also mark a parameter as required, which prevents a

package from executing until you assign a value to the variable.

 

 

During the development process in SSDT, you commonly execute a task or an entire package

within

SSDT to test results before deploying the project. SSDT creates an ISPAC file to hold the

information

required to execute the package and stores it in the bin folder for the Integration Services

project. When you finish development and want to prepare the ISPAC file for deployment, use the

Build menu or press F5.

 

Deploy

 

The deployment process uses the ISPAC file to create database objects in the catalog for the project,

packages, and parameters, as shown in Figure 6-17. To do this, you use the Integration Services

Deployment

Wizard, which prompts you for the project to deploy and the project to create or update

as part of the deployment. You can also provide literal values or specify environment variables as

default parameter values for the current project version. These parameter values that you provide in

the wizard are stored in the catalog as server defaults for the project, and they override the default

parameter values stored in the package.

 

SQL Server

Database Engine

(Source)

SQL Server

Database Engine

(Destination)

Integration Services

Catalog

Integration Services

Catalog

Project Project

Packages Packages

Project

Deployment File

Deployment

Wizard

 

FIGURE 6-17 Deployment of the ISPAC file to the catalog

 

You can launch the wizard from within SSDT by right-clicking the project in Solution Explorer and

selecting Deploy. However, if you have an ISPAC file saved to the file system, you can double-click the

file to launch the wizard.

 

 

 

Import

 

When you want to update a package that has already been deployed or to use it as basis for a new

package, you can import a project into SSDT from the catalog or from an ISPAC file, as shown in

Figure 6-18. To import a project, you use the Integration Services Import Project Wizard, which is

available in the template list when you create a new project in SSDT.

 

SQL Server

Database Engine

(Source)

SQL Server

Business Intelligence Development Studio

Integration Services

Catalog

Project

Packages

Project

Packages

Project

Deployment File

Import Project

Wizard

 

FIGURE 6-18 Import a project from the catalog or an ISPAC file

 

Convert

 

If you have legacy packages and configuration files, you can convert them to the latest version of

Integration Services, as shown in Figure 6-19. The Integration Services Project Conversion Wizard is

available in both SSDT and in SQL Server Management Studio. Another option is to use the Integration

Services Package Upgrade Wizard available on the Tools page of the SQL Server Installation

Center.

 

Legacy Packages

and Configurations

SQL Server

2005 Package

Configurations

Project

Deployment File

Migration

Wizard

SQL Server

2005 Package

 

FIGURE 6-19 Convert existing DTSX files and configurations to an ISPAC file

 

 

 

Note You can use the Conversion Wizard to migrate packages created using SQL Server

2005 Integration Services and later. If you use SQL Server Management Studio, the original

DTSX files are not modified, but used only as a source to produce the ISPAC file containing

the upgraded packages.

 

In SSDT, open a package project, right-click the project in Solution Explorer, and select Convert To

Project Deployment Model. The wizard upgrades the DTPROJ file for the project and the DTSX files

for the packages.

 

The behavior of the wizard is different in SQL Server Management Studio. There you right-click the

Projects node of the Integration Services catalog in Object Explorer and select Import Packages. The

wizard prompts you for a destination location and produces an ISPAC file for the new project and the

upgraded packages.

 

Regardless of which method you use to convert packages, there are some common steps that

occur

as packages are upgraded:

 

¦ ¦ Update Execute Package tasks If a package in a package project contains an Execute

Package

task, the wizard changes the external reference to a DTSX file to a project reference

to a package contained within the same project. The child package must be in the same

package

project you are converting and must be selected for conversion in the wizard.

¦ ¦ Create parameters If a package in a package project uses a configuration, you can choose

to convert the configuration to parameters. You can add configurations belonging to other

projects to include them in the conversion process. Additionally, you can choose to remove

configurations from the upgraded packages. The wizard uses the configurations to prompt

you for properties to convert to parameters, and it also requires you to specify project scope

or package scope for each parameter.

¦ ¦ Configure parameters The Conversion Wizard allows you to specify a server value for each

parameter and whether to require the parameter at runtime.

 

 

Parameters

 

As we explained in the previous section, parameters are the replacement for configurations in legacy

packages, but only when you use the project deployment model. The purpose of configurations was

to provide a way to change values in a package at runtime without requiring you to open the package

and make the change directly. You can establish project-level parameters to assign a value to one

or more properties across multiple packages, or you can have a package-level parameter when you

need to assign a value to properties within a single package.

 

 

 

Project Parameters

 

A project parameter shares its values with all packages within the same project. To create a project

parameter in SSDT, follow these steps:

 

1. In Solution Explorer, double-click Project.params.

2. Click the Add Parameter button on the toolbar in the Project.params window.

3. Type a name for the parameter in the Name text box, select a data type, and specify a value

for the parameter as shown here. The parameter value you supply here is known as the design

default value.

 

 

 

 

Note The parameter value is a design-time value that can be overwritten during

or

after deployment to the catalog. You can use the Add Parameters To Configuration

button on the toolbar (the third button from the left) to add selected

parameters to

Visual Studio project configurations, which is useful for testing package executions

under

a variety of conditions.

 

4. Save the file.

 

 

Optionally, you can configure the following properties for each parameter:

 

¦ ¦ Sensitive By default, this property is set to False. If you change it to True, the parameter

value is encrypted when you deploy the project to the catalog. If anyone attempts to view the

parameter value in SQL Server Management Studio or by accessing Transact-SQL views, the

parameter value will display as NULL. This setting is important when you use a parameter to

set a connection string property and the value contains specific credentials.

¦ ¦ Required By default, this property is also set to False. When the value is True, you must

configure

a parameter value during or after deployment before you can execute the package.

The Integration Services engine will ignore the parameter default value that you specify on

this screen when the Required property is True and deploy the package to the catalog.

¦ ¦ Description This property is optional, but it allows you to provide documentation to an

administrator responsible for managing packages deployed to the catalog.

 

 

 

 

Package Parameters

 

Package parameters apply only to the package in which they are created and cannot be shared with

other packages. The center tab in the package designer allows you to access the Parameters window

for your package. The interface for working with package parameters is identical to the project parameters

interface.

 

Parameter Usage

 

After creating project or package parameters, you are ready to implement the parameters in your

package much like you implement variables. That is, anywhere you can use variables in expressions

for tasks, data flow components, or connection managers, you can also use parameters.

 

As one example, you can reference a parameter in expressions, as shown in Figure 6-20. Notice the

parameter appears in the Variables And Parameters list in the top left pane of the Expression Builder.

You can drag the parameter to the Expression text box and use it alone or as part of a more complex

expression. When you click the Evaluate Expression button, you can see the expression result based

on the design default value for the parameter.

 

 

 

FIGURE 6-20 Parameter usage in an expression

 

 

 

Note This expression uses a project parameter that has a prefix of $Project. To create an

expression that uses a package parameter, the parameter prefix is $Package.

 

You can also directly set a task property by right-clicking the task and selecting Parameterize on

the context menu. The Parameterize dialog box displays as shown in Figure 6-21. You select a property,

and then choose whether to create a new parameter or use an existing parameter. If you create

a new parameter, you specify values for each of the properties you access in the Parameters window.

Additionally, you must specify whether to create the parameter within package scope or project

scope.

 

 

 

FIGURE 6-21 Parameterize task dialog box

 

Post-Deployment Parameter Values

 

The design default values that you set for each parameter in SSDT are typically used only to supply a

value for testing within the SSDT environment. You can replace these values during deployment by

specifying server default values when you use the Deployment Wizard or by configuring execution

values when creating an execution object for deployed projects.

 

 

 

Figure 6-22 illustrates the stage at which you create each type of parameter value. If a parameter

has no execution value, the Integration Services engine uses the server default value when executing

the package. Similarly, if there is no server default value, package execution uses the design default

value. However, if a parameter is marked as required, you must provide either a server default value

or an execution value.

 

Design Deployment

Project

Package

Parameter

“pkgOptions”

Package

Design Default Value = 1

Execution

Project

Package

Parameter

“pkgOptions”

Package

Project

Package

Parameter

“pkgOptions”

Package

Server Default Value = 3

Design Default Value = 1

Exception Value = 5

Server Default Value = 3

Design Default Value = 1

 

FIGURE 6-22 Parameter values by stage

 

Note A package will fail when the Integration Services engine cannot resolve a parameter

value. For this reason, it is recommended that you validate projects and packages as

described

in the “Validation” section of this chapter.

 

Server Default Values

 

Server default values can be literal values or environment variable references (explained later in this

chapter), which in turn are literal values. To configure server defaults in SQL Server Management

Studio,

you right-click the project or package in the Integration Services node in Object Explorer,

select

Configure, and change the Value property of the parameter, as shown in Figure 6-23. This

server

default value persists even if you make changes to the design default value in SSDT and

redeploy

the project.

 

 

 

 

 

FIGURE 6-23 Server default value configuration

 

Execution Parameter Values

 

The execution parameter value applies only to a specific execution of a package and overrides all other

values. You must explicitly set the execution parameter value by using the catalog.set_ execution_

parameter_value stored procedure. There is no interface available in SQL Server Management Studio

to set an execution parameter value.

 

set_execution_parameter_value [ @execution_id = execution_id

, [ @object_type = ] object_type

, [ @parameter_name = ] parameter_name

, [ @parameter_value = ] parameter_value

 

To use this stored procedure, you must supply the following arguments:

 

¦ ¦ execution_id You must obtain the execution_id for the instance of the execution. You can

use the catalog.executions view to locate the applicable execution_id.

¦ ¦ object_type The object type specifies whether you are setting a project parameter or a

package parameter. Use a value of 20 for a project parameter and a value of 30 for a package

parameter.

¦ ¦ parameter_name The name of the parameter must match the parameter stored in the

catalog.

¦ ¦ parameter_value Here you provide the value to use as the execution parameter value.

 

 

 

 

Integration Services Catalog

 

The Integration Services catalog is a new feature to support the centralization of storage and the

administration of packages and related configuration information. Each SQL Server instance can

host only one catalog. When you deploy a project using the project deployment model, the project

and its components are added to the catalog and, optionally, placed in a folder that you specify in

the Deployment

Wizard. Each folder (or the root level if you choose not to use folders) organizes its

contents

into two groups: projects and environments, as shown in Figure 6-24.

 

SQL Server Database Engine

Integration Services Catalog

Environment

Reference

Folder

Project

Project

Parameter

Package

Parameter

Package

Environment

Environment

Variable

 

FIGURE 6-24 Catalog database objects

 

Catalog Creation

 

Installation of Integration Services on a server does not automatically create the catalog. To do this,

follow these steps:

 

1. In SQL Server Management Studio, connect to the SQL Server instance, right-click the

Integration

Services Catalogs folder in Object Explorer, and select Create Catalog.

2. In the Create Catalog dialog box, you can optionally select the Enable Automatic Execution Of

Integration Services Stored Procedure At SQL Server Startup check box. This stored procedure

performs a cleanup operation when the service restarts and adjusts the status of packages

that were executing when the service stopped.

3. Notice that the catalog database name cannot be changed from SSISDB, as shown in the

following

figure, so the final step is to provide a strong password and then click OK. The

password

creates a database master key that Integration Services uses to encrypt sensitive

data stored in the catalog.

 

 

 

 

 

 

After you create the catalog, you will see it appear twice as the SSISDB database in Object Explorer.

It displays under both the Databases node as well as the Integration Services node. In the Databases

node, you can interact with it as you would any other database, using the interface to explore

database

objects. You use the Integration Services node to perform administrative tasks.

 

Note In most cases, multiple options are available for performing administrative tasks with

the catalog. You can use the graphical interface by opening the applicable dialog box for a

selected catalog object, or you can use Transact-SQL views and stored procedures to view

and modify object properties. For more information about the Transact-SQL API, see

http://msdn.microsoft.com/en-us/library/ff878003(v=SQL.110).aspx. You can also use

Windows PowerShell to perform administrative tasks by using the SSIS Catalog Managed

Object Model. Refer to http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.management.

integrationservices(v=sql.110).aspx for details about the API.

 

Catalog Properties

 

The catalog has several configurable properties. To access these properties, right-click SSISDB under

the Integration Services node and select Properties. The Catalog Properties dialog box, as shown in

Figure 6-25, displays several properties.

 

 

 

 

 

FIGURE 6-25 Catalog Properties dialog box

 

Encryption

 

Notice in Figure 6-25 that the default encryption algorithm is AES_256. If you put the SSISDB database

in single-user mode, you can choose one of the other encryption algorithms available:

 

¦ ¦ DES

¦ ¦ TRIPLE_DES

¦ ¦ TRIPLE_DES_3KEY

¦ ¦ DESX

¦ ¦ AES_128

¦ ¦ AES_192

 

 

Integration Services uses encryption to protect sensitive parameter values. When anyone uses the

SQL Server Management Studio interface or the Transact-SQL API to query the catalog, the parameter

value displays only a NULL value.

 

Operations

 

Operations include activities such as package execution, project deployment, and project validation,

to name a few. Integration Services stores information about these operations in tables in the catalog.

You can use the Transact-SQL API to monitor operations, or you can right-click the SSISDB database

on the Integration Services node in Object Explorer and select Active Operations. The Active

Operations

dialog box displays the operation identifier, its type, name, the operation start time,

and the caller of the operation. You can select an operation and click the Stop button to end the operation.

 

 

 

 

Periodically, older data should be purged from these tables to keep the catalog from growing

unnecessarily large. By configuring the catalog properties, you can control the frequency of the SQL

Server Agent job that purges the stale data by specifying how many days of data to retain. If you

prefer, you can disable the job.

 

Project Versioning

 

Each time you redeploy a project with the same name to the same folder, the previous version

remains

in the catalog until ten versions are retained. If necessary, you can restore a previous version

by following these steps:

 

1. In Object Explorer, locate the project under the SSISDB node.

2. Right-click the project, and select Versions.

3. In the Project Versions dialog box, shown here, select the version to restore and click the

Restore

To Selected Version button:

 

 

 

 

4. Click Yes to confirm, and then click OK to close the information message box. Notice the

selected version is now flagged as the current version, and that the other version remains

available as an option for restoring.

 

 

You can modify the maximum number of versions to retain by updating the applicable

catalog

property. If you increase this number above the default value of ten, you should continually

monitor the size of the catalog database to ensure that it does not grow too large. To manage the

size of the catalog, you can also decide whether to remove older versions periodically with a SQL

Server agent job.

 

 

 

Environment Objects

 

After you deploy projects to the catalog, you can create environments to work in tandem with

parameters to change parameter values at execution time. An environment is a collection of environment

variables. Each environment variable contains a value to assign to a parameter. To connect an

environment to a project, you use an environment reference. Figure 6-26 illustrates the relationship

between parameters, environments, environment variables, and environment references.

 

Integration Services Catalog

Folder

Project

Project

Parameter

“p1”

Package

Parameter

“p2”

Package

Environment

Environment

Variable

“ev1”

Environment

Variable

“ev2”

Environment

Reference

 

FIGURE 6-26 Environment objects in the catalog

 

Environments

 

One convention you can use is to create one environment for each server you will use for package

execution. For example, you might have one environment for development, one for testing, and one

for production. To create a new environment using the SQL Server Management Studio interface,

follow

these steps:

 

1. In Object Explorer, expand the SSISDB node and locate the Environments folder that

corresponds

to the Projects folder containing your project.

2. Right-click the Environments folder, and select Create Environment.

3. In the Create Environment dialog box, type a name, optionally type a description, and click

OK.

 

 

Environment Variables

 

For each environment, you can create a collection of environment variables. The properties you

configure

for an environment variable are the same ones you configure for a parameter, which is

understandable when you consider that you use the environment variable to replace the parameter

value at runtime. To create an environment variable, follow these steps:

 

1. In Object Explorer, locate the environment under the SSISDB node.

 

 

 

 

2. Right-click the environment, and select Properties to open the Environment Properties dialog

box.

3. Click Variables to display the list of existing environment variables, if any, as shown here:

 

 

 

 

4. On an empty row, type a name for the environment variable in the Name text box, select a

data type in the Type column, type a description (optional), type a value in the Value column

for the environment variable, and select the Sensitive check box if you want the value to be

encrypted in the catalog. Continue adding environment variables on this page, and click OK

when you’re finished.

5. Repeat this process by adding the same set of environment variables to other environments

you intend to use with the same project.

 

 

Environment References

 

To connect environment variables to a parameter, you create an environment reference. There are

two types of environment references: relative and absolute. When you create a relative environment

reference, the parent folder for the environment folder must also be the parent folder for the project

folder. If you later move the package to another without also moving the environment, the package

execution will fail. An alternative is to use an absolute reference, which maintains the relationship

between the environment and the project without requiring them to have the same parent folder.

 

The environment reference is a property of the project. To create an environment reference, follow

these steps:

 

1. In Object Explorer, locate the project under the SSISDB node.

2. Right-click the project, and select Configure to open the Configure <Project> dialog box.

3. Click References to display the list of existing environment references, if any.

 

 

 

 

4. Click the Add button and select an environment in the Browse Environments dialog box. Use

the Local Folder node for a relative environment reference, or use the SSISDB node for an

absolute environment reference.

 

 

 

 

5. Click OK twice to create the reference. Repeat steps 4 and 5 to add reference for all other applicable

environments.

6. In the Configure <Project> dialog box, click Parameters to switch to the parameters page.

7. Click the ellipsis button to the right of the Value text box to display the Set Parameter Value

dialog box, select the Use Environment Variable option, and select the applicable variable in

the drop-down list, as shown here:

 

 

 

 

8. Click OK twice.

 

 

 

 

You can create multiple references for a project, but only one environment will be active during

package execution. At that time, Integration Services will evaluate the environment variable based on

the environment associated with the current execution instance as explained in the next section.

 

Administration

 

After the development and deployment processes are complete, it’s time to become familiar with the

administration tasks that enable operations on the server to keep running.

 

Validation

 

Before executing packages, you can use validation to verify that projects and packages are likely

to run successfully, especially if you have configured parameters to use environment variables. The

validation process ensures that server default values exist for required parameters, that environment

references are valid, and that data types for parameters are consistent between project and package

configurations and their corresponding environment variables, to name a few of the validation checks.

 

To perform the validation, right-click the project or package in the catalog, click Validate, and

select

the environments to include in the validation: all, none, or a specific environment. Validation

occurs asynchronously, so the Validation dialog box closes while the validation processes. You can

open the Integration Services Dashboard report to check the results of validation. Your other options

are to right-click the SSISDB node in Object Explorer and select Active Operations or to use of the

Transact-SQL API to monitor an executing package.

 

Package Execution

 

After deploying a project to the catalog and optionally configuring parameters and environment

references, you are ready to prepare your packages for execution. This step requires you to create

a SQL Server object called an execution. An execution is a unique combination of a package and its

corresponding

parameter values, whether the values are server defaults or environment references.

To configure and start an execution instance, follow these steps:

 

1. In Object Explorer, locate the entry-point package under the SSISDB node.

 

 

 

 

2. Right-click the project, and select Execute to open the Execute Package dialog box, shown

here:

 

 

 

 

3. Here you have two choices. You can either click the ellipsis button to the right of the value and

specify a literal execution value for the parameter, or you can select the Environment check

box at the bottom of the dialog box and select an environment in the corresponding dropdown

list.

 

 

You can continue configuring the execution instance by updating properties on the Connection

Managers tab and by overriding property values and configuring logging on the Advanced tab. For

more information about the options available in this dialog box, see http://msdn.microsoft.com/en-us

/library/hh231080(v=SQL.110).aspx.

 

When you click OK to close the Execute Package dialog box, the package execution begins.

Because

package execution occurs asynchronously, the dialog box does not need to stay open

during execution. You can use the Integration Services Dashboard report to monitor the execution

status, or right-click the SSISDB node and select Active Operations. Another option is the use of the

Transact-

SQL API to monitor an executing package.

 

More often, you will schedule package execution by creating a Transact-SQL script that starts

execution

and save the script to a file that you can then schedule using a SQL Server agent job. You

add a job step using the Operating System (CmdExec) step type, and then configure the step to use

the sqlcmd.exe utility and pass the package execution script to the utility as an argument. You run

the job using the SQL Server Agent service account or a proxy account. Whichever account you use, it

must have permissions to create and start executions.

 

 

 

Logging and Troubleshooting Tools

 

Now that Integration Services centralizes package storage and executions on the server and has

access to information generated by operations, server-based logging is supported and operations

reports are available in SQL Server Management Studio to help you monitor activity on the server and

troubleshoot problems when they occur.

 

Package Execution Logs

 

In legacy Integration Services packages, there are two options you can use to obtain logs during

package execution. One option is to configure log providers within each package and associate log

providers with executables within the package. The other option is to use a combination of Execute

SQL statements or script components to implement a custom logging solution. Either way, the steps

necessary to enable logging are tedious in legacy packages.

 

With no configuration required, Integration Services stores package execution data in the

[catalog].[

executions] table. The most important columns in this table include the start and end times

of package execution, as well as the status. However, the logging mechanism also captures information

related to the Integration Services environment, such as physical memory, the page file size, and

available CPUs. Other tables provide access to parameter values used during execution, the duration

of each executable within a package, and messages generated during package execution. You can

easily write ad hoc queries to explore package logs or build your own custom reports using Reporting

Services for ongoing monitoring of the log files.

 

Note For a thorough walkthrough of the various tables in which package execution log

data is stored, see “SSIS Logging in Denali,” a blog post by Jamie Thomson at

http://sqlblog.com/blogs/jamie_thomson/archive/2011/07/16/ssis-logging-in-denali.aspx.

 

Data Taps

 

A data tap is similar in concept to a data viewer, except that it captures data at a specified point in

the pipeline during package execution outside of SSDT. You can use the T-SQL stored procedure

catalog.

add_data_tap to tap into the data flow during execution if the package has been deployed

to SSIS. The captured data from the data flow is stored in a CSV file you can review after package

execution

completes. No changes to your package are necessary to use this feature.

 

Reports

 

Before you build custom reports from the package execution log tables, review the built-in

reports

now available in SQL Server Management Studio for Integration Services. These reports provide

information

on package execution results for the past 24 hours (as shown in Figure 6-27),

performance,

and error messages from failed package executions. Hyperlinks in each report allow

you to drill through from summary to detailed information to help you diagnose package execution problems.

 

 

 

 

 

 

FIGURE 6-27 Integration Services operations dashboard.

 

To view the reports, you right-click the SSISDB node in Object Explorer, point to Reports, point to

Standard Reports, and then choose from the following list of reports:

 

¦ ¦ All Executions

¦ ¦ All Validations

¦ ¦ All Operations

¦ ¦ Connections

 

 

 

 

Security

 

Packages and related objects are stored securely in the catalog using encryption. Only members of

the new SQL Server database role ssis_admin or members of the existing sysadmin role have permissions

to all objects in the catalog. Members of these roles can perform operations such as creating

the catalog, creating folders in the catalog, and executing stored procedures, to name a few.

 

Members of the administrative roles delegate administrative permissions to users who need

to manage a specific folder. Delegation is useful when you do not want to give these users access

to the higher privileged roles. To give a user folder-level access, you grant the MANAGE_OBJECT_PERMISSIONS

permission to the user.

 

For general permissions management, open the Properties dialog box for a folder (or any other

securable object) and go to the Permissions page. On that page, you can select a security principal

by name and then set explicit Grant or Deny permissions as appropriate. You can use this method to

secure folders, projects, environments, and operations.

 

Package File Format

 

Although legacy packages stored as DTSX files are formatted as XML, their structure is not compatible

with differencing tools and source-control systems you might use to compare packages. In the

current

version of Integration Services, the package file format is pretty-printed, with properties

formatted

as attributes rather than as elements. (Pretty-printing is the enhancement of code with

syntax conventions for easier viewing.) Moreover, attributes are listed alphabetically and attributes

configured with default values have been eliminated. Collectively, these changes not only help you

more easily locate information in the file, but you can more easily compare packages with automated

tools and more reliably merge packages that have no conflicting changes.

 

Another significant change to the package file format is the replacement of the meaningless

numeric lineage identifiers with a refid attribute with a text value that represents the path to the

referenced object. For example, a refid for the first input column of an Aggregate transformation in a

data flow task called Data Flow Task in a package called Package looks like this:

 

Package\Data Flow Task\Aggregate.Inputs[Aggregate Input 1].Columns[LineTotal]

 

Last, annotations are no longer stored as binary streams. Instead, they appear in the XML file as

clear text. With better access to annotations in the file, the more likely it is that annotations can be

programmatically extracted from a package for documentation purposes.

 

 

 

141

 

CHAP TE R 7

 

Data Quality Services

 

The quality of data is a critical success factor for many data projects, whether for general business

operations or business intelligence. Bad data creeps into business applications as a result of user

entry, corruption during transmission, business processes, or even conflicting data standards across

data sources. The Data Quality Services (DQS) feature of Microsoft SQL Server 2012 is a set of

technologies

you use to measure and manage data quality through a combination of computer-assisted

and manual processes. When your organization has access to high-quality data, your business

process can operate more effectively and managers can rely on this data for better decision-making.

By centralizing

data quality management, you also reduce the amount of time that people spend

reviewing

and correcting data.

 

Data Quality Services Architecture

 

In this section, we describe the two primary components of DQS: the Data Quality Server and Data

Quality Client. The DQS architecture also includes components that are built into other SQL Server

2012 features. For example, Integration Services has the DQS Cleansing transformation you use to

apply data-cleansing rules to a data flow pipeline. In addition, Master Data Services supports DQS

matching so that you can de-duplicate data before adding it as master data. We explain more about

these components in the “Integration” section of this chapter. All DQS components can coexist on the

same server, or they can be installed on separate servers.

 

Data Quality Server

 

The Data Quality Server is the core component of the architecture that manages the storage of

knowledge and executes knowledge-related processes. It consists of a DQS engine and multiple

databases

stored in a local SQL Server 2012 instance. These databases contain knowledge bases,

stored procedures for managing the Data Quality Server and its contents, and data about cleansing,

matching, and data-profiling activities.

 

Installation of the Data Quality Server is a multistep process. You start by using SQL Server Setup

and, at minimum, selecting the Database Engine and Data Quality Services on the Feature Selection

page. Then you continue installation by opening the Data Quality Services folder in the Microsoft SQL

Server 2012 program group on the Start menu, and launching Data Quality Server Installer. A command

window opens, and a prompt appears for the database master key password. You must supply

 

 

 

142 PART 2 Business Intelligence Development

 

a strong password having at least eight characters and including at least one uppercase letter, one

lowercase letter, and one special character.

 

After you provide a valid password, installation of the Data Quality Server continues for several

minutes. In addition to creating and registering assemblies on the server, the installation process

creates

the following databases on a local instance of SQL Server 2012:

 

¦ ¦ DQS_MAIN As its name implies, this is the primary database for the Data Quality Server. It

contains the published knowledge bases as well as the stored procedures that support the

DQS engine. Following installation, this database also contains a sample knowledge base

called DQS data, which you can use to cleanse country data or data related to geographical

locations in the United States.

¦ ¦ DQS_PROJECTS This database is for internal use by the Data Quality Server to store data

related to managing knowledge bases and data quality projects.

¦ ¦ DQS_STAGING_DATA You can use this database as intermediate storage for data that

you want to use as source data for DQS operations. DQS can also use this database to store

processed

data that you can later export.

 

 

Before users can use client components with the Data Quality Server, a user with sysadmin

privileges

must create a SQL Server login for each user and map each user to the DQS_MAIN

database

using one of the following database roles created at installation of the Data Quality Server:

 

¦ ¦ dqs_administrator A user assigned to this role has all privileges available to the other

roles plus full administrative privileges, with the exception of adding new users. Specifically, a

member of this role can stop any activity or stop a process within an activity and perform any

configuration task using Data Quality Client.

¦ ¦ dqs_kb_editor A member of this role can perform any DQS activity except administration.

A user must be a member of this role to create or edit a knowledge base.

¦ ¦ dqs_kb_operator This role is the most limited of the database roles, allowing its members

only to edit and execute data quality projects and to view activity-monitoring data.

 

 

Important You must use SQL Server Configuration Manager to enable the TCP/IP protocol

for the SQL Server instance hosting the DQS databases before remote clients can connect

to the Data Quality Server.

 

Data Quality Client

 

Data Quality Client is the primary user interface for the Data Quality Server that you install as a

stand-

alone application. Business users can use this application to interactively work with data

quality

projects, such as cleansing or data profiling. Data stewards use Data Quality Client to create

or maintain

knowledge bases, and DQS administrators use it to configure and manage the Data Quality

Server.

 

 

 

CHAPTER 7 Data Quality Services 143

 

To install this application, use SQL Server Setup and select Data Quality Client on the Feature

Selection

page. It requires the Microsoft .NET Framework 4.0, which installs automatically if necessary.

 

Note If you plan to import data from Microsoft Excel, you must install Excel on the Data

Quality Client computer. DQS supports both 32-bit and 64-bit versions of Excel 2003, but it

supports only the 32-bit version of Excel 2007 or 2010 unless you save the workbook as an

XLS or CSV file.

 

If you have been assigned to one of the DQS database roles, or if you have sysadmin privileges

on the SQL Server instance hosting the Data Quality Server, you can open Data Quality Client, which

is found in the Data Quality Services folder of the Microsoft SQL Server 2012 program group on the

Start menu. You must then identify the Data Quality Server to establish the client-server connection. If

you click the Options button, you can select a check box to encrypt the connection.

 

After opening Data Quality Client, the home screen provides access to the following three types of

tasks:

 

¦ ¦ Knowledge Base Management You use this area of Data Quality Client to create a new

knowledge base, edit an existing knowledge base, use knowledge discovery to enhance a

knowledge base with additional values, or create a matching policy for a knowledge base.

¦ ¦ Data Quality Projects In this area, you create and run data quality projects to perform

data-cleansing or data-matching tasks.

¦ ¦ Administration This area allows you to view the status of knowledge base management

activities,

data quality projects, and DQS Cleansing transformations used in Integration

Services

packages. In addition, it provides access to configuration properties for the Data

Quality Server, logging, and reference data services.

 

 

Knowledge Base Management

 

In DQS, you create a knowledge base to store information about your data, including valid and invalid

values and rules to apply for validating and correcting data. You can generate a knowledge base from

sample data, or you can manually create one. You can reuse a knowledge base with multiple data

quality projects and enhance it over time with the output of cleansing and matching projects. You

can give responsibility for maintaining the knowledge base to data stewards. There are three activities

that you or data stewards perform using the Knowledge Base Management area of Data Quality

Client: Domain Management, Knowledge Discovery, and Matching Policy.

 

Domain Management

 

After creating a knowledge base, you manage its contents and rules through the Domain

Management

activity. A knowledge base is a logical collection of domains, with each domain

corresponding

to a single field. You can create separate knowledge bases for customers and products,

 

 

 

or you can combine these subject areas into a single knowledge base. After you create a domain, you

define trusted values, invalid values, and examples of erroneous data. In addition to this set of values,

you also manage properties and rules for a domain.

 

To prevent potential conflicts resulting from multiple users working on the same knowledge base

at the same time, DQS locks the knowledge base when you begin a new activity. To unlock an activity,

you must publish or discard the results of your changes.

 

Note If you have multiple users who have responsibility for managing knowledge, keep

in mind that only one user at a time can perform the Domain Management activity for a

single knowledge base. Therefore, you might consider using separate knowledge bases for

each area of responsibility.

 

DQS Data Knowledge Base

 

Before creating your own knowledge base, you can explore the automatically installed knowledge

base, DQS Data, to gain familiarity with the Data Quality Client interface and basic knowledge-base

concepts. By exploring DQS Data, you can learn about the types of information that a knowledge

base can contain.

 

To get started, open Data Quality Client and click the Open Knowledge Base button on the home

page. DQS Data displays as the only knowledge base on the Open Knowledge Base page. When

you select DQS Data in the list of existing knowledge bases, you can see the collection of domains

associated

with this knowledge base, as shown in Figure 7-1.

 

 

 

FIGURE 7-1 Knowledge base details for DQS Data

 

 

 

Choose the Domain Management activity in the bottom right corner of the window, and click

Next to explore the domains in this knowledge base. In the Domain list on the Domain Management

page, you choose a domain, such as Country/Region, and access separate tabs to view or change the

knowledge associated with that domain. For example, you can click on the Domain Values tab to see

how values that represent a specific country or region will be corrected when you use DQS to perform

data cleansing, as shown in Figure 7-2.

 

 

 

FIGURE 7-2 Domain values for the Country/Region domain

 

The DQS Data knowledge base, by default, contains only domain values. You can add other

knowledge

to make this data more useful for your own data quality projects. For example, you could

add domain rules to define conditions that identify valid data or create term-based relations to

define how to make corrections to terms found in a string value. More information about these other

knowledge

categories is provided in the “New Knowledge Base” section of this chapter.

 

Tip When you finish reviewing the knowledge base, click the Cancel button and click Yes

to confirm that you do not want to save your work.

 

New Knowledge Base

 

On the home page of Data Quality Client, click the New Knowledge Base button to start the process

of creating a new knowledge base. At a minimum, you provide a name for the knowledge base, but

you can optionally add a description. Then you must specify whether you want to create an empty

knowledge base or create your knowledge base from an existing one, such as DQS Data. Another

option is to create a knowledge base by importing knowledge from a DQS file, which you create by

using the export option from any existing knowledge base.

 

 

 

After creating the knowledge base, you select the Domain Management activity, and click Next so

that you can add one or more domains in the knowledge base. To add a new domain, you can create

the domain manually or import a domain from an existing knowledge base using the applicable

button

in the toolbar on the Domain Management page. As another option, you can create a domain

from the output of a data quality project.

 

Domain When you create a domain manually, you start by defining the following properties for the

domain (as shown in Figure 7-3):

 

¦ ¦ Domain Name The name you provide for the domain must be unique within the knowledge

base and must be 256 characters or less.

¦ ¦ Description You can optionally add a description to provide more information about the

contents of the domain. The maximum number of characters for the description is 2048.

¦ ¦ Data Type Your options here are String, Date, Integer, or Decimal.

¦ ¦ Use Leading Values When you select this option, the output from a cleansing or matching

data quality project will use the leading value in a group of synonyms. Otherwise, the output

will be the input value or its corrected value.

¦ ¦ Normalize String This option appears only when you select String as the Data Type. You

use it to remove special characters from the domain values during the data-processing stage

of knowledge discovery, data cleansing, and matching activities. Normalization might be

helpful for improving the accuracy of matches when you want to de-duplicate data because

punctuation

might be used inconsistently in strings that otherwise are a match.

¦ ¦ Format Output To When you output the results of a data quality project, you can apply

formatting to the domain values if you change this setting from None to one of the available

format options, which will depend on the domain’s data type. For example, you could choose

Mon-yyyy for a Date data type, #,##0.00 for a Decimal data type, #,##0 for an Integer data

type, or Capitalize for a String data type.

¦ ¦ Language This option applies only to String data types. You use it to specify the language to

apply when you enable the Speller.

¦ ¦ Enable Speller You can use this option to allow DQS to check the spelling of values

for a domain with a String data type. The Speller will flag suspected syntax, spelling, and

sentence-

structure errors with a red underscore when you are working on the Domain Values

or Term-

Based Relations tabs of the Domain Management activity, the Manage Domain

Values

step of the Knowledge Discovery activity, or the Manage And View Results step of the Cleansing

activity.

¦ ¦ Disable Syntax Error Algorithms DQS can check string values for syntax errors before

adding

each value to the domain during data cleansing.

 

 

 

 

 

 

FIGURE 7-3 Domain properties in the Create Domain dialog box

 

Domain Values After you create the domain, you can add knowledge to the domain by setting

domain values. Domain

values represent the range of possible values that might exist in a data source

for the current domain, including both correct and incorrect values. The inclusion of incorrect domain

values in a knowledge base allows you to establish rules for correcting those values during cleansing

activities.

 

You use buttons on the Domain Values tab to add a new domain value manually, import new valid

values from Excel, or import new string values with type Correct or Error from a cleansing data quality

project.

 

Tip To import data from Excel, you can use any of the following file types: XLS, XLSX, or

CSV. DQS attempts to add every value found in the file, but only if the value does not already

exist in the domain. DQS imports values in the first column as domain values and

values in other columns as synonyms, setting the value in the first column as the leading

value. DQS will not import a value if it violates a domain rule, if the data type does not

match that of the domain, or if the value is null.

 

When you add or import values, you can adjust the Type to one of the following settings for each

domain value:

 

¦ ¦ Correct You use this setting for domain values that you know are members of the domain

and have no syntax errors.

 

 

 

 

¦ ¦ Error You assign a Type of Error to a domain value that you know is a member of the domain

but has an incorrect value, such as a misspelling or undesired abbreviation. For example, in a

Product domain, you might add a domain value of Mtn 500 Silver 52 with the Error type and

include a corrected value of Mountain-500 Silver, 52. By adding known error conditions to the

domain, you can speed up the data cleansing process by having DQS automatically identify

and fix known errors, which allows you to focus on new errors found during data processing.

However, you can flag a domain value as an Error value without providing a corrected value.

¦ ¦ Invalid You designate a domain value as Invalid when it is not a member of the domain and

you have no correction to associate with it. For example, a value of United States would be an

invalid value in a Product domain.

 

 

After you publish your domain changes to the knowledge base and later return to the Domain

Values tab, you can see the relationship between correct and incorrect values in the domain values

list, as shown in Figure 7-4 for the product Adjustable Race. Notice also the underscore in the user

interface to identify a potential misspelling of “Adj Race.”

 

 

 

FIGURE 7-4 Domain value with Error state and corrected value

 

If there are multiple correct domain values that correspond to a single entity, you can organize

them as a group of synonyms by selecting them and clicking the Set Selected Domain Values As

Synonyms button in the Domain Values toolbar. Furthermore, you can designate one of the synonyms

as a leading value by right-clicking the value and selecting Set As Leading in the context menu.

When DQS encounters any of the other synonyms in a cleansing activity, it replaces the nonleading

synonym

values with the leading value. Figure 7-5 shows an example of synonyms for the leading

value Road-750 Black, 52 after the domain changes have been published.

 

 

 

 

 

FIGURE 7-5 Synonym values with designation of a leading value

 

Data Quality Client keeps track of the changes you make to domain values during your current

session.

To review your work, you click the Show/Hide The Domain Values Changes History Panel

button,

which is accessible by clicking the last button in the Domain Values toolbar.

 

Term-Based Relations Another option you have for adding knowledge to a domain is term-based

relations, which you use to make it easier to find and correct common occurrences in your domain

values. Rather than set up synonyms for variations of a domain value on the Domain Values tab, you

can define a list of string values and specify the corresponding Correct To value on the Term-Based

Relations tab. For example, in the product domain, you could have various products that contain the

abbreviation Mtn, such as Mtn-500 Silver, 40 and Mtn End Caps. To have DQS automatically correct

any occurrence of Mtn within a domain value string to Mountain, you can create a term-based

relation

by specifying a Value/Correct To pair, as shown in Figure 7-6.

 

 

 

FIGURE 7-6 Term-based relation with a value paired to a corrected value

 

 

 

Reference Data You can subscribe to a reference data service (RDS) that DQS uses to cleanse,

standardize,

and enhance your data. Before you can set the properties on the Reference Data tab

for your knowledge base, you must configure the RDS provider as described in the “Configuration”

section

of this chapter.

 

After you configure your DataMarket Account key in the Configuration area, you click the Browse

button on the Reference Data tab to select a reference data service provider for which you have a

current subscription. On the Online Reference Data Service Providers Catalog page, you map the

domain

to an RDS schema column. The letter M displays next to mandatory columns in the RDS

schema that you must include in the mapping.

 

Next, you configure the following settings for the RDS (as shown in Figure 7-7):

 

¦ ¦ Auto Correction Threshold Specify the threshold for the confidence score. During data

cleansing, DQS autocorrects records having a score higher than this threshold.

¦ ¦ Suggested Candidates Specify the number of candidates for suggested values to retrieve

from the reference data service.

¦ ¦ Min Confidence Specify the threshold for the confidence score for suggestions. During data

cleansing, DQS ignores suggestions with a score lower than this threshold.

 

 

 

 

FIGURE 7-7 Reference data service provider settings

 

Domain Rules You use domain rules to establish the conditions that determine whether a domain

value is valid. However, note that domain rules are not used to correct data.

 

After you click the Add A New Domain Rule button on the Domain Rules tab, you provide a name

for the rule and an optional description. Then you define the conditions for the rule in the Build A

Rule pane. For example, if there is a maximum length for a domain value, you can create a rule by

 

 

 

selecting Length Is Less Than Or Equal To in the rule drop-down list and then typing in the condition

value, as shown in Figure 7-8.

 

 

 

FIGURE 7-8 Domain rule to validate the length of domain values

 

You can create compound rules by adding multiple conditions to the rule and specifying whether

the conditions have AND or OR logic. As you build the rule, you can use the Run The Selected Domain

Rule On Test Data button. You must manually enter one or more values as test data for this procedure,

and then click the Test The Domain Rule On All The Terms button. You will see icons display to

indicate whether a value is correct, in error, or invalid. Then when you finish building the rule, you

click the Apply All Rules button to update the status of domain values according to the new rule.

 

Note You can temporarily disable a rule by clearing its corresponding Active check box.

 

Composite Domain Sometimes a complex string in your data source contains multiple terms, each

of which has different rules or different data types and thus requires separate domains. However, you

still need to validate the field as a whole. For example, with a product name like Mountain-500 Black,

40, you might want to validate the model of Mountain-500, the color Black, and the size 40 separately

to confirm the validity of the product name. To address this situation, you can create a composite

domain.

 

Before you can create a composite domain, you must have at least two domains in your knowledge

base. Begin the Domain Management activity, and click the Create A Composite Domain button on

the toolbar. Type a name for the composite domain, provide a description if you like, and then select

the domains to include in the composite domain.

 

 

 

Note After you add a domain to a composite domain, you cannot add it to a second composite

domain.

 

 

CD Properties You can always change, add, or remove domains in the composite domain on the CD

Properties tab. You can also select one of the following parsing methods in the Advanced section of

this tab:

 

¦ ¦ Reference Data If you map the composite domain to an RDS, you can specify that DQS use

the RDS for parsing each domain in the composite domain.

¦ ¦ In Order You use this setting when you want DQS to parse the field values in the same order

that the domains are listed in the composite domain.

¦ ¦ Delimiters When your field contains delimiters, you can instruct DQS to parse values based

on the delimiter you specify, such as a tab, comma, or space. When you use delimiters for

parsing, you have the option to use knowledge-base parsing. With knowledge-base parsing,

DQS identifies the domains for known values in the string and then uses domain knowledge to

determine how to add unknown values to other domains. For example, let’s say that you have

a field in the data source containing the string Mountain-500 Brown, 40. If DQS recognizes

Mountain-500 as a value in the Model domain and 40 as a value in the Size domain, but it

does not recognize Brown in the Color domain, it will add Brown to the Color domain.

 

 

Reference Data You use this tab to specify an RDS provider and map the individual domains of

a composite domain to separate fields of the provider’s RDS schema. For example, you might have

company information for which you want to validate address details. You combine the domains

Address,

City, State, and Zip to create a composite domain that you map to the RDS schema. You then

create a cleansing project to apply the RDS rules to your data.

 

CD Rules You use the CD Rules tab to define cross-domain rules for validating, correcting, and standardizing

domain values for a composite domain. A cross-domain rule uses similar conditions available

to domain rules, but this type of rule must hold true for all domains in the composite domain

rather than for a single domain. Each rule contains an If clause and a Then clause, and each clause

contains one or more conditions and is applicable to separate domains. For example, you can develop

a rule that invalidates a record if the Size value is not S, M, or L when the Model value begins with

Mountain-, as shown in Figure 7-9. As with domain rules, you can create rules with multiple conditions

and you can test a rule before finalizing its definition.

 

 

 

 

 

FIGURE 7-9 Cross-domain rule to validate corresponding size values in composite domain

 

Note If you create a Then clause that uses a definitive condition, DQS will not apply the

rule to both domain values and their synonyms. Definitive conditions are Value Is Equal To,

Value Is Not Equal To, Value Is In, or Value Is Not In. Furthermore, if you use the Value Is

Equal To condition for the Then clause, DQS not only validates data, but also corrects data

using this rule.

 

Value Relations After completing a knowledge-discovery activity, you can view the number of

occurrences

for each combination of values in a composite domain, as shown in Figure 7-10.

 

 

 

FIGURE 7-10 Value relations for a composite domain

 

Linked Domain You can use a linked domain to handle situations that a regular domain cannot

support. For example, if you create a data quality project for a data source that has two fields that use

the same domain values, you must set up two domains to complete the mapping of fields to domains.

Rather than maintain two domains with the same values, you can create a linked domain that inherits

the properties,

values, and rules of another domain.

 

One way to create a linked domain is to open the knowledge base containing the source domain,

right-click the domain, and select Create A Linked Domain. Another way to create a linked domain is

during an activity that requires you to map fields. You start by mapping the first field to a domain and

then attempt to map the second field to the same domain. Data Quality Client prompts you to create

a linked domain, at which time you provide a domain name and description.

 

 

 

After you create the linked domain, you can use the Domain Management activity to complete

tasks such as adding domain values or setting up domain rules by accessing either domain. The

changes are made automatically in the other domain. However, you can change the domain

properties

in the original domain only.

 

End of Domain Management Activity

 

DQS locks the knowledge base when you begin the Domain Management activity to prevent others

from making conflicting changes. If you cannot complete all the changes you need to make in a single

session, you click the Close button to save your work and keep the knowledge base locked. Click the

Finish button when your work is complete. Data Quality Client displays a prompt for you to confirm

the action to take. Click Publish to make your changes permanent, unlock the database, and make the

knowledge base available to others. Otherwise, click No to save your work, keep the database locked,

and exit the Domain Management activity.

 

Knowledge Discovery

 

As an alternative to manually adding knowledge to your knowledge base, you can use the Knowledge

Discovery activity to partially automate that process. You can perform this activity multiple times, as

often as needed, to add domain values to the knowledge base from one or more data sources.

 

To start, you open the knowledge base in the Domain Management area of Data Quality Client

and select the Knowledge Discovery activity. Then you complete a series of three steps: Map,

Discover,

and Manage Domain Values.

 

Map

 

In the Map step, you identify the source data you want DQS to analyze. This data must be available

on the Data Quality Server, either in a SQL Server table or view or in an Excel file. However, it does not

need to be from the same source that you intend to cleanse or de-duplicate with DQS.

 

The purpose of this step is to map columns in the source data to a domain or composite domain in

the knowledge base, as shown in Figure 7-11. You can choose to create a new domain or composite

domain at this point when necessary. When you finish mapping all columns, click the Next button to

proceed to the Discover step.

 

 

 

 

 

FIGURE 7-11 Mapping the source column to the domain for the Knowledge Discovery activity

 

Discover

 

In the Discover step, you use the Start button to begin the knowledge discovery process. As the

process

executes, the Discover step displays the current status for the following three phases of

processing

(as shown in Figure 7-12):

 

¦ ¦ Pre-processing Records During this phase, DQS loads and indexes records from the

source in preparation for profiling the data. The status of this phase displays as the number

of pre-processed records compared to the total number of records in the source. In addition,

DQS updates all data-profiling statistics except the Valid In Domain column. These statistics

are visible

in the Profiler tab and include the total number of records in the source, the total

number of values for the domain by field, and the number of unique values by field. DQS also

compares the values from the source with the values from the domain to determine which

values are found only in the source and identified as new.

¦ ¦ Running Domain Rules DQS uses the domain rules for each domain to update the Valid In

Domain column in the Profiler, and it displays the status as a percentage of completion.

¦ ¦ Running Discovery DQS analyzes the data to add to the Manage Domain Values step and

identifies syntax errors. As this phase executes, the current status displays as a percentage of

completion.

 

 

 

 

 

 

FIGURE 7-12 Source statistics resulting from data discovery analysis

 

You use the source statistics on the Profiler tab to assess the completeness and uniqueness of the

source data. If the source yields few new values for a domain or has a high number of invalid values,

you might consider using a different source. The Profiler tab might also display notifications, as shown

in Figure 7-13, to alert you to such conditions.

 

 

 

FIGURE 7-13 Notifications and source statistics display on the Profiler tab

 

Manage Domain Values

 

In the third step of the Knowledge Discovery activity, you review the results of the data-discovery

analysis. The unique new domain values display in a list with a Type setting and suggested corrections,

where applicable. You can make any necessary changes to the values, type, and corrected

values, and you can add new domain values, delete values, and work with synonyms just like you can

when working on the Doman Values tab of the Domain Management activity.

 

 

 

Notice in Figure 7-14 that the several product models beginning with Mountain- were marked as

Error and DQS proposed a corrected value of Mountain-500 for each of the new domain values. It did

this because Mountain-500 was the only pre-existing product model in the domain. DQS determined

that similar product models found in the source must be misspellings and proposed corrections to

the source values to match them to the pre-existing domain value. In this scenario, if the new product

models are all correct, you can change the Type setting to Correct. When you make this change, Data

Quality Client automatically removes the Correct To value.

 

 

 

FIGURE 7-14 The Knowledge Discovery activity produces suggested corrections

 

After reviewing each domain and making corrections, click the Finish button to end the Knowledge

Discovery activity. Then click the Publish button to complete the activity, update the knowledge base

with the new domain values, and leave the knowledge base in an unlocked state. If you click the No

button instead of the Publish button, the results of the activity are discarded and the knowledge base

is unlocked.

 

Matching Policy

 

Another aspect of adding knowledge to a knowledge base is defining a matching policy. This policy

is necessary for data quality projects that use matching to correct data problems such as misspelled

customer names or inconsistent address formats. A matching policy contains one or more matching

rules that DQS uses to determine the probability of a match between two records.

 

 

 

You begin by opening a knowledge base in the Domain Management area of Data Quality Client

and selecting the Matching Policy activity. The process to create a matching policy consists of three

steps: Map, Matching Policy, and Matching Results.

 

Tip You might consider creating a knowledge base that you use only for matching

projects.

In this matching-only knowledge base, include only domains that have values that

are both discrete and uniquely identify a record, such as names and addresses.

 

Map

 

The first step of the Matching Policy activity is similar to the first step of the Knowledge Discovery

activity. You start the creation of a matching policy by mapping a field from an Excel or SQL Server

data source to a domain or composite domain in the selected knowledge base. If a corresponding

domain does not exist in the knowledge base, you have the option to create one. You must select a

source field in this step if you want to reference that field in a matching rule in the next step. Use the

Next button to continue to the Discover step.

 

Matching Policy

 

In the Matching Policy step, you set up one or more matching rules that DQS uses to assign a

matching

score for each pair of records it compares. DQS considers the records to be a match when

this matching score is greater than the minimum matching score you establish for the matching

policy.

 

To begin, click the Create A Matching Rule button. Next, assign a name, an optional description,

and a minimum matching score to the matching rule. The lowest minimum matching score you can

assign is 80 percent, unless you change the DQS configuration on the Administration page. In the

Rule Editor toolbar, click the Add A New Domain Element button, select a domain, and configure the

matching rule parameters for the selected domain, as shown in Figure 7-15.

 

 

 

FIGURE 7-15 Creation of a matching rule for a matching policy

 

 

 

You can choose from the following values when configuring the Similarity parameter:

 

¦ ¦ Similar You select this value when you want DQS to calculate a matching score for a field

in two records and set the similarity score to 0 (to indicate no similarity) when the matching

score is less than 60. If the field has a numeric data type, you can set a threshold for similarity

using a percentage or integer value. If the field has a date data type, you can set the threshold

using a numeric value for day, month, or year.

¦ ¦ Exact When you want DQS to identify two records to be a match only when the same field

in each record is identical, you select this value. DQS assigns a matching score of 100 for the

domain when the fields are identical. Otherwise, it assigns a matching score of 0.

 

 

Whether you add one or more domains to a matching rule, you must configure the Weight

parameter

for each domain that you do not set as a prerequisite. DQS uses the weight to determine

how the individual domain’s matching score affects the overall matching score. The sum of the weight

values must be equal to 100.

 

When you select the Prerequisite check box for a domain, DQS sets the Similarity parameter to

Exact and considers values in a field to be a match only when they are identical in the two compared

records. Regardless of the result, a prerequisite domain has no effect on the overall matching score

for a record. Using the prerequisite option is an optimization that speeds up the matching process.

 

You can test the rule by clicking the Start button on the Matching Policy page. If the results are

not what you expect, you can modify the rule and test the rule again. When you retest the matching

policy, you can choose to either execute the matching policy on the processed matches from a

previous execution or on data that DQS reloads from the source. The Matching Results tab, shown in

Figure 7-16, displays the results of the current test and the previous test so that you can determine

whether your changes improve the match results.

 

 

 

FIGURE 7-16 Comparison of results from consecutive executions of a matching rule

 

A review of the Profiler tab can help you decide how to modify a match rule. For example, if a field

has a high percentage of unique records, you might consider eliminating the field from a match rule

or lower the weight value. On the other hand, having a low percentage of unique records is useful

only if the field has a high level of completeness. If both uniqueness and completeness are low, you

should exclude the field from the matching policy.

 

You can click the Restore Previous Rule button to revert the rule settings to their prior state if you

prefer. When you are satisfied with the results, click the Next button to continue to the next step.

 

 

 

Matching Results

 

In the Matching Results step, you choose whether to review results as overlapping clusters or

nonoverlapping

clusters. With overlapping clusters, you might see separate clusters that contain the

same records, whereas with nonoverlapping clusters you see only clusters with records in common.

Then you click the Start button to apply all matching rules to your data source.

 

When processing completes, you can view a table that displays a filtered list of matched records

with a color code to indicate the applicable matching rule, as shown in Figure 7-17. Each cluster

of records has a pivot record that DQS randomly selects from the cluster as the record to keep.

Furthermore,

each cluster includes one or more matched records along with its matching score. You

can change the filter to display unmatched records, or you can apply a separate filter to view matched

records having scores greater than or equal to 80, 85, 90, 95, or 100 percent.

 

 

 

FIGURE 7-17 Matched records based on two matching rules

 

You can review the color codes on the Matching Rules tab, and you can evaluate statistics about

the matching results on the Matching Results tab. You can also double-click a matched record in the

list to see the Pivot and Matched Records fields side by side, the score for each field, and the overall

score for the match, as shown in Figure 7-18.

 

 

 

FIGURE 7-18 Matching-score details displaying field scores and an overall score

 

 

 

If necessary, you can return to the previous step to fine-tune a matching rule and then return to

this step. Before you click the Restart button, you can choose the Reload Data From Source option

to copy the source data into a staging table where DQS re-indexes it. Your other option is to choose

Execute On Previous Data to use the data in the staging table without re-indexing it, which could

process the matching policy more quickly.

 

Once you are satisfied with the results, click the Finish button. You then have the option to publish

the matching policy to the knowledge base. At this point, you can use the matching policy with a

matching data quality project.

 

Data Quality Projects

 

When you have a knowledge base in place, you can create a data quality project to use the

knowledge

it contains to cleanse source data or use its matching policy to find matching records

in source data. After you run the data quality project, you can export its results to a SQL Server

database

or to a CSV file. As another option, you can import the results to a domain in the Domain Management

activity.

 

Regardless of which type of data quality project you want to create, you start the project in the

same way, by clicking the New Data Quality Project button on the home page of Data Quality Client.

When you create a new project, you provide a name for the data quality project, provide an optional

description, and select a knowledge base. You can then select the Cleansing activity for any knowledge

base; you can select the Matching activity for a knowledge base only when that knowledge

base has a matching policy. To launch the activity’s wizard, click the Create button. At this point, the

project is locked and inaccessible to other users.

 

Cleansing Projects

 

A cleansing data quality project begins with an analysis of source data using knowledge contained

in a knowledge base and a categorization of that data into groups of correct and incorrect data.

After DQS completes the analysis and categorization process, you can approve, reject, or change the proposed

corrections.

 

 

When you create a cleansing data quality project, the wizard leads you through four steps: Map,

Cleanse, Manage And View Results, and Export.

 

Map

 

The first step in the cleansing data quality project is to map the data source to domains in the

selected

knowledge base, following the same process you used for the Knowledge Discovery and

Matching Policy activities. The data source can be either an Excel file or a table or view in a SQL Server

database. When you finish mapping all columns, click the Next button to proceed to the Cleanse step.

 

 

 

Note If you map a field to a composite domain, only the rules associated with the

composite

domain will apply, rather than the rules for the individual domains assigned

to the composite domain. Furthermore, if the composite domain is mapped to a reference

data service, DQS sends the source data to the reference data service for parsing and

cleansing. Otherwise, DQS performs the parsing using the method you specified for the

composite domain.

 

Cleanse

 

To begin the cleansing process, click the Start button. When the analysis process completes, you

can view the statistics in the Profiler tab, as shown in Figure 7-19. These statistics reflect the results

of categorization that DQS performs: correct records, corrected records, suggested records, and

invalid records. DQS uses advanced algorithms to cleanse data and calculates a confidence score to

determine the category applicable to each record in the source data. You can configure confidence

thresholds

for auto-correction and auto-suggestion. If a record’s confidence score falls below either

of these thresholds and is neither correct nor invalid, DQS categorizes the record as new and leaves it

for you to manually correct if necessary in the next step.

 

 

 

FIGURE 7-19 Profiler statistics after the cleansing process completes

 

Manage and View Results

 

In the next step of the cleansing data quality project, you see separate tabs for each group of records

categorized by DQS: Suggested, New, Invalid, Corrected, and Correct. The tab labels show the number

of records allocated to each group. When you open a tab, you can see a table of domain values

for that group and the number of records containing each domain value, as shown in Figure 7-20.

 

When applicable, you can also see the proposed corrected value, confidence score, and reason

for the proposed correction for each value. If you select a row in the table, you can see the individual

records that contain the original value. You must use the horizontal scroll bar to see all the fields for

the individual records.

 

When you enable the Speller feature for a domain, the cleansing process identifies potential

spelling

errors by displaying a wavy red underscore below the domain value. You can right-click the

value to see suggestions and then select one or add the potential error to the dictionary.

 

 

 

 

 

FIGURE 7-20 Categorized results after automated data cleansing

 

If DQS identifies a value in the source data as a synonym, it suggests a correction to the leading

value. This feature is useful for standardization of your data. You must first enable the domain for

leading values and define synonyms in the knowledge base before running a cleansing data quality

project.

 

After reviewing the proposed corrections, you can either approve or reject the change for each

value or for each record individually. Another option is to click the Approve All Terms button or the

Reject All Terms button in the toolbar. You can also replace a proposed correction by typing a new

value in the Correct To box, and then approve the manual correction. In most cases, approved values

move to the Corrected tab and rejected values move to the Invalid tab. However, if you approve a

value on the New tab, it moves to the Correct tab.

 

Export

 

At no time during the cleansing process does DQS change source data. Instead, you can export the

results of the cleansing data quality project to a SQL Server table or to a CSV file. A preview of the

output displays in the final step of the project, as shown in Figure 7-21.

 

 

 

 

 

FIGURE 7-21 Review of the cleansing results to export

 

Before you export the data, you must decide whether you want to export the data only or export

both the data and cleansing information. Cleansing information includes the original value, the

cleansed value, the reason for a correction, a confidence score, and the categorization of the record.

By default, DQS uses the output format for the domain as defined in the knowledge base unless

you clear the Standardize Output check box. When you click the Export button, Data Quality Client

exports the data to the specified destination. You can then click the Finish button to close and unlock

the data quality project.

 

Matching Projects

 

By using a matching policy defined for a knowledge base, a matching data quality project can

identify both exact and approximate matches in a data source. Ideally, you run the matching process

after running the cleansing process and exporting the results. You can then specify the export file or

destination table as the source for the matching project.

 

A matching data quality project consists of three steps: Map, Matching, and Export.

 

Map

 

The first step for a matching project begins in the same way as a cleansing project, by requiring you

to map fields from the source to a domain. However, in a matching project, you must map a field to

each domain specified in the knowledge base’s matching policy.

 

 

 

Matching

 

In the Matching step, you choose whether to generate overlapping clusters or nonoverlapping

clusters,

and then click the Start button to launch the automated matching process. When the process

completes, you can review a table of matching results by cluster. The interface is similar in functionality

to the one you use when reviewing the matching results during the creation of a matching policy.

However, in the matching project, an additional column includes a check box that you can select to

reject a record as a match.

 

Export

 

After reviewing the matching results, you can export the results as the final step of the matching

project.

As shown in Figure 7-22, you must choose the destination and the content to export. You

have the following two options for exporting content:

 

¦ ¦ Matching Results This content type includes both matched and unmatched records. The

matched records include several columns related to the matching process, including the

cluster identifier, the matching rule that identified the match, the matching score, the approval

status, and a flag to indicate the pivot record.

¦ ¦ Survivorship Results This content type includes only the survivorship record and

unmatched

records. You must select a survivorship rule when choosing this export option to

specify which of the matched records in a cluster are preserved in the export. All other records

in a cluster are discarded. If more than one record satisfies the survivorship criteria, DQS keeps

the record with the lowest record identifier.

 

 

 

 

FIGURE 7-22 Selection of content to export

 

 

 

Important When you click the Finish button, the project is unlocked and available for

later use. However, DQS uses the knowledge-base contents at the time that you finished

the project and ignores any subsequent changes to the knowledge base. To access any

changes to the knowledge base, such as a modified matching policy, you must create a

new matching project.

 

Administration

 

In the Administration feature of Data Quality Client, you can perform activity-monitoring and

configuration

tasks. Activity monitoring is accessible by any user who can open Data Quality Client.

On the other hand, only an administrator can access the configuration tasks.

 

Activity Monitoring

 

You can use the Activity Monitoring page to review the status of current and historic activities

performed

on the Data Quality Server, as shown in Figure 7-23. DQS administrators can terminate an

activity or a step within an activity when necessary by right-clicking on the activity or step.

 

 

 

FIGURE 7-23 Status of activities and status of activity steps of the selected activity

 

 

 

Activities that appear on this page include knowledge discovery, domain management, matching

policy, cleansing projects, matching projects, and the cleansing transformation in an Integration

Services package. You can see who initiated each activity, the start and end time of the activity, and

the elapsed time. To facilitate locating specific activities, you can use a filter to find activities by date

range and by status, type, subtype, knowledge base, or user. When you select an activity on this page,

you can view the related activity details, such as the steps and profiler information.

 

You can click the Export The Selected Activity To Excel button to export the activity details,

process steps, and profiling information to Excel. The export file separates this information into four worksheets:

 

 

¦ ¦ Activity This sheet includes the details about the activity, including the name, type, subtype,

current status, elapsed time, and so on.

¦ ¦ Processes This sheet includes information about each activity step, including current status,

start and end time, and elapsed time.

¦ ¦ Profiler – Source The contents of this sheet depend on the activity subtype. For the

Cleansing

subtype, you see the number of total records, correct records, corrected records,

and invalid records. For the Knowledge Discovery, Domain Management, Matching Policy, and

Matching subtypes, you see the number of records, total values, new values, unique value, and

new unique values.

¦ ¦ Profiler – Fields This sheet’s contents also depend on the activity subtype. For the Cleansing

and SSIS Cleansing subtypes, the sheet contains the following information by field: domain,

corrected values, suggested values, completeness, and accuracy. For the Knowledge Discovery,

Domain Management, Matching Policy, and Matching subtypes, the sheet contains the following

information by field: domain, new value count, unique value count, count of values that

are valid in the domain, and completeness.

 

 

Configuration

 

The Configuration area of Data Quality Client allows you to set up reference data providers, set

properties for the Data Quality Server, and configure logging. You must be a DQS administrator to

perform configuration tasks. You access this area from the Data Quality Client home page by clicking

the Configuration button.

 

Reference Data

 

Rather than maintain domain values and rules in a knowledge base, you can subscribe to a reference

data service through Windows Azure Marketplace. Most reference data services are available as a

monthly paid subscription, but some providers offer a free trial. (Also, Digital Trowel provides a free

service to cleanse and standardize data for US public and private companies.) When you subscribe to

a service, you receive an account key that you must register in Data Quality Client before you can use

reference data in your data-quality activities.

 

 

 

The Reference Data tab is the first tab that displays in the Configuration area, as shown in

Figure

7-24. Here you type or paste your account key in the DataMarket Account ID box, and then

click the Validate DataMarket Account ID button to the right of the box. You might need to provide a

proxy server and port number if your DQS server requires a proxy server to connect to the Internet.

 

Note If you do not have a reference data service subscription, you can use the Create A

DataMarket Account ID link to open the Windows Azure Marketplace site in your browser.

You must have a Windows Live ID to access the site. Click the Data link at the top of the

page, and then click the Data Quality Services link in the Category list. You can view the

current list of reference data service providers at https://datamarket.azure.com/browse

/Data?Category=dqs.

 

 

 

FIGURE 7-24 Reference data service account configuration

 

As an alternative to using a DataMarket subscription for reference data, you can configure settings

for a third-party reference data service by clicking the Add New Reference Data Service Provider

button

and supplying the requisite details: a name for the service, a comma-delimited list of fields as

a schema, a secure URI for the reference data service, a maximum number of records per batch, and a

subscriber account identifier.

 

 

 

General Settings

 

You use the General Settings tab, shown in Figure 7-25, to configure the following settings:

 

¦ ¦ Interactive Cleansing Specify the minimum confidence score for suggestions and the

minimum confidence score for auto-corrections. DQS uses these values as thresholds when

determining how to categorize records for a cleansing data quality project.

¦ ¦ Matching Specify the minimum matching score for DQS to use for a matching policy.

¦ ¦ Profiler Use this check box to enable or disable profiling notifications. These notifications

appear in the Profiler tab when you are performing a knowledge base activity or running a

data quality project.

 

 

 

 

FIGURE 7-25 General settings to set score thresholds and enable notifications

 

Log Settings

 

Log files are useful for troubleshooting problems that might occur. By default, the DQS log files

capture

events with an Error severity level, but you can change the severity level to Fatal, Warn, Info,

or Debug by activity, as shown in Figure 7-26.

 

 

 

 

 

FIGURE 7-26 Log settings for the Data Quality Server

 

DQS generates the following three types of log files:

 

¦ ¦ Data Quality Server You can find server-related activity in the DQServerLog.DQS_MAIN.log

file in the Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Log folder.

¦ ¦ Data Quality Client You can view client-related activity in the DQClientLog.log file in the

%APPDATA%\SSDQS\Log folder.

¦ ¦ DQS Cleansing Transformation When you execute a package containing the DQS

Cleansing

transformation, DQS logs the cleansing activity in the DQSSSISLog.log file available

in the %APPDATA%\SSDQS\Log folder.

 

 

In addition to configuring log severity settings by activity, you can configure them at the module

level in the Advanced section of the Log Settings tab. By using a more granular approach to log settings,

you can get better insight into a problem you are troubleshooting. The Microsoft.Ssdqs.Core.

Startup is configured with a default severity of Info to track events related to starting and stopping

the DQS service. You can use the drop-down list in the Advanced section to select another module

and specify the log severity level you want.

 

Integration

 

DQS cleansing and matching functionality is built into two other SQL Server 2012 features—Integration

Services and Master Data Services—so that you can more effectively manage data quality

across your organization. In Integration Services, you can use DQS components to routinely perform

data cleansing in a scheduled package. In Master Data Services, you can compare external data to

master data to find matching records based on the matching policy you define for a knowledge base.

 

 

 

Integration Services

 

In earlier versions of SQL Server, you could use an Integration Services package to automate the process

of cleansing data by using Derived Column or Script transformations, but the creation of a data

flow to perform complex cleansing could be tedious. Now you can take advantage of DQS to use the

rules or reference data in a knowledge base for data cleansing. The integration between Integration

Services and DQS allows you to perform the same tasks that a cleansing data quality project supports,

but on a scheduled basis. Another advantage of using the DQS Cleansing transformation in Integration

Services is the ability to cleanse data from a source other than Excel or a SQL Server database.

 

Because the DQS functionality in Integration Services is built into the product, you can begin using

it right away without additional installation or configuration. Of course, you must have both a Data

Quality Server and a knowledge base available. To get started, you add a DQS connection manager to

the package, add a Data Flow Task, and then add a DQS Cleansing transformation to the data flow.

 

DQS Connection Manager

 

You use the DQS connection manager to establish a connection from the Integration Services

package

to a Data Quality Server. When you add the connection manager to your package, you

supply

the server name. The connection manager interface includes a button to test the connection

to the Data Quality Server. You can identify a DQS connection manager by its icon, as shown in

Figure 7-27.

 

 

 

FIGURE 7-27 DQS connection manager

 

DQS Cleansing Transformation

 

As we explain in the “Cleansing Projects” section of this chapter, DQS uses advanced algorithms

to cleanse data and calculates a confidence score to categorize records as Correct, Corrected,

Suggested,

or Invalid. The DQS Cleansing transformation in a data flow transfers data to the Data

Quality Server, which in turn executes the cleansing process and sends the data back to the transformation

with a corrected value, when applicable, and a status. You can then add a Conditional Split

transformation to the data flow to route each record to a separate destination based on its status.

 

Note If the knowledge base maps to a reference data service, the Data Quality Server

might also forward the data to the service for cleansing and enhancement.

 

Configuration of the DQS Cleansing transformation begins with the selection of a connection

manager

and a knowledge base. After you select the knowledge base, the available domains and

composite domains are displayed, as shown in Figure 7-28.

 

 

 

 

 

FIGURE 7-28 Connection manager configuration for the DQS Cleansing transformation

 

On the Mapping tab, you map each column in the data flow pipeline to its respective domain, as

shown in Figure 7-29. If you are mapping a column to a composite domain, the column must contain

the domain values as a comma-delimited string in the same order in which the individual domains

appear in the composite domain.

 

 

 

FIGURE 7-29 Mapping a pipeline column to a domain

 

 

 

For each input column, you also define the aliases for the following output columns: source,

output,

and status. The transformation editor supplies default values for you, but you can change

these aliases if you like. During package execution, the column with the source alias contains the

original value in the source, whereas the column with the output alias contains the same value for

correct and invalid records or the corrected value for suggested or corrected records. The column

with the status alias contains values to indicate the outcome of the cleansing process: Auto Suggest,

Correct, Invalid, or New.

 

On the Advanced tab of the transformation editor, you can configure the following options:

 

¦ ¦ Standardize Output This option, which is enabled by default, automatically standardizes the

data in the Output Alias column according to the Format Output settings for each domain. In

addition, this option changes synonyms to leading values if you enable Use Leading Values for

the domain.

¦ ¦ Enable Field-Level Columns You can optionally include the confidence score or the reason

for a correction as additional columns in the transformation output.

¦ ¦ Enable Record-Level Columns If a domain maps to a reference data service that returns

additional data columns during the cleansing process, you can include this data as an

appended

column. In addition, you can include a column to contain the schema for the appended

data.

 

 

Master Data Services

 

If you are using Master Data Services (MDS) for master data management, you can use the datamatching

functionality in DQS to de-duplicate master data. You must first enable DQS integration on

the Web Configuration page of the Master Data Services configuration manager, and you must create

a matching policy for a DQS knowledge base. Then you add data to an Excel worksheet and use the

MDS Add-in for Excel to combine that data with MDS-managed data in preparation for matching. The

matching process adds columns to the worksheet similar to the columns you view during a matchingdata-

quality project, including the matching score. We provide more information about how to use

the DQS matching with MDS in Chapter 8, “Master Data Services.”

 

 

 

176 PART 2 Business Intelligence Development

 

Configuration

 

Whether you perform a new installation of MDS or upgrade from a previous version, you must

use the Master Data Services Configuration Manager. You can open this tool from the Master Data

Services

folder in the Microsoft SQL Server 2012 program group on the Start menu. You use it to

create

or upgrade the MDS database, configure MDS system settings, and create a web application

for MDS.

 

Important If you are upgrading from a previous version of MDS, you must log in using the

Administrator account that was used to create the original MDS database. You can identify

this account by finding the user with an ID value of 1 in the mdm.tblUser table in the MDS

database.

 

The first configuration step to perform following installation is to configure the MDS database. On

the Database Configuration page of Master Data Services Configuration Manager, perform one of the

following tasks:

 

¦ ¦ New installation Click the Create Database button, and complete the Database Wizard.

In the wizard, you specify the SQL Server instance and authentication type and provide

credentials

having permissions to create a database on the selected instance. You also provide

a database name and specify collation. Last, you specify a Microsoft Windows account to

establish as the MDS administrator account.

 

 

Important Before you begin a new installation, you must install Internet

Information Services (IIS).

 

¦ ¦ Upgrade If you want to keep your database in a SQL Server 2008 R2 instance, click the

Repair Database button if it is enabled. Then, whether you want to store your MDS database

in SQL Server 2008 R2 or SQL Server 2012, click the Upgrade Database button. The Upgrade

Database Wizard displays the SQL Server instance and MDS database name, as well as the

progress of the update. The upgrade process re-creates tables using the new schema and

stored procedures.

 

 

Note The upgrade process excludes business rules that you use to generate values

for the code attribute. We explain the new automatic code generation in the “Entity

Management” section later in this chapter. Furthermore, the upgrade process does

not include model deployment packages. You must create new packages in your

SQL Server 2012 installation.

 

When the new or upgraded database is created, you see the System Settings display on the

Database

Configuration page. You use these settings to control timeouts for the database or the web

 

 

 

CHAPTER 8 Master Data Services 177

 

service, to name a few. If you upgraded your MDS installation, there are two new system settings that

you can configure:

 

¦ ¦ Show Add-in For Excel Text On Website Home Page This setting controls whether users

see a link to install the MDS Add-in on the MDS home page, as shown in Figure 8-1.

¦ ¦ Add-in For Excel Install Path On Website Home Page This setting defaults to the MDS

Add-in download page on the Microsoft web site.

 

 

 

 

FIGURE 8-1 Add-in for Excel text on MDS website home page

 

Master Data Manager

 

Master Data Manager is the web application that data stewards use to manage master data and

administrators

use to manage model objects and configure security. For the most part, it works as it

did in the previous version of MDS, although the Explorer and Integration Management functional

areas now use Silverlight 5. As a result, you will find certain tasks are easier and faster to perform.

In keeping with this goal of enabling easier and faster processes, you will find the current version of

MDS also introduces a new staging process and slight changes to the security model.

 

Explorer

 

In the Explorer functional area, you add or delete members for an entity, update attribute values for

members, arrange those members within a hierarchy, and optionally organize groups of members as

collections. In SQL Server 2012, MDS improves the workflow for performing these tasks.

 

Entity Management

 

When you open an entity in the Explorer area, you see a new interface, as shown in Figure 8-2. The

set of buttons now display with new icons and with descriptions that clarify their purpose. When you

click the Add Member button, you type in all attribute values for the new member in the Details pane.

To delete a member, select the member and then click the Delete button. You can also more easily

 

 

 

 

edit attribute values for a member by selecting it in the grid and then typing a new value for the

attribute

in the Details pane.

 

 

 

FIGURE 8-2 Member management for a selected entity

 

Rather than use a business rule to automatically create values for the Code attribute as you do in

SQL Server 2008 R2, you can now configure the entity to automatically generate the code value. This

automatic assignment of a value applies whether you are adding a member manually in Master Data

Manager or importing data through the staging process. To do this, you must have permission to

access the System Administration function area and open the entity. On the Entity Maintenance page,

select the Create Code Values Automatically check box, as shown in Figure 8-3. You can optionally

change the number in the Start With box. If you already have members added to the entity, MDS

increments the maximum value by one when you add a new member.

 

 

 

FIGURE 8-3 Check box to automatically generate code values

 

 

 

Note The automatic generation of a value occurs only when you leave the code value

blank. You always have the option to override it with a different value when you add a new

member.

 

Many-to-Many Mapping

 

Another improvement in the current version of MDS is the ability to use the Explorer functional area

in Master Data Manager to view entities for which you have defined many-to-many mappings. To see

how this works, consider a scenario in which you have products you want to decompose into separate

parts and you can associate any single part with multiple products. You manage the relationships

between products and parts in MDS by creating three entities: products, parts, and a mapping entity

to define the relationship between products and parts, as shown in Figure 8-4. Notice that you need

only a code value and attributes to store code values for the two related entities, but no name value.

 

 

 

FIGURE 8-4 Arrangement of members in a derived hierarchy

 

When you open the Product entity and select a member, you can open the Related Entities pane

on the right side of the screen where a link displays, such as ProductParts (Attribute: Product Code).

When you click this link, a new browser window opens to display the ProductParts entity window with

a filter applied to show records related only to the selected product. From that screen, you can click

the Go To The “<Name> Entity” To View Attribute Details button, as shown in Figure 8-5, to open yet

another browser window that displays the entity member and its attribute values.

 

 

 

FIGURE 8-5 Click through to a related entity available in the Details pane

 

 

 

Hierarchy Management

 

The new interface in the Explorer functional area, shown in Figure 8-6, makes it easier for you to

move members within a hierarchy when you want to change the parent for a member. A hierarchy

pane displays a tree view of the hierarchy. There you select the check box for each member to move.

Then you click the Copy button at the top of the hierarchy pane, select the check box of the member

to which you want to move the previously selected members, and then click the Paste button. In a

derived hierarchy, you must paste members to the same level only.

 

 

 

FIGURE 8-6 Arrangement of members in a derived hierarchy

 

Collection Management

 

You can organize a subset of entity members as a collection in MDS. A new feature is the ability to

assign a weight to each collection member within the user interface, as shown in Figure 8-7. You use

MDS only to store the weight for use by subscribing systems that use the weight to apportion values

across the collection members. Accordingly, the subscription view includes the weight column.

 

 

 

 

 

FIGURE 8-7 New collection-management interface

 

Integration Management

 

When you want to automate aspects of the master data management process, MDS now has a new,

high-performance staging process. One benefit of this new staging process is the ability to load

members and attribute values at one time, rather than separately in batches. To do this, you load data

into the following tables as applicable, where name is the name of the staging table for the entity:

 

¦ ¦ stg.name_Leaf Use this table to stage additions, updates, or deletions for leaf members and

their attributes.

¦ ¦ stg.name_Consolidated Use this table to stage additions, updates, or deletions for

consolidated

members and their attributes.

¦ ¦ stg.name_Relationship Use this table to assign members in an explicit hierarchy.

 

 

There are two ways to start the staging process after you load data into the staging tables: by using

the Integration Management functional area or executing stored procedures. In the Integration

Management functional area, you select the model in the drop-down list and click the Start Batches

button, which you can see in Figure 8-8. The data processes in batches, and you can watch the status

change from Queued To Run to Running to Completed.

 

 

 

 

 

FIGURE 8-8 Staging a batch in the Integration Management functional area

 

If the staging process produces errors, you see the number of errors appear in the grid, but

you cannot view them in the Integration Management functional area. Instead, you can click the

Copy Query button to copy a SQL query that you can paste into a query window in SQL Server

Management

Studio. The query looks similar to this:

 

SELECT * from [stg].[viw_Product_MemberErrorDetails] WHERE Batch_ID = 1

 

This view includes an ErrorDescription column describing the reason for flagging the staged record

as an error. It also includes AttributeName and AttributeValue columns to indicate which attribute and

value caused the error.

 

If you use a stored procedure to execute the staging process, you use one of the following stored

procedures, where name corresponds to the staging table:

 

¦ ¦ stg.udp_name_Leaf

¦ ¦ stg.udp_name_Consolidated

¦ ¦ stg.udp_name_Relationship

 

 

Each of these stored procedures takes the following parameters:

 

¦ ¦ VersionName Provide the version name of the model, such as VERSION_1. The collation

setting of the SQL Server database determines whether the value for this parameter is case

sensitive.

¦ ¦ LogFile Use a value of 1 to log transactions during staging or a value of 0 if you do not want

to log transactions.

¦ ¦ BatchTag Provide a string of 50 characters or less to identify the batch in the staging table.

This tag displays in the batch grid in the Integration Management functional area.

 

 

For example, to load leaf members and log transactions for the batch, execute the following code

in SQL Server Management Studio:

 

EXEC [stg].[udp_name_Leaf] @VersionName = N’VERSION_1′, @LogFlag = 1, @BatchTag = N’batch1′

GO

 

 

 

Note Transaction logging during staging is optional. You can enable logging only if you

use stored procedures for staging. Logging does not occur if you launch the staging

process

from the Integration Management functional area.

 

Important No validation occurs during the staging process. You must validate manually

in the Version Management functional area or use the mdm.udpValidateModel stored procedure.

Refer to http://msdn.microsoft.com/en-us/library/hh231023(SQL.110).aspx for more

information about this stored procedure.

 

You can continue to use the staging tables and stored procedure introduced for the SQL Server

2008 R2 MDS staging process if you like. One reason that you might choose to do this is to manage

collections, because the new staging process in SQL Server 2012 does not support collections.

Therefore,

you must use the previous staging process to create or delete collections, add members to

or remove them from collections, or reactivate members and collections.

 

User and Group Permissions

 

Just as in the previous version of MDS, you assign permissions by functional area and by model

object. When you assign Read-Only or Update permissions for a model to a user or group on the

Models tab of the Manage User page, the permission also applies to lower level objects. For example,

when you grant a user or group the Update permission to the Product model, as shown in Figure 8-9,

the users can also add, change, or delete members for any entity in the model and can change any

attribute value. You must explicitly change permissions on selected entities to Read-only when you

want to give users the ability to view but not change entity members, and to Deny when you do

not want them to see the entity members. You can further refine security by setting permissions on

attribute

objects (below the Leaf, Consolidate, or Collection nodes) to control which attribute values a

user can see or change.

 

 

 

 

 

FIGURE 8-9 Model permissions object tree and summary

 

In SQL Server 2008 MDS, the model permissions object tree also includes nodes for derived hierarchies,

explicit hierarchies, and attribute groups, but these nodes are no longer available in

SQL Server 2012 MDS. Instead, derived hierarchies inherit permissions from the model and explicit

hierarchies

inherit permissions from the associated entity. In both cases, you can override these

default permissions on the Hierarchy Members tab of the Manage User page to manage which entity

members that users can see or change.

 

For attribute group permissions, you now use the Attribute Group Maintenance page in the System

Administration functional area to assign permissions to users or groups, as shown in Figure 8-10.

Users

or groups appearing in the Assigned list have Update permissions only. You can no longer

assign

Read-Only permissions for attribute groups.

 

 

 

 

 

FIGURE 8-10 Users and Groups security for attribute groups

 

Model Deployment

 

A new high-performance, command-line tool is now available for deploying packages. If you use the

Model Deployment Wizard in the web application, it deploys only the model structure. As an alternative,

you can use the MDSModelDeploy tool to create and deploy a package with model objects only

or a package with both model objects and data. You find this tool in the Program Files\Microsoft SQL

Server\110\Master Data Services\Configuration folder.

 

Note You cannot reuse packages you created using SQL Server 2008 MDS. You can deploy

a SQL Server 2012 package only to a SQL Server 2012 MDS instance.

 

The executable for this tool uses the following syntax:

 

MDSModelDeploy <commands> [ <options> ]

 

 

 

You can use the following commands with this tool:

 

¦ ¦ listservices View a list of all service instances.

¦ ¦ listmodels View a list of all models.

¦ ¦ listversions View a list of all versions for a specified model.

¦ ¦ createpackage Create a package for a specified model.

¦ ¦ deployclone Create a duplicate of a specified model, retaining names and identifiers. The

model cannot exist in the target service instance.

¦ ¦ deploynew Create a new model. MDS creates new identifiers for all model objects.

¦ ¦ deployupdate Deploy a model, and update the model version. This option requires you to

use a package with a model having the same names and identifiers as the target model.

¦ ¦ help View usage, options, and examples of a command. For example, type the following

command to learn how to use the deploynew command: MDSModelDeploy help deploynew.

 

 

To help you learn how to work with MDS, several sample packages containing models and data

are available. To deploy the Product package to the MDS1 Web service instance, type the following

command

in the command prompt window:

 

MDSModelDeploy deploynew -package ..\Samples\Packages\product_en.pkg -model Product -service

MDS1

 

Note To execute commands by using this utility, you must have permissions to access the

System Administration functional area and you must open the command prompt window

as an administrator.

 

During deployment, MDS first creates the model objects, and then creates the business rules

and subscription views. MDS populates the model with master data as the final step. If any of these

steps fail during deployment of a new or cloned model, MDS deletes the model. If you are updating

a model, MDS retains the changes from the previous steps that completed successfully unless the

failure occurs in the final step. In that case, MDS updates master data members where possible rather

than failing the entire step and rolling back.

 

Note After you deploy a model, you must manually update user-defined metadata, file

attributes, and user and group permissions.

 

 

 

MDS Add-in for Excel

 

The most extensive addition to MDS in SQL Server 2012 is the new user-interface option for enabling

data stewards and administrators to manage master data inside Microsoft Excel. Data stewards can

retrieve data and make changes using the familiar environment of Excel after installing the MDS

Add-

in for Excel. Administrators can also use this add-in to create new model objects, such as entities,

and load data into MDS.

 

Installation of the MDS Add-in

 

By default, the home page of Master Data Manager includes a link to the download page for the

MDS Add-in on the Microsoft web site. On the download page, you choose the language and version

(

32-bit or 64-bit) that matches your Excel installation. You open the MSI file that downloads to

start the setup wizard, and then follow the prompts to accept the license agreement and confirm the

installation. When the installation completes, you can open Excel to view the new Master Data tab in

the ribbon, as shown in Figure 8-11.

 

 

 

FIGURE 8-11 Master Data tab in the Excel ribbon

 

Note The add-in works with either Excel 2007 or Excel 2010.

 

Master Data Management

 

The MDS add-in supports the primary tasks you need to perform for master data management. After

connecting to MDS, you can load data from MDS into a worksheet to use for reference or to make

additions or changes in bulk. You can also apply business rules and correct validation issues, check for

duplicates using Data Quality Services integration, and then publish the modified data back to MDS.

 

Connections

 

Before you can load MDS data into a worksheet, you must create a connection to the MDS database.

If you open a worksheet into which you previously loaded data, the MDS add-in automatically

connects

to MDS when you refresh the data or publish the data. To create a connection in the MDS

Add-in for Excel, follow these steps:

 

1. On the Master Data tab of the ribbon, click the arrow under the Connect button, and click

Manage Connections.

 

 

 

 

2. In the Manage Connections dialog box, click the Create A New Connection link.

3. In the Add New Connection dialog box, type a description for the connection. This description

displays when you click the arrow under the Connect button.

4. In the MDS Server Address box, type the URL that you use to open the Master Data Manager

web application, such as http://myserver/mds, and click the OK button. The connection

displays

in the Existing Connections section of the Manage Connections dialog box.

5. Click the Test button to test the connection, and then click the OK button to close the dialog

box that confirms the connection or displays an error.

6. Click the Connect button.

7. In the Master Data Explorer pane, select a model and version from the respective drop-down

lists, as shown here:

 

 

 

 

Data Retrieval

 

Before you load data from MDS into a worksheet, you can filter the data. Even if you do not filter the

data, there are some limitations to the volume of data you can load. Periodically, you can update the

data in the worksheet to retrieve the latest updates from MDS.

 

Filter data Rather than load all entity members from MDS into a spreadsheet, which can be a

time-consuming task, you can select attributes and apply filters to minimize the amount of data you

retrieve from MDS. You can choose to focus on selected attributes to reduce the number of columns

to retrieve. Another option is to use filter criteria to eliminate members.

 

 

 

To filter and retrieve leaf data from MDS, follow these steps:

 

1. In the Master Data Explorer pane, select the entity you want to load into the spreadsheet.

2. On the Master Data tab of the ribbon, click the Filter button.

3. In the Filter dialog box, select the columns to load by selecting an attribute type, an explicit

hierarchy (if you select the Consolidated attribute type), an attribute group, and individual

attributes.

 

 

Tip You can change the order of attributes by using the Up and Down arrows to

the right of the attribute list to move each selected attribute.

 

4. Next, select the rows to load by clicking the Add button and then selecting an attribute, a

filter operator, and filter criteria. You can repeat this step to continue adding filter criteria.

5. Click the Update Summary button to view the number of rows and columns resulting from

your filter selections, as shown here:

 

 

 

 

 

 

6. In the Filter dialog box, click the Load button. The data loads into the current spreadsheet, as

shown here:

 

 

 

 

Load data Filtering data before you load is optional. You can load all members for an entity by

clicking the Load Or Refresh button in the ribbon. A warning displays if there are more than 100,000

rows or more than 100 columns, but you can increase or decrease these values or disable the warning

by clicking the Settings button on the ribbon and changing the properties on the Data page of

the Settings dialog box. Regardless, if an entity is very large, the add-in automatically restricts the

retrieval of data to the first one million members. Also, if a column is a domain-based attribute, the

add-in retrieves only the first 1000 values.

 

Refresh data After you load MDS data into a worksheet, you can update the same worksheet

by adding columns of data from sources other than MDS or columns containing formulas. When you

want to refresh the MDS data without losing the data you added, click the Load Or Refresh button in

the ribbon.

 

The refresh process modifies the contents of the worksheet. Deleted members disappear, and new

members appear at the bottom of the table with green highlighting. Attribute values update to match

the value stored in MDS, but the cell does not change color to identify a new value.

 

Warning If you add new members or change attribute values, you must publish these

changes before refreshing the data. Otherwise, you lose your work. Cell comments on MDS

data are deleted, and non-MDS data in rows below MDS data might be replaced if the

refresh

process adds new members to the worksheet.

 

 

 

Review transactions and annotations You can review transactions for any member by

right-

clicking the member’s row and selecting View Transactions in the context menu. To view

an annotation

or to add an annotation for a transaction, select the transaction row in the View

Transactions

dialog box, which is shown in Figure 8-12.

 

 

 

FIGURE 8-12 Transactions and annotations for a member

 

Data Publication

 

If you make changes to the MDS data in the worksheet, such as altering attribute values, adding new

members, or deleting members, you can publish your changes to MDS to make it available to other

users. Each change you make saves to MDS as a transaction, which you have the option to annotate to

document the reason for the change. An exception is a deletion, which you cannot annotate although

the deletion does generate a transaction.

 

When you click the Publish button, the Publish And Annotate dialog box displays (unless you

disable

it in Settings). You can provide a single annotation for all changes or separate annotations for

each change, as shown in Figure 8-13. An annotation must be 500 characters or less.

 

Warning Cell comments on MDS data are deleted during the publication process. Also,

a change to the code value for a member does not save as a transaction and renders all

previous

transactions for that member inaccessible.

 

 

 

 

 

FIGURE 8-13 Addition of an annotation for a published data change

 

During the publication process, MDS validates your changes. First, MDS applies business rules to

the data. Second, MDS confirms the validity of attribute values, including the length and data type.

If a member passes validation, the MDS database updates with the change. Otherwise, the invalid

data displays in the worksheet with red highlighting and the description of the error appears in the

$

InputStatus$ column. You can apply business rules prior to publishing your changes by clicking the

Apply Rules button in the ribbon.

 

Model-Building Tasks

 

Use of the MDS Add-in for Excel is not limited to data stewards. If you are an administrator, you can

also use the add-in to create entities and add attributes. However, you must first create a model by

using Master Data Manager, and then you can continue adding entities to the model by using the

add-in.

 

Entities and Attributes

 

Before you add an entity to MDS, you create data in a worksheet. The data must include a header

row and at least one row of data. Each row should include at least a Name column. If you include a

Code column, the column values must be unique for each row. You can add other columns to create

attributes for the entity, but you do not need to provide values for them. If you do, you can use text,

numeric, or data values, but you cannot use formulas or time values.

 

To create a new entity in MDS, follow these steps:

 

1. Select all cells in the header and data rows to load into the new entity.

2. Click the Create Entity button in the ribbon.

3. In the Create Entity dialog box, ensure the range includes only the data you want to load and

do not clear the My Data Has Headers check box.

 

 

 

 

4. Select a model and version from the respective drop-down lists, and provide a name in the

New Entity Name box.

5. In the Code drop-down list, select the column that contains unique values for entity members

or select the Generate Code Automatically option.

6. In the Name drop-down list, select the column that contains member names, as shown next,

and then click OK. The add-in creates the new entity in the MDS database and validates the

data.

 

 

 

 

Note You might need to correct the data type or length of an attribute after creating the

entity. To do this, click any cell in the attribute’s column, and click the Attribute Properties

button in the ribbon. You can make changes as necessary in the Attribute Properties dialog

box. However, you cannot change the data type or length of the Name or Code column.

 

Domain-Based Attributes

 

If you want to restrict column values of an existing entity to a specific set of values, you can create a

domain-based attribute from values in a worksheet or an existing entity. To create a domain-based

attribute, follow these steps:

 

1. Load the entity into a worksheet, and click a cell in the column that you want to change to a

domain-based attribute.

2. Click the Attribute Properties button in the ribbon.

3. In the Attribute Properties dialog box, select Constrained List (Domain-Based) in the Attribute

Type drop-down list.

 

 

 

 

4. Select an option from the Populate The Attribute With Values From drop-down list. You

can choose The Selected Column to create a new entity based on the values in the selected

column,

as shown next, or you can choose an entity to use values from that entity.

 

 

 

 

5. Click OK. The column now allows you to select from a list of values. You can change the

available

values in the list by loading the entity on which the attribute is based into a separate

worksheet, making changes by adding new members or updating values, and then publishing

the changes back to MDS.

 

 

Shortcut Query Files

 

You can easily load frequently accessed data by using a shortcut query file. This file contains

information

about the connection to the MDS database, the model and version containing the MDS

data, the entity to load, filters to apply, and the column order. After loading MDS data into a worksheet,

you create a shortcut query file by clicking the Save Query button in the ribbon and selecting

Save As Query in the menu. When you want to use it later, you open an empty worksheet, click the

Save Query button, and select the shortcut query file from the list that displays.

 

You can also use the shortcut query file as a way to share up-to-date MDS data with other users

without emailing the worksheet. Instead, you can email the shortcut query file as long as you have

Microsoft Outlook 2010 or later installed on your computer. First, load MDS data into a worksheet,

and then click the Send Query button in the ribbon to create an email message with the shortcut

query file as an attachment. As long as the recipient of the email message has the add-in installed, he

can double-click the file to open it.

 

Data Quality Matching

 

Before you add new members to an entity using the add-in, you can prepare data in a worksheet

and combine it with MDS data for comparison. Then you use Data Quality Services (DQS) to identify

duplicates. The matching process adds detail columns to show matching scores you can use to decide

which data to publish to MDS.

 

 

 

As we explained in Chapter 7, “Data Quality Services,” you must enable DQS integration in the

Master Data Services Configuration Manager and create a matching policy in a knowledge base. In

addition, both the MDS database and the DQS_MAIN database must exist in the same SQL Server

instance.

 

The first step in the data-quality matching process is to combine data from two worksheets into a

single worksheet. The first worksheet must contain data you load from MDS. The second worksheet

must contain data with a header row and one or more detail rows. To combine data, follow these

steps:

 

1. On the first worksheet, click the Combine Data button in the ribbon.

2. In the Combine Data dialog box, click the icon next to the Range To Combine With MDS Data

text box.

3. Click the second worksheet, and highlight the header and detail rows to combine with MDS

data.

4. In the Combine Data dialog box, click the icon to the right of the Range To Combine With

MDS Data box.

5. Navigate to the second worksheet, highlight the header row and detail rows, and then click

the icon next to the range in the collapsed Combine Data dialog box.

6. In the expanded Combine Data dialog box, in the Corresponding Column drop-down list,

select

a column from the second worksheet that corresponds to the entity column that

displays

to its left, as shown here:

 

 

 

 

 

 

7. Click the Combine button. The rows from the second worksheet display in the first worksheet

below the existing rows, and the SOURCE column displays whether the row data comes from

MDS or from an external source, as shown here:

 

 

 

 

8. Click the Match Data button in the ribbon.

9. In the Match Data dialog box, select a knowledge base from the DQS Knowledge Base

drop-

down list, and map worksheet columns to each domain listed in the dialog box.

 

 

 

 

Note Rather than using a custom knowledge base as shown in the preceding

screen shot, you can use the default knowledge base, DQS Data. In that case, you

add a row to the dialog box for each column you want to use for matching and

assign

a weight value. The sum of weight values for all rows must equal 100.

 

10. Click OK, and then click the Show Details button in the ribbon to view columns containing

matching details. The SCORE column indicates the similarity between the pivot record

(

indicated by Pivot in the PIVOT_MARK column) and the matching record. You can use this

information to eliminate records from the worksheet before publishing your changes and

additions

to MDS.

 

 

 

 

 

 

Miscellaneous Changes

 

Thus far in this chapter, our focus has been on the new features available in MDS. However, there

are some more additions and changes to review. SQL Server 2012 offers some new features for

SharePoint

integration, and it retains some features from the SQL Server 2008 R2 version that are

still available, but deprecated. There are also features that are discontinued. In this section, we review

these feature changes and describe alternatives where applicable.

 

SharePoint Integration

 

There are two ways you can integrate MDS with SharePoint. First, when you add the Master Data

Manager web site to a SharePoint page, you can add &hosted=true as a query parameter to reduce

the amount of required display space. This query parameter removes the header, menu bar, and

padding

at the bottom of the page. Second, you can save shortcut query files to a SharePoint

document

library to provide lists of reference data to other users.

 

Metadata

 

The Metadata model continues to display in Master Data Manager, but it is deprecated. Microsoft

recommends that you do not use it because it will be removed in a future release of SQL Server.

You cannot create versions of the Metadata model, and users cannot view metadata in the Explorer functional

area.

 

 

Bulk Updates and Export

 

Making changes to master data one record at a time can be a tedious process. In the previous version

of MDS, you can update an attribute value for multiple members at the same time, but this capability

is no longer available in SQL Server 2012. Instead, you can use the staging process to load the

new values into the stg.name_Leaf table as we described earlier in the “Integration Management”

section of this chapter. As an alternative, you can use the MDS add-in to load the entity into an Excel

worksheet (described in the “Master Data Management” section of this chapter), update the attribute

values in bulk using copy and paste, and then publish the results to MDS.

 

The purpose of storing master data in MDS is to have access to this data for other purposes. You

use the Export To Excel button on the Member Information page when using the previous version of

MDS, but this button is not available in SQL Server 2012. When you require MDS data in Excel, you

use the MDS add-in to load entity members from MDS into a worksheet, as we described in the “Data

Retrieval” section of this chapter.

 

 

 

Transactions

 

MDS uses transactions to log every change users make to master data. In the previous version, users

review transactions in the Explorer functional area and optionally reverse their own transactions to

restore a prior value. Now only administrators can revert transactions in the Version Management

functional area.

 

MDS allows you to annotate transactions. In SQL Server 2008 R2, MDS stores annotations as

transactions

and allows you to delete them by reverting the transaction. However, annotations

are now permanent in SQL Server 2012. Although you continue to associate an annotation with a

transaction,

MDS stores the annotation separately and does not allow you to delete it.

 

Windows PowerShell

 

In the previous version of MDS, you can use PowerShell cmdlets for administration of MDS. One

cmdlet allows you to create the database, another cmdlet allows you to configure settings for MDS,

and other cmdlets allow you to retrieve information about your MDS environment. No cmdlets are

available in the current version of MDS.

 

 

 

199

 

CHAP TE R 9

 

Analysis Services and PowerPivot

 

In SQL Server 2005 and SQL Server 2008, there is only one mode of SQL Server Analysis Services

(SSAS) available. Then in SQL Server 2008 R2, VertiPaq mode debuts as the engine for PowerPivot

for SharePoint. These two server modes persist in SQL Server 2012 with some enhancements, and

now you also have the option to deploy an Analysis Services instance in tabular mode. In addition,

Microsoft SQL Server 2012 PowerPivot for Excel has several new features that extend the types of

analysis it can support.

 

Analysis Services

 

Before you deploy an Analysis Services instance, you must decide what type of functionality you

want to support and install the appropriate server mode. In this section, we compare the three

server modes, explain the various Analysis Services templates from which to choose when starting

a new Analysis Services project, and introduce the components of the new tabular mode. We

also review several new options this release provides for managing your server. Last, we discuss the

programmability

enhancements in the current release.

 

Server Modes

 

In SQL Server 2012, an Analysis Services instance can run in one of the following server modes:

multidimensional,

tabular, or PowerPivot for SharePoint. Each server mode supports a different type

of database by using different storage structures, memory architectures, and engines. Multidimensional

mode uses the Analysis Services engine you find in SQL Server 2005 and later versions. Both

tabular mode and PowerPivot for SharePoint mode use the VertiPaq engine introduced in SQL Server

2008 R2, which compresses data for storage in memory at runtime. However, tabular mode does not

have a dependency on SharePoint like PowerPivot for SharePoint does.

 

Each server mode supports a different set of data sources, tools, languages, and security features.

Table 9-1 provides a comparison of these features by server mode.

 

 

 

200 PART 2 Business Intelligence Development

 

TABLE 9-1 Server-Mode Comparison of Various Sources, Tools, Languages, and Security Features

 

Feature

 

Multidimensional

 

Tabular

 

PowerPivot for SharePoint

 

Data Sources

 

Relational database

 

Relational database

 

Analysis Services

 

Reporting Services report

 

Azure DataMarket dataset

 

Data feed

 

Excel file

 

Text file

 

Relational database

 

Analysis Services

 

Reporting Services report

 

Azure DataMarket dataset

 

Data feed

 

Excel file

 

Text file

 

Development Tool

 

SQL Server Data Tools

 

SQL Server Data Tools

 

PowerPivot for Excel

 

Management Tool

 

SQL Server Management

Studio

 

SQL Server Management

Studio

 

SharePoint Central

Administration

 

PowerPivot Configuration Tool

 

Reporting and Analysis

Tool

 

Report Builder

 

Report Designer

 

Excel PivotTable

 

PerformancePoint dashboard

 

 

Report Builder

 

Report Designer

 

Excel PivotTable

 

PerformancePoint dashboard

 

Power View

 

Report Builder

 

Report Designer

 

Excel PivotTable

 

PerformancePoint dashboard

 

Power View

 

Application

Programming Interface

 

AMO

 

AMOMD.NET

 

AMO

 

AMOMD.NET

 

No support

 

Query and Expression

Language

 

MDX for calculations and

queries

 

DMX for data-mining

queries

 

DAX for calculations and

queries

 

MDX for queries

 

DAX for calculations and queries

 

MDX for queries

 

Security

 

Cell-level security

 

Role-based permissions

in SSAS

 

Row-level security

 

Role-based permissions in

SSAS

 

File-level security using

SharePoint permissions

 

 

 

 

 

Another factor you must consider is the set of model design features that satisfy your users’

business

requirements for reporting and analysis. Table 9-2 shows the model design features that

each server mode supports.

 

TABLE 9-2 Server-Mode Comparison of Design Features

 

Model Design Feature

 

Multidimensional

 

Tabular

 

PowerPivot for

SharePoint

 

Actions

 

.

 

Aggregations

 

.

 

Calculated Measures

 

.

 

.

 

.

 

Custom Assemblies

 

.

 

Custom Rollups

 

.

 

Distinct Count

 

.

 

.

 

.

 

Drillthrough

 

.

 

.

 

 

 

 

 

 

 

CHAPTER 9 Analysis Services and PowerPivot 201

 

Model Design Feature

 

Multidimensional

 

Tabular

 

PowerPivot for

SharePoint

 

Hierarchies

 

.

 

.

 

.

 

Key Performance

Indicators

 

.

 

.

 

.

 

Linked Objects

 

.

 

.

 

(Linked tables

only)

 

Many-to-Many

Relationships

 

.

 

Parent-Child Hierarchies

 

.

 

.

 

.

 

Partitions

 

.

 

.

 

Perspectives

 

.

 

.

 

.

 

Semi-additive Measures

 

.

 

.

 

.

 

Translations

 

.

 

Writeback

 

.

 

 

 

 

 

You assign the server mode during installation of Analysis Services. On the Setup Role page of

SQL Server Setup, you select the SQL Server Feature Installation option for multidimensional or

tabular mode, or you select the SQL Server PowerPivot For SharePoint option for the PowerPivot for

SharePoint

mode. If you select the SQL Server Feature Installation option, you will specify the server

mode to install on the Analysis Services Configuration page. On that page, you must choose either

the Multidimensional And Data Mining Mode option or the Tabular Mode option. After you complete

the installation, you cannot change the server mode of an existing instance.

 

Note Multiple instances of Analysis Services can co-exist on the same server, each running

a different server mode.

 

Analysis Services Projects

 

SQL Server Data Tools (SSDT) is the model development tool for multidimensional models, data-mining

models, and tabular models. Just as you do with any business intelligence project, you open

the File menu in SSDT, point to New, and then select Project to display the New Project dialog box.

In the Installed Templates list in that dialog box, you can choose from several Analysis Services

templates,

as shown in Figure 9-1.

 

 

 

 

 

FIGURE 9-1 New Project dialog box displaying Analysis Services templates

 

There are five templates available for Analysis Services projects:

 

¦ ¦ Analysis Services Multidimensional and Data Mining Project You use this template

to develop the traditional type of project for Analysis Services, which is now known as the

multidimensional model and is the only model that includes support for the Analysis Services

data-mining features.

¦ ¦ Import from Server (Multidimensional and Data Mining) You use this template when

a multidimensional model exists on a server and you want to create a new project using the

same model design.

¦ ¦ Analysis Services Tabular Project You use this template to create the new tabular model.

You can deploy this model to an Analysis Services instance running in tabular mode only.

¦ ¦ Import from PowerPivot You use this template to import a model from a workbook

deployed

to a PowerPivot for SharePoint instance of Analysis Services. You can extend the

model using features supported in tabular modeling and then deploy this model to an

Analysis

Services instance running in tabular mode only.

¦ ¦ Import from Server (Tabular) You use this template when a tabular model exists on a

server and you want to create a new project using the same model design.

 

 

 

 

Tabular Modeling

 

A tabular model is a new type of database structure that Analysis Services supports in SQL Server

2012. When you create a tabular project, SSDT adds a Model.bim file to the project and creates

a workspace database on the Analysis Services instance that you specify. It then uses this workspace

database

as temporary storage for data while you develop the model by importing data and

designing

objects that organize, enhance, and secure the data.

 

Tip You can use the tutorial at http://msdn.microsoft.com/en-us/library

/hh231691(SQL.110).aspx to learn how to work with a tabular model project.

 

Workspace Database

 

As you work with a tabular model project in SSDT, a corresponding workspace database resides in

memory. This workspace database stores the data you add to the project using the Table Import

Wizard. Whenever you view data in the diagram view or the data view of the model designer, SSDT

retrieves the data from the workspace database.

 

When you select the Model.bim file in Solution Explorer, you can use the Properties window to

access

the following workspace database properties:

 

¦ ¦ Data Backup The default setting is Do Not Backup To Disk. You can change this to Backup

To Disk to create a backup of the workspace database as an ABF file each time you save the

Model.bim file. However, you cannot use the Backup To Disk option if you are using a remote

Analysis Services instance to host the workspace database.

¦ ¦ Workspace Database This property displays the name that Analysis Services assigns to the

workspace database. You cannot change this value.

¦ ¦ Workspace Retention Analysis Services uses this value to determine whether to keep

the workspace database in memory when you close the project in SSDT. The default option,

Unload From Memory, keeps the database on disk, but removes it from memory. For faster

loading when you next open the project, you can choose the Keep In Memory option. The

third option, Delete Workspace, deletes the workspace database from both memory and disk,

which takes the longest time to reload because Analysis Services requires additional time to

import data into the new workspace database. You can change the default for this setting if

you open the Tools menu, select Options, and open the Data Modeling page in the Analysis

Server settings.

¦ ¦ Workspace Server This property specifies the server you use to host the workspace

database.

For best performance, you should use a local instance of Analysis Services.

 

 

 

 

Note You must be an administrator for the Analysis Services instance hosting the

workspace

database.

 

Table Import Wizard

 

You use the Table Import Wizard to import data from one or more data sources. In addition to providing

connection information, such as a server name and database name for a relational data

source, you must also complete the Impersonation Information page of the Table Import Wizard.

Analysis Services uses the credentials you specify on this page to import and process data. For

credentials,

you can provide a Windows login and password or you can designate the Analysis Services

service account.

 

The next step in the Table Import Wizard is to specify how you want to retrieve the data. For

example, if you are using a relational data source, you can select from a list of tables and views or

provide a query. Regardless of the data source you use, you have the option to filter the data before

importing it into the model. One option is to eliminate an entire column by clearing the check box

in the column header. You can also eliminate rows by clicking the arrow to the right of the column

name, and clearing one or more check boxes for a text value, as shown in Figure 9-2.

 

 

 

FIGURE 9-2 Selection of rows to include during import

 

As an alternative, you can create more specific filters by using the Text Filters or Numeric Filters

options, as applicable to the column’s data type. For example, you can create a filter to import only

values equal to a specific value or values containing a designated string.

 

 

 

Note There is no limit to the number of rows you can import for a single table, although

any column in the table can have no more than 2 billion distinct values. However, the

query performance of the model is optimal when you reduce the number of rows as much

as possible.

 

Tabular Model Designer

 

After you import data into the model, the model designer displays the data in the workspace as

shown in Figure 9-3. If you decide that you need to rename columns, you can double-click on the

column name and type a new name. For example, you might add a space between words to make the

column name more user friendly. When you finish typing, press Enter to save the change.

 

 

 

FIGURE 9-3 Model with multiple tabs containing data

 

When you import data from a relational data source, the import process detects the existing

relationships

and adds them to the model. To view the relationships, switch to Diagram View (shown

in Figure 9-4), either by clicking the Diagram button in the bottom right corner of the workspace or

by opening the Model menu, pointing to Model View, and selecting Diagram View. When you point

to a line connecting two tables, the model designer highlights the related columns in each table.

 

 

 

 

 

FIGURE 9-4 Model in diagram view highlighting columns in a relationship between two tables

 

Relationships

 

You can add new relationships by clicking a column in one table and dragging the cursor to the

corresponding

column in a second table. Because the model design automatically detects the primary

table and the related lookup table, you do not need to select the tables in a specific order. If you

prefer, you can open the Table menu and click Manage Relationships to view all relationships in one

dialog box, as shown in Figure 9-5. You can use this dialog box to add a new relationship or to edit or

delete an existing relationship.

 

 

 

FIGURE 9-5 Manage Relationships dialog box displaying all relationships in the model

 

Note You can create only one-to-one or one-to-many relationships.

 

You can also create multiple relationships between two tables, but only one relationship at a time

is active. Calculations use the active relationship by default, unless you override this behavior by using

the USERELATIONSHIP() function as we explain in the “DAX” section later in this chapter.

 

 

 

Calculated Columns

 

A calculated column is a column of data you derive by using a Data Analysis Expression (DAX)

formula.

For example, you can concatenate values from two columns into a single column, as shown

in Figure 9-6. To create a calculated column, you must switch to Data View. You can either right-click

an existing column and then select Insert Column, or you can click Add Column on the Column menu.

In the formula bar, type a valid DAX formula, and press Enter. The model designer calculates and

displays column values for each row in the table.

 

 

 

FIGURE 9-6 Calculated column values and the corresponding DAX formula

 

Measures

 

Whereas the tabular model evaluates a calculated column at the row level and stores the result in

the tabular model, it evaluates a measure as an aggregate value within the context of rows, columns,

filters, and slicers for a pivot table. To add a new measure, click any cell in the calculation area, which

then displays as a grid below the table data. Then type a DAX formula in the formula bar, and press

Enter to add a new measure, as shown in Figure 9-7. You can override the default measure name, such

as Measure1, by replacing the name with a new value in the formula bar.

 

 

 

FIGURE 9-7 Calculation area displaying three measures

 

 

 

To create a measure that aggregates only row values, you click the column header and then

click the AutoSum button in the toolbar. For example, if you select Count for the ProductKey

column,

the measure grid displays the new measure with the following formula:

Count of ProductKey:=COUNTA([ProductKey]).

 

Key Performance Indicators

 

Key performance indicators (KPIs) are a special type of measure you can use to measure progress

toward a goal. You start by creating a base measure in the calculation area of a table. Then you rightclick

the measure and select Create KPI to open the Key Performance Indicator (KPI) dialog box as

shown in Figure 9-8. Next you define the measure or absolute value that represents the target value

or goal of the KPI. The status thresholds are the boundaries for each level of progress toward the goal,

and you can adjust them as needed. Analysis Services compares the base measure to the thresholds

to determine which icon to use when displaying the KPI status.

 

 

 

FIGURE 9-8 Key performance indicator definition

 

Hierarchies

 

A hierarchy is useful for analyzing data at different levels of detail using logical relationships that

allow a user to navigate from one level to the next. In Diagram View, right-click on the column you

want to set as the parent level and select Create Hierarchy, or click the Create Hierarchy button that

appears when you hover the cursor over the column header. Type a name for the hierarchy, and drag

columns to the new hierarchy, as shown in Figure 9-9. You can add only columns from the same table

to the hierarchy. If necessary, create a calculated column that uses the RELATED() function in a DAX

formula to reference a column from a related table in the hierarchy.

 

 

 

 

 

FIGURE 9-9 Hierarchy in the diagram view of a model

 

Perspectives

 

When you have many objects in a model, you can create a perspective to display a subset of the

model objects so that users can more easily find the objects they need. Select Perspectives from

the Model menu to view existing perspectives or to add a new perspective, as shown in Figure 9-10.

When you define a perspective, you select tables, columns, measures, KPIs, and hierarchies to include.

 

 

 

FIGURE 9-10 Perspective definition

 

Partitions

 

At a minimum, each table in a tabular model has one partition, but you can divide a table into

multiple

partitions when you want to manage the reprocessing of each partition separately. For

example,

you might want to reprocess a partition containing current data frequently but have no

need to reprocess a partition containing historical data. To open the Partition Manager, which you use

to create and configure partitions, open the Table menu and select Partitions. Click the Query Editor

button to view the SQL statement and append a WHERE clause, as shown in Figure 9-11.

 

 

 

 

 

FIGURE 9-11 Addition of a WHERE clause to a SQL statement for partition

 

For example, if you want to create a partition for a month and year, such as March 2004, the

WHERE clause looks like this:

 

WHERE

(([OrderDate] >= N’2004-03-01 00:00:00′) AND

([OrderDate] < N’2004-04-01 00:00:00′))

 

After you create all the partitions, you open the table in Data View, open the Model menu, and

point to Process. You then have the option to select Process Partitions to refresh the data in each

partition selectively or select Process Table to refresh the data in all partitions. After you deploy the

model to Analysis Services, you can use scripts to manage the processing of individual partitions.

 

Roles

 

The tabular model is secure by default. You must create Analysis Services database roles and assign

Windows users or groups to a role to grant users access to the model. In addition, you add one of the

following permissions to the role to authorize the actions that the role members can perform:

 

¦ ¦ None A member cannot use the model in any way.

¦ ¦ Read A member can query the data only.

 

 

 

 

¦ ¦ Read And Process A member can query the data and execute process operations. However,

the member can neither view the model database in SQL Server Management Studio (SSMS)

nor make changes to the database.

¦ ¦ Process A member can process the data only, but has no permissions to query the data or

view the model database in SSMS.

¦ ¦ Administrator A member has full permissions to query the data, execute process

operations,

view the model database in SSMS, and make changes to the model.

 

 

To create a new role, click Roles on the Model menu to open the Role Manager dialog box. Type a

name for the role, select the applicable permissions, and add members to the role. If a user belongs

to roles having different permissions, Analysis Services combines the permissions and uses the least

restrictive permissions wherever it finds a conflict. For example, if one role has None set as the

permission

and another role has Read permissions, members of the role will have Read permissions.

 

Note As an alternative, you can add roles in SSMS after deploying the model to Analysis

Services.

 

To further refine security for members of roles with Read or Read And Process permissions, you

can create row-level filters. Each row filter is a DAX expression that evaluates as TRUE or FALSE and

defines the rows in a table that a user can see. For example, you can create a filter using the expression

=Category[EnglishProductCategoryName]=”Bikes” to allow a role to view data related to Bikes

only. If you want to prevent a role from accessing any rows in a table, use the expression =FALSE().

 

Note Row filters do not work when you deploy a tabular model in DirectQuery mode. We

explain DirectQuery mode later in this chapter.

 

Analyze in Excel

 

Before you deploy the tabular model, you can test the user experience by using the Analyze In Excel

feature. When you use the Analyze In Excel feature, SSDT opens Excel (which must be installed on the

same computer), creates a data-source connection to the model workspace, and adds a pivot table to

the worksheet. When you open this item on the Model menu, the Analyze In Excel dialog box displays

and prompts you to select a user or role to provide the security context for the data-source connection.

You must also choose a perspective, either the default perspective (which includes all model

objects) or a custom perspective, as shown in Figure 9-12.

 

 

 

 

 

FIGURE 9-12 Selection of a security context and perspective to test model in Excel

 

Reporting Properties

 

If you plan to implement Power View (which we explain in Chapter 10, “Reporting Services”), you

can access a set of reporting properties in the Properties window for each table and column. At a

minimum, you can change the Hidden property for the currently selected table or column to control

whether the user sees the object in the report field list. In addition, you can change reporting properties

for a selected table or column to enhance the user experience during the development of Power

View reports.

 

If you select a table in the model designer, you can change the following report properties:

 

¦ ¦ Default Field Set Select the list of columns and measures that Power View adds to the

report

canvas when a user selects the current table in the report field list.

¦ ¦ Default Image Identify the column containing images for each row in the table.

¦ ¦ Default Label Specify the column containing the display name for each row.

¦ ¦ Keep Unique Rows Indicate whether duplicate values display as unique values or as a single

value.

¦ ¦ Row Identifier Designate the column that contains values uniquely identifying each row in

the table.

 

 

If you select a column in the model designer, you can change the following report properties:

 

¦ ¦ Default Label Indicate whether the column contains a display name for each row. You can

set this property to True for one column only in the table.

¦ ¦ Image URL Indicate whether the column contains a URL to an image on the Web or on

a SharePoint site. Power View uses this indicator to retrieve the file as an image rather than

return the URL as a text.

 

 

 

 

¦ ¦ Row Identifier Indicate whether the column contains unique identifiers for each row. You

can set this property to True for one column only in the table.

¦ ¦ Table Detail Position Set the sequence order of the current column relative to other

columns

in the default field set.

 

 

DirectQuery Mode

 

When the volume of data for your model is too large to fit into memory or when you want queries

to return the most current data, you can enable DirectQuery mode for your tabular model. In DirectQuery

mode, Analysis Services responds to client tool queries by retrieving data and aggregates

directly from the source database rather than using data stored in the in-memory cache. Although

using cache provides faster response times, the time required to refresh the cache continually with

current data might be prohibitive when you have a large volume of data. To enable DirectQuery

mode, select the Model.bim file in Solution Explorer and then open the Properties window to change

the DirectQueryMode property from Off (the default) to On.

 

After you enable DirectQuery mode for your model, some design features are no longer available.

Table 9-3 compares the availability of features in in-memory mode and DirectQuery mode.

 

TABLE 9-3 In-Memory Mode vs. DirectQuery Mode

 

Feature

 

In-Memory

 

DirectQuery

 

Data Sources

 

Relational database

 

Analysis Services

 

Reporting Services report

 

Azure DataMarket dataset

 

Data feed

 

Excel file

 

Text file

 

SQL Server 2005 or later

 

Calculations

 

Measures

 

KPIs

 

Calculated columns

 

Measures

 

KPIs

 

DAX

 

Fully functional

 

Time intelligence functions invalid

 

Some statistical functions evaluate differently

 

 

Security

 

Analysis Services roles

 

SQL Server permissions

 

Client tool support

 

SSMS

 

Power View

 

Excel

 

SSMS

 

Power View

 

 

 

 

 

Note For a more in-depth explanation of the impact of switching the tabular model to

DirectQuery mode, refer to http://msdn.microsoft.com/en-us/library

/hh230898(SQL.110).aspx.

 

 

 

Deployment

 

Before you deploy a tabular model, you configure the target Analysis Services instance, running in

tabular mode and provide a name for the model in the project properties, as shown in Figure 9-13.

You must also select one of the following query modes, which you can change later in SSMS if necessary:

 

 

¦ ¦ DirectQuery Queries will use the relational data source only.

¦ ¦ DirectQuery With In-Memory Queries will use the relational data source unless the client

uses a connection string that specifies otherwise.

¦ ¦ In-Memory Queries will use the in-memory cache only.

¦ ¦ In-Memory With DirectQuery Queries will use the cache unless the client uses a

connection

string that specifies otherwise.

 

 

Note During development of a model in DirectQuery mode, you import a small amount

of data to use as a sample. The workspace database runs in a hybrid mode that caches

data during development. However, when you deploy the model, Analysis Services uses the

Query Mode value you specify in the deployment properties.

 

 

 

FIGURE 9-13 Project properties for tabular model deployment

 

After configuring the project properties, you deploy the model by using the Build menu or by

right-clicking the project in Solution Explorer and selecting Deploy. Following deployment, you can

use SSMS to can manage partitions, configure security, and perform backup and restore operations

for the tabular model database. Users with the appropriate permissions can access the deployed

tabular model as a data source for PowerPivot workbooks or for Power View reports.

 

 

 

Multidimensional Model Storage

 

The development process for a multidimensional model follows the same steps you use in SQL Server

2005 and later versions, with one exception. The MOLAP engine now uses a new type of storage for

string data that is more scalable than it was in previous versions of Analysis Services. Specifically, the

restriction to a 4-gigabyte maximum file size no longer exists, but you must configure a dimension to

use the new storage mode.

 

To do this, open the dimension designer in SSDT and select the parent node of the dimension

in the Attributes pane. In the Properties window, in the Advanced section, change the

StringStoreCompatibilityLevel

to 1100. You can also apply this setting to the measure group for a

distinct

count measure that uses a string as the basis for the distinct count. When you execute a

Process Full command, Analysis Services loads data into the new string store. As you add more data

to the database, the string storage file continues to grow as large as necessary. However, although

the file size limitation is gone, the file can contain only 4 billion unique strings or 4 billion records,

whichever occurs first.

 

Server Management

 

SQL Server 2012 includes several features that can help you manage your server. In this release,

you can more easily gather information for performance monitoring and diagnostic purposes

by capturing

events or querying Dynamic Management Views (DMVs). Also, you can configure

server properties to support a Non-Uniform Memory Access (NUMA) architecture or more than 64 processors.

 

 

Event Tracing

 

You can now use the SQL Server Extended Events framework, which we introduced in Chapter 5,

“Programmability

and Beyond-Relational Enhancements,” to capture any Analysis Services event

as an alternative to creating traces by using SQL Server Profiler. For example, consider a scenario

in which you are troubleshooting query performance on an Analysis Services instance running in

multidimensional

mode. You can execute an XML For Analysis (XMLA) create object script to enable

tracing for specific events, such as Query Subcube, Get Data From Aggregation, Get Data From Cache,

and Query End events. There are also some multidimensional events new to SQL Server 2012: Locks

Acquired, Locks Released, Locks Waiting, Deadlock, and LockTimeOut. The event-tracing process

stores the data it captures in a file and continues storing events until you disable event tracing by

executing an XMLA delete object script.

 

There are also events available for you to monitor the other server modes: VertiPaq SE Query

Begin, VertiPaq SE Query End, Direct Query Begin, and Direct Query End events. Furthermore, the

Resource Usage event is also new and applicable to any server mode. You can use it to capture the

size of reads and writes in kilobytes and the amount of CPU usage.

 

 

 

Note More information about event tracing is available at http://msdn.microsoft.com

/en-us/library/gg492139(SQL.110).aspx.

 

XML for Analysis Schema Rowsets

 

New schema rowsets are available not only to explore metadata of a tabular model, but also to

monitor the Analysis Services server. You can query the following schema rowsets by using Dynamic

Management Views in SSMS for VertiPaq engine and tabular models:

 

¦ ¦ DISCOVER_CALC_DEPENDENCY Find dependencies between columns, measures, and

formulas.

¦ ¦ DISCOVER_CSDL_METADATA Retrieve the Conceptual Schema Definition Language (CSDL)

for a tabular model. (CSDL is explained in the upcoming “Programmability” section.)

¦ ¦ DISCOVER_XEVENT_TRACE_DEFINITION Monitor SQL Server Extended Events.

¦ ¦ DISCOVER_TRACES Use the new column, Type, to filter traces by category.

¦ ¦ MDSCHEMA_HIERARCHIES Use the new column, Structure_Type, to filter hierarchies by

Natural, Unnatural, or Unknown.

 

 

Note You can learn more about the schema rowsets at http://msdn.microsoft.com

/en-us/library/ms126221(v=sql.110).aspx and http://msdn.microsoft.com/en-us/library

/ms126062(v=sql.110).aspx.

 

Architecture Improvements

 

You can deploy tabular and multidimensional Analysis Services instances on a server with a NUMA

architecture and more than 64 processors. To do this, you must configure the instance properties to

specify the group or groups of processors that the instance uses:

 

¦ ¦ Thread pools You can assign each process, IO process, query, parsing, and VertiPag thread

pool to a separate processor group.

¦ ¦ Affinity masks You use the processor group affinity mask to indicate whether the Analysis

Services instance should include or exclude a processor in a processor group from Analysis

Services operations.

¦ ¦ Memory allocation You can specify memory ranges to assign to processor groups.

 

 

 

 

Programmability

 

The BI Semantic Model (BISM) schema in SQL Server 2012 is the successor to the Unified Dimensional

Model (UDM) schema introduced in SQL Server 2005. BISM supports both an entity approach using

tables and relationships and a multidimensional approach using hierarchies and aggregations. This

release extends Analysis Management Objects (AMOs) and XMLA to support the management of

BISM models.

 

Note For more information, refer to the “Tabular Models Developer Roadmap” at

http://technet.microsoft.com/en-us/library/gg492113(SQL.110).aspx.

 

As an alternative to the SSMS graphical interface or to custom applications or scripts that you build

with AMO or XMLA, you can now use Windows PowerShell. Using cmdlets for Analysis Services, you

can navigate objects in a model or query a model. You can also perform administrative functions

like restarting the service, configuring members for security roles, performing backup or restore

operations,

and processing cube dimensions or partitions.

 

Note You can learn more about using PowerShell with Analysis Services at

http://msdn.microsoft.com/en-us/library/hh213141(SQL.110).aspx.

 

Another new programmability feature is the addition of Conceptual Schema Definition Language

(CSDL) extensions to present the tabular model definition to a reporting client. Analysis Services sends

the model’s entity definitions in XML format in response to a request from a client. In turn, the client

uses this information to show the user the fields, aggregations, and measures that are available for

reporting and the available options for grouping, sorting, and formatting the data. The extensions

added to CSDL to support tabular models include new elements for models, new attributes and

extensions

for entities, and properties for visualization and navigation.

 

 

Note A reference to the CSDL extensions is available at http://msdn.microsoft.com/en-us

/library/hh213142(SQL.110).aspx.

 

 

 

PowerPivot for Excel

 

PowerPivot for Excel is a client application that incorporates SQL Server technology into Excel 2010

as an add-in product. The updated version of PowerPivot for Excel that is available as part of the SQL

Server 2012 release includes several minor enhancements to improve usability that users will appreciate.

Moreover, it includes several major new features to make the PowerPivot model consistent with

the structure of the tabular model.

 

Installation and Upgrade

 

Installation of the PowerPivot for Excel add-in is straightforward, but it does have two prerequisites.

You must first install Excel 2010, and then Visual Studio 2010 Tools for Office Runtime. If you were

previously using SQL Server 2008 R2 PowerPivot for Excel, you must uninstall it because there is no

upgrade option for the add-in. After completing these steps, you can install the SQL Server 2012

PowerPivot for Excel add-in.

 

Note You can download Visual Studio 2010 Tools for Office Runtime at

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=20479 and

download

the PowerPivot for Excel add-in at http://www.microsoft.com/download/en

/details.aspx?id=28150.

 

Usability

 

The usability enhancements in the new add-in make it easier to perform certain tasks during

PowerPivot

model development. The first noticeable change is in the addition of buttons to the Home

tab of the ribbon in the PowerPivot window, shown in Figure 9-14. In the View group, at the far right

of the ribbon, the Data View button and the Diagram View button allow you to toggle your view of

the model, just as you can do when working with the tabular model in SSDT. There are also buttons to

toggle the display of hidden columns and the calculation area.

 

 

 

FIGURE 9-14 Home tab on the ribbon in the PowerPivot window

 

 

 

Another new button on the Home tab is the Sort By Column button, which opens the dialog box

shown in Figure 9-15. You can now control the sorting of data in one column by the related values in

another column in the same table.

 

 

 

FIGURE 9-15 Sort By Column dialog box

 

The Design tab, shown in Figure 9-16, now includes the Freeze and Width buttons in the Columns

group to help you manage the interface as you review the data in the PowerPivot model. These

buttons

are found on the Home tab in the previous release of the PowerPivot for Excel add-in.

 

 

 

FIGURE 9-16 Design tab on the ribbon in the PowerPivot window

 

Notice also the new Mark As Date Table button on the Design tab. Open the table containing

dates, and click this button. A dialog box displays to prompt you for a column in the table that

contains

unique datetime values. Then you can create DAX expressions that use time intelligence

functions and get correct results without performing all the steps necessary in the previous version of

PowerPivot for Excel. You can select an advanced date filter as a row or column filter after you add a

date column to a pivot table, as shown in Figure 9-17.

 

 

 

 

 

FIGURE 9-17 Advanced filters available for use with date fields

 

The Advanced tab of the ribbon does not display by default. You must click the File button in the

top left corner above the ribbon and select the Switch To Advanced Mode command to display the

new tab, shown in Figure 9-18. You use this tab to add or delete perspectives, toggle the display of

implicit measures, add measures that use aggregate functions, and set reporting properties. With the

exception of Show Implicit Measures, all these tasks are similar to the corresponding tasks in tabular

model development. The Show Implicit Measures button toggles the display of measures that PowerPivot

for Excel creates automatically when you add a numeric column to a pivot table’s values in the

Excel workbook.

 

 

 

FIGURE 9-18 Advanced tab on the ribbon in the PowerPivot window

 

 

 

Not only can you add numeric columns as a pivot table value for aggregation, you can now also

add numeric columns to rows or to columns as distinct values. For example, you can place the [Sales

Amount] column both as a row and as a value to see the sum of sales for each amount, as shown in

Figure 9-19.

 

 

 

FIGURE 9-19 Use of numeric value on rows

 

In addition, you will find the following helpful enhancements in this release:

 

¦ ¦ In the PowerPivot window, you can configure the data type for a calculated column.

¦ ¦ In the PowerPivot window, you can right-click a column and select Hide From Client Tools

to prevent the user from accessing the field. The column remains in the Data View in the

PowerPivot

window with a gray background, but you can toggle its visibility in the Data View

by clicking the Show Hidden button on the Home tab of the ribbon.

¦ ¦ In the Excel window, the number format you specify for a measure persists.

¦ ¦ You can right-click a numeric value in the Excel window, and then select the Show Details

command on the context menu to open a separate worksheet listing the individual rows that

comprise the selected value. This feature does not work for calculated values other than simple

aggregates such as sum or count.

¦ ¦ In the PowerPivot window, you can add a description to a table, a measure, a KPI label, a KPI

value, a KPI status, or a KPI target. The description displays as a tooltip in the PowerPivot Field

List in the Excel window.

¦ ¦ The PowerPivot Field List now displays hierarchies at the top of the field list for each table, and

then displays all other fields in the table in alphabetical order.

 

 

Model Enhancements

 

If you have experience with PowerPivot prior to the release of SQL Server 2012 and then create your

first tabular model in SSDT, you notice that many steps in the modeling process are similar. Now as

you review the updated PowerPivot for Excel features, you should notice that the features of the

tabular model that were different from the previous version of PowerPivot for Excel are no longer

distinguishing features. Specifically, the following design features are now available in the PowerPivot

model:

 

¦ ¦ Table relationships

¦ ¦ Hierarchies

 

 

 

 

Note You can learn more about these functions at http://technet.microsoft.com/en-us

/library/ee634822(SQL.110).aspx.

 

PowerPivot for SharePoint

 

The key changes to PowerPivot for SharePoint in SQL Server 2012 provide you with a more

straightforward

installation and configuration process and a wider range of tools for managing the

server environment. These changes should help you get your server up and running quickly and keep

it running smoothly as usage increases over time.

 

Installation and Configuration

 

PowerPivot for SharePoint has many dependencies within the SharePoint Server 2010 farm that

can be challenging to install and configure correctly. Before you install PowerPivot for SharePoint,

you must install SharePoint Server 2010 and SharePoint Server 2010 Service Pack 1, although it is

not necessary

to run the SharePoint Configuration Wizard. You might find it easier to install the

PowerPivot

for SharePoint instance and then use the PowerPivot Configuration Tool to complete the

configuration of both the SharePoint farm and PowerPivot for SharePoint as one process.

 

Note For more details about the PowerPivot for SharePoint installation process, refer to

http://msdn.microsoft.com/en-us/library/ee210708(v=sql.110).aspx. Instructions for using

the PowerPivot Configuration Tool (or for using SharePoint Central Administration or

PowerShell cmdlets instead) are available at http://msdn.microsoft.com/en-us/library

/ee210609(v=sql.110).aspx.

 

Note You can now perform all configuration tasks by using PowerShell script containing

SharePoint PowerShell cmdlets and PowerPivot cmdlets. To learn more, see

http://msdn.microsoft.com/en-us/library/hh213341(SQL.110).aspx.

 

Management

 

To keep PowerPivot for SharePoint running optimally, you must frequently monitor its use of

resources

and ensure that the server can continue to support its workload. The SQL Server 2012

release extends the management tools that you have at your disposal to manage disk-space usage,

to identify potential problems with server health before users are adversely impacted, and to address

ongoing data-refresh failures.

 

 

 

Disk Space Usage

 

When you deploy a workbook to PowerPivot for SharePoint, PowerPivot for SharePoint stores

the workbook in a SharePoint content database. Then when a user later requests that workbook,

PowerPivot

for SharePoint caches it as a PowerPivot database on the server’s disk at \Program Files

\Microsoft SQL Server\MSAS11.PowerPivot\OLAP\Backup\Sandboxes\<serviceApplicationName> and

then loads the workbook into memory. When no one accesses the workbook for the specified period

of time, PowerPivot for SharePoint removes the workbook from memory, but it leaves the workbook

on disk so that it reloads into memory faster if someone requests it.

 

PowerPivot for SharePoint continues caching workbooks until it consumes all available disk space

unless you specify limits. The PowerPivot System Service runs a job on a periodic basis to remove

workbooks from cache if they have not been used recently or if a new version of the workbook exists

in the content database. In SQL Server 2012, you can configure the amount of total space that

PowerPivot for SharePoint can use for caching and how much data to delete when the total space is

used. You configure these settings at the server level by opening SharePoint Central Administration,

navigating to Application Management, selecting Manage Services On Server, and selecting SQL

Server Analysis Services. Here you can set the following two properties:

 

¦ ¦ Maximum Disk Space For Cached Files The default value of 0 instructs Analysis Services

that it can use all available disk space, but you can provide a specific value in gigabytes to

establish a maximum.

¦ ¦ Set A Last Access Time (In Hours) To Aid In Cache Reduction When the workbooks

in cache exceed the maximum disk space, Analysis Services uses this setting to determine

which workbooks to delete from cache. The default value is 4, in which case Analysis Services

removes

all workbooks that have been inactive for 4 hours or more.

 

 

As another option for managing the cache, you can go to Manage Service Applications (also in

Application

Management) and select Default PowerPivot Service Application. On the PowerPivot

Management Dashboard page, click the Configure Service Application Settings link in the Actions

section, and modify the following properties as necessary:

 

¦ ¦ Keep Inactive Database In Memory (In Hours) By default, Analysis Services keeps a

workbook

in memory for 48 hours following the last query. If a workbook is frequently

accessed

by users, Analysis Services never releases it from memory. You can decrease this

value if necessary.

¦ ¦ Keep Inactive Database In Cache (In Hours) After Analysis Services releases a workbook

from memory, the workbook persists in cache and consumes disk space for 120 hours, the

default time span, unless you reduce this value.

 

 

 

 

Server Health Rules

 

The key to managing server health is by identifying potential threats before problems occur. In this

release, you can customize server health rules to alert you to issues with resource consumption or

server availability. The current status of these rules is visible in Central Administration when you open

Monitoring and select Review Problems And Solutions.

 

To configure rules at the server level, go to Application Management, select Manage Services

On Server, and then select the SQL Server Analysis Services link. Review the default values for the

following

health rules, and change the settings as necessary:

 

¦ ¦ Insufficient CPU Resource Allocation Triggers a warning when the CPU utilization of

msmdsrv.exe remains at or over a specified percentage during the data-collection interval. The

default is 80 percent.

¦ ¦ Insufficient CPU Resources On The System Triggers a warning when the CPU usage of

the server remains at or above a specified percentage during the data collection interval. The

default is 90 percent.

¦ ¦ Insufficient Memory Threshold Triggers a memory warning when the available memory

falls below the specified value as a percentage of memory allocated to Analysis Services. The

default is 5 percent.

¦ ¦ Maximum Number of Connections Triggers a warning when the number of connections

exceeds the specified number. The default is 100, which is an arbitrary number unrelated to

the capacity of your server and requires adjustment.

¦ ¦ Insufficient Disk Space Triggers a warning when the percentage of available disk space on

the drive on which the backup folder resides falls below the specified value. The default is 5

percent.

¦ ¦ Data Collection Interval Defines the period of time during which calculations for serverlevel

health rules apply. The default is 4 hours.

 

 

To configure rules at the service-application level, go to Application Management, select Manage

Service Applications, and select Default PowerPivot Service Application. Next, select Configure Service

Application Settings in the Actions list. You can then review and adjust the values for the following

health rules:

 

¦ ¦ Load To Connection Ratio Triggers a warning when the number of load events relative to

the number of connection events exceeds the specified value. The default is 20 percent. When

this number is too high, the server might be unloading databases too quickly from memory or

from the cache.

 

 

 

 

¦ ¦ Data Collection Interval Defines the period of time during which calculations for service

application-level health rules apply. The default is 4 hours.

¦ ¦ Check For Updates To PowerPivot Management Dashboard.xlsx Triggers a warning

when the PowerPivot Management Dashboard.xlsx file fails to change during the specified

number of days. The default is 5. Under normal conditions, the PowerPivot Management

Dashboard.xlsx file refreshes daily.

 

 

Data Refresh Configuration

 

Because data-refresh operations consume server resources, you should allow data refresh to occur

only for active workbooks and when the data refresh consistently completes successfully. You now

have the option to configure the service application to deactivate the data-refresh schedule for a

workbook if either of these conditions is no longer true. To configure the data-refresh options, go to

Application Management, select Manage Service Applications, and select Default PowerPivot Service

Application. Next, select Configure Service Application Settings in the Actions list and adjust the

following

settings as necessary:

 

¦ ¦ Disable Data Refresh Due To Consecutive Failures If the data refresh fails a consecutive

number of times, the PowerPivot service application deactivates the data-refresh schedule for

a workbook. The default is 10, but you can set the value to 0 to prevent deactivation.

¦ ¦ Disable Data Refresh For Inactive Workbooks If no one queries a workbook during the

time required to execute the specified number of data-refresh cycles, the PowerPivot service

application deactivates the workbook. The default is 10, but you can set the value to 0 if you

prefer to keep the data-refresh operation active.

 

 

 

 

229

 

CHAP TE R 10

 

Reporting Services

 

Each release of Reporting Services since its introduction in Microsoft SQL Server 2000 has expanded

its feature base to improve your options for sharing reports, visualizing data, and empowering

users with self-service options. SQL Server 2012 Reporting Services is no exception, although almost

all the improvements affect only SharePoint integrated mode. The exception is the two new renderers

available in both native mode and SharePoint integrated mode. Reporting Services in SharePoint

integrated

mode has a completely new architecture, which you now configure as a SharePoint

shared service application. For expanded data visualization and self-service capabilities in SharePoint

integrated mode, you can use the new ad reporting tool, Power View. Another self-service feature

available only in SharePoint integrated mode is data alerts, which allows you to receive an email when

report data meets conditions you specify. If you have yet to try SharePoint integrated mode, these

features will surely entice you to begin!

 

New Renderers

 

Although rendering a report as a Microsoft Excel workbook has always been available in Reporting

Services, the ability to render a report as a Microsoft Word document has been possible only since

SQL Server 2008. Regardless, these renderers produce XLS and DOC file formats respectively, which

allows compatibility with Excel 2003 and Word 2003. Users of Excel 2010 and Word 2010 can open

these older file formats, of course, but they can now enjoy some additional benefits by using the new

renderers.

 

Excel 2010 Renderer

 

By default, the Excel rendering option now produces an XLSX file in Open Office XML format, which

you can open in either Excel 2007 or Excel 2010 if you have the client installed on your computer.

The benefit of the new file type is the higher number of maximum rows and columns per worksheet

that the later versions of Excel support—1,048,576 rows and 16,384 columns. You can also export

reports with a wider range of colors as well, because the XLSX format supports 16 million colors in the

24-bit color spectrum. Last, the new renderer uses compression to produce a smaller file size for the

exported report.

 

 

 

230 PART 2 Business Intelligence Development

 

Tip If you have only Excel 2003 installed, you can open this file type if you install the

Microsoft Office Compatibility Pack for Word, Excel, and PowerPoint, which you can download

at http://office.microsoft.com/en-us/products/microsoft-office-compatibilitypack-

for-word-excel-and-powerpoint-HA010168676.aspx. As an alternate solution, you can

enable the Excel 2003 renderer in the RsReportSErver.config and RsReportDesigner.config

files by following the instructions at http://msdn.microsoft.com/en-us/library

/dd255234(SQL.110).aspx#AvailabilityExcel.

 

Word 2010 Renderer

 

Although the ability to render a report as a DOCX file in Open Office XML format does not offer as

many benefits as the new Excel renderer, the new Word render does use compression to generate a

smaller file than the Word 2003 renderer. You can also create reports that use new features in Word

2007 or Word 2010.

 

Tip Just as with the Excel renderer, you can use the new Word renderer when you have

Word 2003 on your computer if you install the Microsoft Office Compatibility Pack for

Word, Excel, and PowerPoint, available for download at http://office.microsoft.com/en-us

/products/microsoft-office-compatibility-pack-for-word-excel-and-powerpoint-

HA010168676.aspx. If you prefer, you can enable the Word 2003 renderer in the

RsReportSErver.config and RsReportDesigner.config files by following the instructions at

http://msdn.microsoft.com/en-us/library/dd283105(SQL.110).aspx#AvailabilityWord.

 

SharePoint Shared Service Architecture

 

In previous version of Reporting Services, installing and configuring both Reporting Services and

SharePoint components required many steps and different tools to complete the task because

SharePoint

integrated mode was depending on features from two separate services. For the current

release, the development team has completely redesigned the architecture for better performance,

scalability, and administration. With the improved architecture, you also experience an easier

configuration

process.

 

Feature Support by SharePoint Edition

 

General Reporting Services features are supported in all editions of SharePoint. That is, all features

that were available in previous versions of Reporting Services continue to be available in all editions.

However, the new features in this release are available only in SharePoint Enterprise Edition.

Table 10-1 shows the supported features by SharePoint edition.

 

 

 

CHAPTER 10 Reporting Services 231

 

TABLE 10-1 SharePoint Edition Feature Support

 

Reporting Services

Feature

 

SharePoint Foundation

2010

 

SharePoint Server 2010

Standard Edition

 

SharePoint Server 2010

Enterprise Edition

 

General report viewing and

subscriptions

 

.

 

.

 

.

 

Data Alerts

 

.

 

Power View

 

.

 

 

 

 

 

Shared Service Architecture Benefits

 

With Reporting Services available as a shared service application, you can experience the following

new benefits:

 

¦ ¦ Scale Reporting Services across web applications and across your SharePoint Server 2010

farms with fewer resources than possible in previous versions.

¦ ¦ Use claims-based authentication to control access to Reporting Services reports.

¦ ¦ Rely on SharePoint backup and recovery processes for Reporting Services content.

 

 

Service Application Configuration

 

After installing the Reporting Services components, you must create and then configure the service

application. You no longer configure the Reporting Services settings by using the Reporting Services

Configuration Manager. Instead, you use the graphical interface in SharePoint Central Administration

or use SharePoint PowerShell cmdlets.

 

When you create the service application, you specify the application pool identity under which

Reporting Services runs. Because the SharePoint Shared Service Application pool now hosts Reporting

Services, you no longer see a Windows service for a SharePoint integrated-mode report server in the

service management console. You also create three report server databases—one for storing server

and catalog data; another for storing cached data sets, cached reports, and other temporary data;

and a third one for data-alert management. The database names include a unique identifier for the

service application, enabling you to create multiple service applications for Reporting Services in the

same SharePoint farm.

 

As you might expect, most configuration settings for the Reporting Services correspond to settings

you find in the Reporting Services Configuration Manager or the server properties you set in

SQL Server Management Studio when working with a native-mode report server. However, before

you can use subscriptions or data alerts, you must configure SQL Server Agent permissions correctly.

One way to do this is to open SharePoint Central Administration, navigate to Application Management,

access Manage Service Applications, click the link for the Reporting Services service application,

and then open the Provision Subscriptions And Alerts page. On that page, you can provide the

credentials if your SharePoint administrator credentials have db_owner permissions on the Reporting

Services databases.

If you prefer, you can download a Transact-SQL script from the same page, or run

 

 

 

a PowerShell cmdlet to build the same Transact-SQL script, that you can later execute in SQL Server

Management Studio.

 

Whether you use the interface or the script, the provisioning process creates the RSExec role if

necessary, creates a login for the application pool identity, and assigns it to the RSExec role in each

of the three report server databases. If any scheduled jobs, such as subscriptions or alerts, exist in the

report server database, the script assigns the application pool identity as the owner of those jobs. In

addition, the script assigns the login to the SQLAgentUserRole and grants it the necessary permissions

for this login to interact with SQL Server Agent and to administer jobs.

 

Note For detailed information about installing and configuring Reporting Services in

SharePoint integrated mode, refer to http://msdn.microsoft.com/en-us/library

/cc281311(SQL.110).aspx.

 

Power View

 

Power View is the latest self-service feature available in Reporting Services. It is a browser-based

Silverlight application that requires Reporting Services to run in SharePoint integrated mode using

SharePoint Server 2010 Enterprise Edition. It also requires a specific type of data source—either a

tabular model that you deploy to an Analysis Services server or a PowerPivot workbook that you

deploy to a SharePoint document library.

 

Rather than working in design mode and then previewing the report, as you do when using Report

Designer or Report Builder, you work directly with the data in the presentation layout of Power View.

You start with a tabular view of the data that you can change into various data visualizations. As you

explore and examine the data, you can fine-tune and adjust the layout by modifying the sort order,

adding more views to the report, highlighting values, and applying filters. When you finish, you can

save the report in the new RDLX file format to a SharePoint document library or PowerPivot Gallery or

export the report to Microsoft PowerPoint to make it available to others.

 

Note To use Power View, you can install Reporting Services in SharePoint integrated

mode using any of the following editions: Evaluation, Developer, Business Intelligence,

or Enterprise. The browser you use depends on your computer’s operating system. You

can use Internet Explorer 8 and later when using a Windows operating system or Safari 5

when using a Mac operating system. Windows Vista and Windows Server 2008 both support

Internet Explorer 7. Windows Vista, Windows 7, and Windows Server 2008 support

Firefox 7.

 

 

 

Data Sources

 

Just as for any report you create using the Report Designer in SQL Server Data Tools or using Report

Builder, you must have a data source available for use with your Power View report. You can use any

of the following data source types:

 

¦ ¦ PowerPivot workbook Select a PowerPivot workbook directly from the PowerPivot Gallery

as a source for your Power View report.

¦ ¦ Shared data source Create a Reporting Services Shared Data Source (RSDS) file with

the Data Source Type property set to Microsoft BI Semantic Model For Power View. You

then define

a connection string that references a PowerPivot workbook (such as

http://SharePointServer/PowerPivot Gallery/myWorkbook.xlsx) or an Analysis Services tabular

model (such as Data Source=MyAnalysisServer; Initial Catalog=MyTabularModel). (We introduced

tabular models in Chapter 9, “Analysis Services and PowerPivot.”). You use this type of

shared data source only for the creation of Power View reports.

 

 

Note If your web application uses claims forms-based authentication, you must

configure the shared data source to use stored credentials and specify a Windows

login for the stored credentials. If the web application uses Windows classic or

Windows claims authentication and your RSDS file references an Analysis Services

outside the SharePoint farm, you must configure Kerberos authentication

and use

integrated security.

 

¦ ¦ Business Intelligence Semantic Model (BISM) connection file Create a BISM file to

connect

to either a PowerPivot workbook or to an Analysis Services tabular model. You can

use this file as a data source for both Power View reports and Excel workbooks.

 

 

Note If your web application uses Windows classic or Windows claims authentication

and your BISM file references an Analysis Services outside the SharePoint farm, you must

configure

Kerberos authentication and use integrated security.

 

BI Semantic Model Connection Files

 

Whether you want to connect to an Analysis Services tabular database or to a PowerPivot

workbook deployed to a SharePoint server, you can use a BISM connection file as a data source

for your Power View reports and even for Excel workbooks. To do this, you must first add the BI

Semantic Model Connection content type to a document library in a PowerPivot site.

 

To create a BISM file for a PowerPivot workbook, you open the document library in which

you want to store the file and for which you have Contribute permission. Click New Document

on the Documents tab of the SharePoint ribbon, and select BI Semantic Model Connection on

 

 

 

the menu. You then configure the properties of the connection according to the data source, as

follows:

 

¦ ¦ PowerPivot workbook Provide the name of the file, and set the Workbook URL

Or Server Name property as the SharePoint URL for the workbook, such as

http://SharePointServer/PowerPivot Gallery/myWorkbook.xlsx.

¦ ¦ Analysis Services tabular database Provide the name of the file; set the Workbook

URL Or Server Name property using the server name, the fully qualified domain name, the

Internet Protocol (IP) address, or the server and instance name (such as MyAnalysisServer

\MyInstance); and set the Database property using the name of the tabular model on the

server.

 

 

You must grant users the Read permission on the file to enable them to open the file and use

it as a data source. If the data source is a PowerPivot workbook, users must also have Read permission

on the workbook. If the data source is a tabular model, the shared service requesting

the tabular model data must have administrator permission on the tabular instance and users

must be in a role with Read permission for the tabular model.

 

Power View Design Environment

 

You can open the Power View design environment for a new report in one of the following ways:

 

¦ ¦ From a PowerPivot workbook Click the Create Power View Report icon in the upperright

corner of a PowerPivot workbook that displays in the PowerPivot Gallery, as shown in Figure

10-1.

¦ ¦ From a data connection library Click the down arrow next to the BISM or RSDS file, and

then click Create Power View Report.

 

 

 

 

FIGURE 10-1 Create Power View Report icon in PowerPivot Gallery

 

In the Power View design environment that displays when you first create a report, a blank view

workspace appears in the center, as shown in Figure 10-2. As you develop the report, you add tables

and data visualizations to the view workspace. The view size is a fixed height and width, just like the

slide size in PowerPoint. If you need more space to display data, you can add more views to the report

and then navigate between views by using the Views pane.

 

Above the view workspace is a ribbon that initially displays only the Home tab. As you add fields

to the report, the Design and Layout tabs appear. The contents of each tab on the ribbon change

dynamically to show only the buttons and menus applicable to the currently selected item.

 

 

 

View Workspace Dynamic Ribbon Field List

Views Pane Layout Section

 

FIGURE 10-2 Power View design environment

 

The field list from the tabular model appears in the upper-right corner. When you open the design

environment for a new report, you see only the table names of the model. You can expand a table

name to see the fields it contains, as shown in Figure 10-3. Individual fields, such as Category, display

in the field list without an icon. You also see row label fields with a gray-and-white icon, such as

Drawing. The row label fields identify a column configured with report properties in the model, as we

explain in Chapter 9. Calculated columns, such as Attendees, display with a sigma icon, and measures,

such as Quantity Served YTD, appear with a calculator icon.

 

You select the check box next to a field to include that field in your report, or you can drag the

field into the report. You can also double-click a table name in the field list to add the default field

set, as defined in the model, to the current view.

 

Tip The examples in this chapter use the sample files available for download at

http://www.microsoft.com/download/en/details.aspx?id=26718 and

http://www.microsoft.com/download/en/details.aspx?id=26719. You can use these files

with the tutorial at http://social.technet.microsoft.com/wiki/contents/articles/6175.aspx for

a hands-on experience with Power View.

 

 

 

Fields

Row Label Fields

Calculated Columns

Measure

 

FIGURE 10-3 Tabular model field list

 

You click a field to add it to the view as a single column table, as shown in Figure 10-4. You must

select the table before you select additional fields, or else selection of the new field starts a new table.

As you add fields, Power View formats the field according to data type.

 

 

 

FIGURE 10-4 Single-column table

 

Tip Power View retrieves only the data it can display in the current view and retrieves

additional

rows as the user scrolls to see more data. That way, Power View can optimize

performance even when the source table contains millions of rows.

 

The field you select appears below the field list in the layout section. The contents of the layout

section change according to the current visualization. You can use the layout section to rearrange the

sequence of fields or to change the behavior of a field, such as changing which aggregation function

to use or whether to display rows with no data.

 

 

 

After you add fields to a table, you can move it and resize it as needed. To move an item in the

view, point to its border and, when the cursor changes to a hand, drag it to the desired location. To

resize it, point again to the border, and when the double-headed arrow appears, drag the border to

make the item smaller or larger.

 

To share a report when you complete the design, you save it by using the File menu. You can save

the file only if you have the Add Items permission on the destination folder or the Edit Items permission

to overwrite an existing file. When you save a file, if you keep the default option set to Save

Preview Images With Report, the views of your report display in the PowerPivot gallery. You should

disable this option if the information in your report is confidential. The report saves as an RDLX file,

which is not compatible with the other Reporting Services design environments.

 

Data Visualization

 

A table is only one way to explore data in Power View. After you add several fields to a table, you can

then convert it to a matrix, a chart, or a card. If you convert it to a scatter chart, you can add a play

axis to visualize changes in the data over multiple time periods. You can also create multiples of the

same chart to break down its data by different categorizations.

 

Charts

 

When you add a measure to a table, the Design tab includes icons for chart visualizations, such as

column,

bar, line, or scatter charts. After you select the icon for a chart type, the chart replaces the

table. You can then resize the visualization to improve legibility, as shown in Figure 10-5. You can continue

to add more fields to a visualization or remove fields by using the respective check boxes in the

field list. On the Layout tab of the ribbon, you can use buttons to add a chart title, position the legend

when your chart contains multiple series, or enable data labels.

 

 

 

FIGURE 10-5 Column chart

 

Tip You can use an existing visualization as the starting point for a new visualization by

selecting it and then using the Copy and Paste buttons on the Home tab of the ribbon. You

can paste the visualization to the same view or to a new view. However, you cannot use

shortcut keys to perform the copy and paste.

 

 

 

Arrangement

 

You can overlap and inset items, as shown in Figure 10-6. You use the Arrange buttons on the

Home tab of the ribbon to bring an item forward or send it back. When you want to view an item

in isolation,

you can click the button in its upper-right corner to fill the entire view with the selected

visualization.

 

 

 

FIGURE 10-6 Overlapping visualizations

 

Cards and Tiles

 

Another type of visualization you can use is cards, which is a scrollable list of grouped fields arranged

in a card format, as shown in Figure 10-7. Notice that the default label and default image fields are

more prominent than the other fields. The size of the card changes dynamically as you add or remove

fields until you resize using the handles, after which the size remains fixed. You can double-click the

sizing handle on the border of the card container to revert to auto-sizing.

 

 

 

FIGURE 10-7 Card visualization

 

 

 

You can change the sequence of fields by rearranging them in the layout section. You can also

move a field to the Tile By area to add a container above the cards that displays one tile for each

value in the selected field, as shown in Figure 10-8. When you select a tile, Power View filters the

collection

of cards to display those having the same value as the selected tile.

 

 

 

FIGURE 10-8 Tiles

 

The default layout of the tiles is tab-strip mode. You can toggle between tab-strip mode and

cover-flow mode by using the respective button in the Tile Visualizations group on the Design tab

of the ribbon. In cover-flow mode, the label or image appears below the container and the current

selection appears in the center of the strip and slightly larger than the other tile items.

 

Tip You can also convert a table or matrix directly to a tile container by using the Tiles

button on the Design tab of the ribbon. Depending on the model design, Power View

displays

the value of the first field in the table, the first row group value, or the default

field set. All other fields that were in the table or matrix display as a table inside the tile container.

 

 

Play Axis

 

When your table or chart contains two measures, you can convert it to a scatter chart to show one

measure on the horizontal axis and the second measure on the vertical axis. Another option is to use

a third measure to represent size in a bubble chart that you define in the layout section, as shown

in Figure 10-9. With either chart type, you can also add a field to the Color section for grouping

purposes.

Yet one more option is to add a field from a date table to the Play Axis area of the layout

section.

 

 

 

 

 

 

FIGURE 10-9 Layout section for a bubble chart

 

With a play axis in place, you can click the play button to display the visualization in sequence for

each time period that appears on the play axis or you can use the slider on the play axis to select a

specific time period to display. A watermark appears in the background of the visualization to indicate

the current time period. When you click a bubble or point in the chart, you filter the visualization to

focus on the selection and see the path that the selection follows over time, as shown in Figure 10-10.

You can also point to a bubble in the path to see its values display as a tooltip.

 

 

 

FIGURE 10-10 Bubble chart with a play axis

 

 

 

Multiples

 

Another way to view data is to break a chart into multiple copies of the same chart. You place the

field for which you want to create separate charts in the Vertical Multiples or Horizontal Multiples

area of the layout section. Then you use the Grid button on the Layout tab of the ribbon to select

the number of tiles across and down that you want to include, such as three tiles across and two tiles

down, as shown in Figure 10-11. If the visualization contains more tiles than the grid can show, a scroll

bar appears to allow you to access the other tiles. Power View aligns the horizontal and vertical axes

in the charts to facilitate comparisons between charts.

 

 

 

FIGURE 10-11 Vertical multiples of a line chart

 

Sort Order

 

Above some types of visualizations, the sort field and direction display. To change the sort field, click

on it to view a list of available fields, as shown in Figure 10-12. You can also click the abbreviated

direction label to reverse the sort. That is, click Asc to change an ascending sort to a descending sort

which then displays as Desc in the sort label.

 

 

 

FIGURE 10-12 Sort field selection

 

Multiple Views

 

You might find it easier to explore your data when you arrange visualizations as separate views. All

views in your report must use the same tabular model as their source, but otherwise each view can

contain separate visualizations. Furthermore, the filters you define for a view, as we explain later in

this chapter, apply only to that view. Use the New View button on the Home tab of the toolbar to

create

a new blank view or to duplicate the currently selected view, as shown in Figure 10-13.

 

 

 

 

 

FIGURE 10-13 Addition of a view to a Power View report

 

Highlighted Values

 

To help you better see relationships, you can select a value in your chart, such as a column or a

legend

item. Power View then highlights related values in the chart. You can even select a value in

one chart, such as Breads, to highlight values in other charts in the same view that are related to

Breads, as shown in Figure 10-14. You clear the highlighting by clicking in the chart area without

clicking

another bar.

 

 

 

FIGURE 10-14 Highlighted values

 

 

 

Filters

 

Power View provides you with several ways to filter the data in your report. You can use a slicer or

tile container to a view to incorporate the filter selection into the body of the report. As an alternative,

you can add a view filter or a visualization filter to the Filter Area of the design environment for

greater flexibility in the value selections for your filters.

 

Slicer

 

When you create a single-column table in a view, you can click the Slicer button on the Design tab

of the ribbon to use the values in the table as a filter. When you select one or more labels in a slicer,

Power View not only filters all visualizations in the view, but also other slicers in the view, as shown in

Figure 10-15. You can restore the unfiltered view by clicking the Clear Filter icon in the upper-right

corner of the slicer.

 

 

 

FIGURE 10-15 Slicers in a view

 

Tile Container

 

If you add a visualization inside a tile container, as shown in Figure 10-16, the tile selection filters

both the cards as well as the visualizations. However, the filtering behavior is limited only to the tile

selection.

Other visualizations or slicers in the same view remain unchanged.

 

 

 

FIGURE 10-16 Tile container to filter a visualization

 

 

 

When you add a visualization to a tile container, Power View does not synchronize the horizontal

and vertical axes for the visualizations as it does with multiples. In other words, when you select

one tile, the range of values for the vertical axis might be greater than the range of values on the

same axis for a different tile. You can use the buttons in the Synchronize group of the Layout tab to

synchronize

axes, series, or bubbles across the tiles to more easily compare the values in the chart

from one tile to another.

 

View Filter

 

A view filter allows you to define filter criteria for the current view without requiring you to include

it in the body of the report like a slicer. Each view in your report can have its own set of filters. If you

duplicate a view containing filters, the new view contains a copy of the first view’s filters. That is,

changing filter values for one view has no effect on another view, even if the filters are identical.

 

You define a view filter in the Filters area of the view workspace, which by default is not visible

when you create a new report. To toggle the visibility of this part of the workspace, click the Filters

Area button on the Home tab of the ribbon. After making the Filters area visible, you can collapse it

when you need to increase the size of your view by clicking the left arrow that appears at the top of

the Filters area.

 

To add a new filter in the default basic mode, drag a field to the Filters area and then select a

value. If the field contains string or date data types, you select one or more values by using a check

box. If the field contains a numeric data type, you use a slider to set the range of values. You can see

an example of filters for fields with numeric and string data types in Figure 10-17.

 

 

 

FIGURE 10-17 Filter selection of numeric and string data types in basic filter mode

 

To use more flexible filter criteria, you can switch to advanced filter mode by clicking the first icon

in the toolbar that appears to the right of the field name in the Filters area. The data type of the

field determines the conditions you can configure. For a filter with a string data type, you can create

conditions to filter on partial words using operators such as Contains, Starts With, and so on. With a

numeric data type, you can use an operator such as Less Than or Greater Than Or Equal To, among

others. If you create a filter based on a date data type, you can use a calendar control in combination

with operators such as Is Before or Is On Or After, and others, as shown in Figure 10-18. You can also

create compound conditions by using AND and OR operators.

 

 

 

 

 

FIGURE 10-18 Filter selection of date data types in advanced filter mode

 

Visualization Filter

 

You can also use the Filters area to configure the filter for a selected visualization. First, you must click

the Filter button in the top-right corner of the visualization, and then fields in the visualization display

in the Filters area, as shown in Figure 10-19. When you change values for fields in a visualization’s

filter, Power View updates only that visualization. All other visualizations in the same view are unaffected

by the filter. Just as you can with a view filter, you can configure values for a visualization filter

in basic filter mode or advanced filter mode.

 

 

 

FIGURE 10-19 Visualization filter

 

Display Modes

 

After you develop the views and filters for your report, you will likely spend more time navigating

between views and interacting with the visualizations than editing your report. To provide alternatives

for your viewing experience, Power View offers the following three display modes, which you enable

by using the respective button in the Display group of the Home tab of the ribbon:

 

¦ ¦ Fit To Window The view shrinks or expands to fill the available space in the window. When

you use this mode, you have access to the ribbon, field list, Filters area, and Views pane. You

can switch easily between editing and viewing activities.

¦ ¦ Reading Mode In this mode, Power View keeps your browser’s tabs and buttons visible, but

it hides the ribbon and the field list, which prevents you from editing your report. You can

navigate between views using the arrow keys on your keyboard, the multiview button in the

lower right corner of the screen that allows you to access thumbnails of the views, or the arrow

buttons in the lower right corner of the screen. Above the view, you can click the Edit Report

button to switch to Fit To Window mode or click the other button to switch to Full Screen

mode.

 

 

 

 

¦ ¦ Full Screen Mode Using Full Screen mode is similar to using the slideshow view in

PowerPoint.

Power View uses your entire screen to display the current view. This mode is

similar to Reading mode and provides the same navigation options, but instead it hides your

browser and uses the entire screen.

 

 

You can print the current view when you use Fit To Window or Reading mode only. To print, open

the File menu and click Print. The view always prints in landscape orientation and prints only the

data you currently see on your screen. In addition, the Filters area prints only if you expand it first. If

you have a play axis on a scatter or bubble chart, the printed view includes only the current frame.

Likewise,

if you select a tile in a tile container before printing, you see only the selected tile in the

printed view.

 

PowerPoint Export

 

A very useful Power View feature is the ability to export your report to PowerPoint. On the File menu,

click Export To PowerPoint and save the PPTX file. Each view becomes a separate slide in the PowerPoint

file. When you edit each slide, you see only a static image of the view. However, as long as

you have the correct permissions and an active connection to your report on the SharePoint server

when you display a slide in Reading View or Slideshow modes, you can click the Click To Interact

link in the lower-right corner of the slide to load the view from Power View and enable interactivity.

You can change filter values in the Filters area, in slicers, and in tile containers, and you can highlight

values. However, you cannot create new filters or new data visualizations from PowerPoint. Also, if

you navigate

to a different slide and then return to the Power View slide, you must use the Click To

Interact link to reload the view for interactivity.

 

Data Alerts

 

Rather than create a subscription to email a report on a periodic basis, regardless of the data values

that it contains, you can create a data alert to email a notification only when specific conditions in

the data are true at a scheduled time. This new self-service feature is available only with Reporting

Services

running in SharePoint integrated mode and works as soon as you have provisioned subscriptions

and alerts, as we described earlier in the “Service Application Configuration” section of this

chapter. However, it works only with reports you create by using Report Designer or Report Builder.

You cannot create data alerts for Power View reports.

 

Data Alert Designer

 

You can create one or more data alerts for any report you can access, as long as the data store uses a

data source with stored credentials or no credentials. The report must also include at least one data

region. In addition, the report must successfully return data at the time you create the new alert. If

these prerequisites are met, then after you open the report for viewing, you can select New Data

Alert from the Actions menu in the report toolbar. The Data Alert Designer, as shown in Figure 10-20,

displays.

 

 

 

 

 

FIGURE 10-20 Data Alert Designer

 

Tip You must have the SharePoint Create Alert permission to create an alert for any report

for which you have permission to view.

 

You use the Data Alert Designer to define rules for one or more data regions in the report that

control whether Reporting Services sends an alert. You also specify the recurring schedule for the

process that evaluates the rules and configure the email settings for Reporting Services to use when

generating the email notifications. When you save the resulting alert definition, Reporting Services

saves it in the alerting database and schedules a corresponding SQL Server Agent job, as shown in

Figure 10-21.

 

Data Alert

Designer

SQL Server

Agent Job

Alerting

Save DB

data alert

Create

data alert

Run

report

 

FIGURE 10-21 Data alert creation process

 

In the Data Alert Designer, you select a data region in the report to view a preview of the first 100

rows of data for reference as you develop the rules for that data region, which is also known as a data

feed. You can create a simple rule that sends a data alert if the data feed contains data when the SQL

 

 

 

Server Agent job runs. More commonly, you create one or more rules that compare a field value to

value that you enter or to a value in another field. The Data Alert Designer combines multiple rules

for the same data feed by using a logical AND operator only. You cannot change it to an OR operator.

Instead, you must create a separate data alert with the additional condition.

 

In the Schedule Settings section of the Data Alert Designer, you can configure the daily, weekly,

hourly, or minute intervals at which to run the SQL Server Agent job for the data alert. In the

Advanced

settings, shown in Figure 10-22, you can set the date and time at which you want to start

the job, and optionally set an end date. You also have the option to send the alert only when the alert

results change.

 

 

 

FIGURE 10-22 Data alert advanced schedule settings

 

Last, you must provide email settings for the data alert by specifying at least one email address as

a recipient for the data alert. If you want to send the data alert to multiple recipients, separate each

email address with a semicolon. You can also include a subject and a description for the data alert,

both of which are static strings.

 

Alerting Service

 

The Reporting Services alerting service manages the process of refreshing the data feed and applying

the rules in the data alert definition, as shown in Figure 10-23. Regardless of the results, the alerting

service adds an alerting instance to the alerting database to record the outcome of the evaluation. If

any rows in the data feed satisfy the conditions of the rules during processing, the alerting services

generate an email containing the alert results.

 

SQL Server

Agent Job

Alerting

DB

Create

alert

instance

Alerting

DB

Send

email

Apply

rules

Read data

feed

Reporting Services Alerting Service

 

FIGURE 10-23 Alerting service processing of data alerts

 

 

 

The email for a successful data alert includes the user name of the person who created the alert,

the description of the alert from the alert definition, and the rows from the data feed that generated

the alert, as shown in Figure 10-24. It also includes a link to the report, a description of the alert rules,

and the report parameters used when reading the data feed. The message is sent from the account

you specify in the email settings of the Reporting Services shared services application.

 

 

 

FIGURE 10-24 Email message containing a successful data alert

 

Note If an error occurs during alert processing, the alerting service saves the alerting

instance to the alerting database and sends an alert message describing the error to the

recipients.

 

Data Alert Manager

 

You use the Data Alert Manager to lists all data alerts you create for a report, as shown in

Figure 10-25. To open the Data Alert Manager, open the document library containing your report,

click the down arrow to the right of the report name, and select Manage Data Alerts. You can see

the alerts listed for the current report, but you can use the drop-down list at the top of the page to

change the view to a list of data alerts for all reports or a different report.

 

 

 

FIGURE 10-25 Data Alert Manager

 

 

 

The Data Alert Manager shows you the number of alerts sent by data alert, the last time it was run,

the last time it was modified, and the status of its last execution. If you right-click a data alert on this

page, you can edit the data alert, delete it, or run the alert on demand. No one else can view, edit, or

run the data alerts you create, although the site administrator can view and delete your data alerts.

 

If you are a site administrator, you can use the Manage Data Alerts link on the Site Settings page to

open the Data Alert Manager. Here you can select a user from a drop-down list to view all data alerts

created by the user for all reports. You can also filter the data alerts by report. As a site administrator,

you cannot edit these data alerts, but you can delete them if necessary.

 

Alerting Configuration

 

Data does not accumulate indefinitely in the alerting database. The report server configuration file

contains several settings you can change to override the default intervals for cleaning up data in the

alerting database or to disable alerting. There is no graphical interface available for making changes

to these settings. Instead, you must manually edit the RsReportServer.config file to change the

settings

shown in Table 10-2.

 

TABLE 10-2 Alerting Configuration Settings in RsReportServer.config

 

Alerting Setting

 

Description

 

Default Value

 

AlertingCleanupCyclingMinutes

 

Interval for starting the cleanup cycle of alerting

 

20

 

AlertingExecutionLogCleanupMinutes

 

Number of minutes to retain data in the alerting execution

log

 

10080

 

AlertingDataCleanupMinutes

 

Number of minutes to retain temporary alerting data

 

360

 

AlertingMaxDataRetentionDays

 

Number of days to retain alert execution metadata, alert

instances,

and execution results

 

180

 

IsAlertingService

 

Enable or disable the alerting service with True or False

values, respectively

 

True

 

 

 

 

 

The SharePoint configuration database contains settings that control how many times the alerting

service retries execution of an alert and the delay between retries. The default for MaxRetries is 3 and

for SecondsBeforeRetry is 900. You can change these settings by using PowerShell cmdlets. These settings

apply to all alert retries, but you can configure different retry settings for each of the following

events:

 

¦ ¦ FireAlert On-demand execution of an alert launched by a user

¦ ¦ FireSchedule Scheduled execution of an alert launched by a SQL Server Agent job

¦ ¦ CreateSchedule Process to save a schedule defined in a new or modified data alert

¦ ¦ UpdateSchedule Modification of a schedule in a data alert

¦ ¦ DeleteSchedule Deletion of a data alert

¦ ¦ GenerateAlert Alerting-service processing of data feeds, rules, and alert instances

¦ ¦ DeliverAlert Preparation and delivery of email message for a data alert

 

 

 

 

Index

 

251

 

Symbols

 

32-bit editions of SQL Server 2012, 17

 

64-bit editions of SQL Server 2012, 17

 

64-bit processors, 17

 

debugging on, 99

 

[catalog].[executions] table, 137

 

A

 

absolute environment references, 133–135

 

Active Directory security modules, 71

 

Active Operations dialog box, 130

 

active secondaries, 32–34

 

Administrator account, 176

 

Administrator permissions, 211

 

administrators

 

data alert management, 250

 

Database Engine access, 72

 

entity management, 192

 

master data management in Excel, 187–196

 

tasks of, 135–139

 

ADO.NET connection manager, 112–113

 

ad reporting tool, 229

 

Advanced Encryption Standard (AES), 71

 

affinity masks, 216

 

aggregate functions, 220

 

alerting service, 248–249. See also data alerts

 

All Executions report, 138

 

allocation_failure event, 90

 

All Operations report, 138

 

Allow All Connections connection mode, 33

 

Allow Only Read-Intent Connections connection mode,

33

 

All Validations report, 138

 

ALTER ANY SERVER ROLE, permissions on, 72

 

ALTER ANY USER, 70

 

ALTER SERVER AUDIT, WHERE clause, 63

 

ALTER USER DEFAULT_SCHEMA option, 59

 

AlwaysOn, 4–6, 21–23

 

AlwaysOn Availability Groups, 4–5, 23–32

 

availability group listeners, 28–29

 

availability replica roles, 26

 

configuring, 29–31

 

connection modes, 27

 

data synchronization modes, 27

 

deployment examples, 30–31

 

deployment strategy, 23–24

 

failover modes, 27

 

monitoring with Dashboard, 31–32

 

multiple availability group support, 25

 

multiple secondaries support, 25

 

prerequisites, 29–30

 

shared and nonshared storage support, 24

 

AlwaysOn Failover Cluster Instances, 5, 34–36

 

Analysis Management Objects (AMOs), 217

 

Analysis Services, 199–217

 

cached files settings, 225

 

cache reduction settings, 225

 

credentials for data import, 204

 

database roles, 210–211

 

DirectQuery mode, 213

 

event tracing, 215

 

in-memory vs. DirectQuery mode, 213

 

model designer, 205–206

 

model design feature support, 200–202

 

multidimensional models, 215

 

multiple instances, 201

 

with NUMA architecture, 216

 

PowerShell cmdlets for, 217

 

Process Full command, 215

 

processors, number of, 216

 

programmability, 217

 

projects, 201–202

 

query modes, 214

 

schema rowsets, 216

 

 

 

252

 

Analysis Services (continued)

 

server management, 215–216

 

server modes, 199–201

 

tabular models, 203–214

 

templates, 201–202

 

Analysis Services Multidimensional and Data Mining

Project template, 202

 

Analysis Services tabular database, 233, 234

 

Analysis Services Tabular Project template, 202

 

annotations, 191, 198

 

reviewing, 191

 

applications

 

data-tier, 10

 

file system storage of files and documents, 77

 

asynchronous-commit availability mode, 27

 

asynchronous database mirroring, 21–22

 

asynchronous data movement, 24

 

attribute group permissions, 184–185

 

attribute object permissions, 183

 

attributes

 

domain-based, 193–194

 

modifying, 193

 

Attunity, 103, 104, 111

 

auditing, 62–67

 

audit, creating, 64–65

 

audit data loss, 62

 

audit failures, continuing server operation, 63

 

audit failures, failing server operation, 63, 65

 

Extended Events infrastructure, 66

 

file destination, 63, 64

 

record filtering, 66–67

 

resilience and, 62–65

 

support on all SKUs, 62

 

user-defined audit events, 63, 65–66

 

users of contained databases, 63

 

audit log

 

customized information, 63, 65–66

 

filtering events, 63, 66–67

 

Transact-SQL stack frame information, 63

 

authentication

 

claims-based, 231, 233

 

Contained Database Authentication, 68–69

 

middle-tier, 66

 

user, 9

 

automatic failover, 27

 

availability. See also high availability (HA)

 

new features, 4–6

 

systemwide outages and, 62

 

availability databases

 

backups on secondaries, 33–34

 

failover, 25

 

hosting of, 26

 

availability group listeners, 28–29

 

availability groups. See AlwaysOn Availability Groups

 

availability replicas, 26

 

B

 

backup history, visualization of, 39–40

 

backups on secondaries, 33–34

 

bad data, 141

 

batch-mode query processing, 45

 

beyond-relational paradigm, 11, 73–90

 

example, 76

 

goals, 74

 

pain points, 73

 

bidirectional data synchronization, 9

 

BIDS (Business Intelligence Development Studio), 93.

See also SSDT (SQL Server Data Tools)

 

binary certificate descriptions, 71

 

binary large objects (BLOBs), 74

 

storage, 75

 

BISM (Business Intelligence Semantic Model)

 

connection files, as Power View data source, 233

 

Connection type, adding to PowerPivot site

document library, 233–234

 

schema, 217

 

BLOBs (binary large objects), 74

 

storage, 75

 

B-tree indexes, 42–43

 

situations for, 48

 

bubble charts in Power View, 239

 

Business Intelligence Development Studio (BIDS), 93

 

Business Intelligence edition of SQL Server 2012, 14

 

Business Intelligence Semantic Model (BISM).

See BISM (Business Intelligence Semantic

Model)

 

C

 

cache

 

managing, 225

 

retrieving data from, 213

 

Cache Connection Manager, 98

 

calculated columns, 207

 

cards in Power View, 238

 

catalog, 119, 128–135

 

adding and retrieving projects, 119

 

administrative tasks, 129

 

creating, 128–129

 

Analysis Services

 

 

 

253

 

adding and retrieving projects (continued)

 

database name, 128–129

 

encrypted data in, 139

 

encryption algorithms, 130

 

environment objects, 132–135

 

importing projects from, 121

 

interacting with, 129

 

operations information, 130–131

 

permissions on, 139

 

previous project version information, 131

 

properties, 129–131

 

purging, 131

 

catalog.add_data_tap procedure, 137

 

Catalog Properties dialog box, 129–130

 

catalog.set_execution_parameter_value stored

procedure, 127

 

CDC (change data capture), 57, 111–114

 

CDC Control Task Editor, 113

 

CDC Source, 113–114

 

CDC Splitter, 113–114

 

certificates

 

binary descriptions, 71

 

key length, 71

 

change data capture (CDC), 57, 111–114

 

control flow, 112–113

 

data flow, 113–114

 

change data capture tables, 111

 

chargeback, 8

 

charts in Power View, 237, 241

 

Check For Updates To PowerPivot Management

Dashboard.xlsx rule, 227

 

child packages, 102

 

circular arc segments, 87–89

 

circular strings, 88

 

claims-based authentication, 231, 233

 

cleansing, 162

 

confidence score, 150

 

corrections, 163

 

exporting results, 163–164

 

scheduled, 171

 

syntax error checking, 146

 

cleansing data quality projects, 161–164

 

monitoring, 167

 

Cleansing transformation, 141

 

configuring, 171–172

 

monitoring, 167

 

client connections, allowing, 33

 

cloud computing licensing, 15

 

Cloud Ready Information Platform, 3

 

cloud services, 9

 

clustering. See also failover clustering

 

clusters

 

health-detection policies, 5

 

instance-level protection, 5

 

multisubnet, 34

 

code, pretty-printing, 139

 

collections management, 183

 

columns

 

calculated, 207

 

with defaults, 40

 

filtering view, 109

 

mappings, 108–109

 

reporting properties, 212–213

 

variable number, 105

 

columnstore indexes, 6, 41–56

 

batch-mode query processing, 45

 

best practices, 54

 

business data types supported, 46

 

columns to include, 49–50

 

compression and, 46

 

creating, 49–54

 

data storage, 42–43

 

design considerations, 47–48

 

disabling, 48

 

hints, 53–54

 

indicators and performance cost details, 52–53

 

loading data, 48–49

 

nonclustered, 50

 

query speed improvement, 44–45

 

restrictions, 46–47

 

storage organization, 45

 

updating, 48–49

 

Columnstore Index Scan Operator icon, 52

 

column store storage model, 42

 

composite domains, 151–153

 

cross-domain rules, 152

 

managing, 152

 

mapping fields to, 162, 172

 

parsing methods, 152

 

reference data, 152

 

value relations, 153

 

compound curves, 88–89

 

compression of column-based data, 44

 

Conceptual Schema Definition Language (CSDL)

extensions, 217

 

confidence score, 150, 162

 

specifying, 169

 

configuration files, legacy, 121

 

CONMGR file extension, 99

 

CONMGR file extension

 

 

 

connection managers

 

creating, 104

 

expressions, 100

 

package connections, converting to, 99

 

sharing, 98–99

 

connection modes, 27

 

Connections Managers folder, 98

 

Connections report, 138

 

Contained Database Authentication, 68–69

 

enabling, 68–69

 

security threats, 70–71

 

users, creating, 69–70

 

contained databases, 9

 

auditing users, 63

 

database portability and, 69

 

domain login, creating users for, 70

 

initial catalog parameter, 69

 

users with password, creating, 70

 

control flow

 

Expression Task, 101–102

 

status indicators, 101

 

control flow items, 97–98

 

Convert To Package Deployment Model command, 118

 

correct domain values, 147

 

CPU usage, maximum cap, 8

 

Create A New Availability Group Wizard, 28

 

Create Audit Wizard, 64

 

CREATE CERTIFICATE FROM BINARY, 71

 

Create Domain dialog box, 147

 

CreateSchedule event, 250

 

CREATE SERVER AUDIT, WHERE clause, 63

 

CREATE SERVER ROLE, permissions on, 72

 

CREATE USER DEFAULT_SCHEMA, 59

 

cross-domain rules, 152

 

cryptography

 

enhancements, 71

 

hashing algorithms, 71

 

CSDL (Conceptual Schema Definition Language)

extensions, 217

 

CSV files, extracting data, 105

 

curved polygons, 89

 

Customizable Near operator, 82

 

Custom Proximity Operator, 82

 

D

 

DACs (data-tier applications), 10

 

dashboard

 

AlwaysOn Availability Groups, monitoring, 31–32

 

launching, 31

 

data

 

cleansing. See cleansing

 

data integration, 73–74

 

de-duplicating, 141, 146, 173

 

loading into tables, 48–49

 

log shipping, 22

 

managing across systems, 11

 

normalizing, 146

 

partitioning, 47, 48–49

 

protecting, 23

 

redundancy, 4

 

refreshing, 227

 

spatial, 86–90

 

structured and nonstructured, integrating, 73–74

 

volume of, 41

 

data-access modes, 104

 

Data Alert Designer, 246–248

 

Data Alert Manager, 249–250

 

data alerts, 229, 246–250

 

alerting service, 248–249

 

configuring, 250

 

creating, 247

 

Data Alert Designer, 246–248

 

Data Alert Manager, 249–250

 

email messages, 249

 

email settings, 248

 

errors, 249

 

managing, 249–250

 

retry settings, 250

 

scheduling, 248

 

Data Analysis Expression (DAX). See DAX (Data

Analysis Expression)

 

data and services ecosystem, 74–75

 

Database Audit Specification objects, 62

 

database authentication, 67–71

 

Database Engine

 

change data capture support, 111

 

local administrator access, 72

 

upgrading, 175

 

user login, 67

 

database instances. See also SQL Server instances

 

orphaned or unused logins, 68

 

database mirroring, 23

 

alternative to, 23

 

asynchronous, 21–22

 

synchronous, 22

 

database portability

 

contained databases and, 69–70

 

Database Engine authentication and, 67

 

database protection, 4–5

 

connection managers

 

 

 

Database Recovery Advisor, 39–40

 

Database Recovery Advisor Wizard, 34

 

database restore process, visual timeline, 6

 

databases

 

Analysis Services tabular, 233

 

authentication to, 67

 

bidirectional data synchronization, 9

 

contained, 9, 68–69

 

failover to single unit, 4

 

portability, 9

 

report server, 231

 

SQL Azure, deploying to, 9

 

Database Wizard, 176

 

datacenters, secondary replicas in, 4

 

Data Collection Interval rule, 226, 227

 

data destinations, 103–104

 

ODBC Destinations, 104–105

 

data extraction, 105

 

data feeds, 247–248

 

refreshing, 248

 

data flow

 

capturing data from, 137

 

components, developing, 107

 

groups, 109–110

 

input and output column mapping, 108–109,

172–173

 

sources and destinations, 103–106

 

status indicators, 101

 

data flow components, 97–98

 

column references, 108–109

 

disconnected, 108

 

grouping, 109–110

 

data flow designer, error indicator, 108–109

 

data-flow tasks, 103–104

 

CDC components, 112

 

data latency, 32

 

DataMarket subscriptions, 168

 

Data Quality Client, 142–143

 

Activity Monitoring, 166–167

 

Administration feature, 166–170

 

client-server connection, 143

 

Configuration area, 167–170

 

Configuration button, 167

 

domain value management, 149

 

Excel support, 143

 

installation, 143

 

New Data Quality Project button, 161

 

New Knowledge Base button, 145

 

Open Knowledge Base button, 144

 

tasks, 143

 

Data Quality Client log, 170

 

data quality management, 141

 

activity monitoring, 166–167

 

configuring, 167

 

improving, 107

 

knowledge base management, 143–161

 

data quality projects, 143, 161–166

 

cleansing, 161–164

 

exporting results, 161

 

matching, 164–166

 

output, 146

 

Data Quality Server, 141–142

 

activity monitoring, 166–167

 

connecting to Integration Services package, 171

 

installation, 141–142

 

log settings, 170

 

remote client connections, 142

 

user login creation, 142

 

Data Quality Server log, 170

 

Data Quality Services (DQS). See DQS (Data Quality

Services)

 

data sources. See also source data

 

creating, 103–104

 

importing data from, 204

 

installing types, 104

 

ODBC Source, 104–105

 

data stewards, 143

 

master data management in Excel, 187–196

 

data storage

 

column-based, 42–43

 

in columnstore indexes, 42–43

 

row-based, 42–43

 

data synchronization modes, 27

 

data tables. See also tables

 

converting to other visualizations, 237–241

 

designing, 234–237

 

play axis area, 239–240

 

data taps, 137

 

data-tier applications (DACs), 10

 

data types, columnstore index-supported, 46

 

data viewer, 110–111

 

data visualization, 229

 

arrangement, 238

 

cards, 238

 

charts, 237

 

multiples, 241

 

play axis, 239–240

 

in Power View, 237–241

 

separate views, 241–242

 

slicing data, 243

 

data visualization

 

 

 

data visualization (continued)

 

sort order, 241

 

tile containers, 243–244

 

tiles, 239

 

view filters, 244–245

 

visualization filters, 245

 

data warehouses

 

queries, improving, 6

 

query speeds, 44–45

 

sliding-window scenarios, 7

 

date filters, 219–220, 244–245

 

DAX (Data Analysis Expression)

 

new functions, 222–224

 

row-level filters, 211

 

dbo schema, 58

 

debugger for Transact-SQL, 7

 

debugging on 64-bit processors, 99

 

Default Field Set property, 212

 

Default Image property, 212

 

Default Label property, 212

 

default schema

 

creation script, 59

 

for groups, 58–59

 

for SQL Server users, 58

 

delegation of permissions, 139

 

DeleteSchedule event, 250

 

delimiters, parsing, 152

 

DeliverAlert event, 250

 

dependencies

 

database, 9

 

finding, 216

 

login, 67

 

Deploy Database To SQL Azure wizard, 9

 

deployment models, 116–122

 

switching, 118

 

deployment, package, 120

 

Description parameter property, 123

 

Designing and Tuning for Performance Your SSIS

Packages in the Enterprise link, 96–97

 

destinations, 103–105

 

Developer edition of SQL Server 2012, 15

 

Digital Trowel, 167

 

directory name, enabling, 79

 

DirectQuery mode, 213–215

 

DirectQuery With In-Memory mode, 214

 

disaster recovery, 21–40

 

disconnected components, 108

 

discovery. See knowledge discovery

 

DMV (Dynamic Management View), 8

 

documents, finding, 85

 

DOCX files, 230

 

domain-based attributes, 193–194

 

domain management, 143–154

 

DQS Data knowledge base, 144–145

 

exiting, 154

 

knowledge base creation, 145–154

 

monitoring, 167

 

domain rules, 150–151

 

disabling, 151

 

running, 155

 

domains

 

composite, 151–152

 

creating, 146

 

data types, 146

 

importing, 146

 

linked, 153–154

 

mapping to source data, 158, 161–162, 164

 

properties, 146–147

 

term-based relations, 149

 

domain values

 

cross-domain rules, 152

 

domain rules, 150–151

 

formatting, 146

 

leading values, 148–149, 173

 

managing, 156–157

 

setting, 147

 

spelling checks, 146

 

synonyms, 148–149, 163, 173

 

type settings, 147–148

 

downtime, reducing, 40

 

dqs_administrator role, 142

 

DQS Cleansing transformation, 107

 

DQS Cleansing Transformation log, 170

 

DQS Data knowledge base, 144–145, 196

 

DQS (Data Quality Services), 107, 141–173

 

administration, 143, 166–170

 

architecture, 141–143

 

configuration, 167–170

 

connection manager, 171

 

Data Quality Client, 142–143. See also Data

Quality Client

 

data quality projects, 161–166

 

Data Quality Server, 141–142. See also Data

Quality Server

 

Domain Management area, 143–154

 

Excel, importing data from, 147

 

integration with SQL Server features, 170–173

 

knowledge base management, 143–161

 

Knowledge Discovery area, 154–157

 

log settings, 169–170

 

data visualization

 

 

 

DQS (Data Quality Services) (continued)

 

Matching Policy area, 157–161

 

profiling notifications, 169

 

dqs_kb_editor role, 142

 

dqs_kb_operator role, 142

 

DQS_MAIN database, 142

 

DQS_PROJECTS database, 142

 

DQS_STAGING_DATA database, 142

 

drop-rebuild index strategy, 47

 

DTSX files, 139

 

converting to ISPAC files, 121–122

 

Dynamic Management View (DMV), 8

 

E

 

EKM (Extensible Key Management), 57

 

encryption

 

enhancements, 71

 

for parameter values, 130

 

Enterprise edition of SQL Server 2012, 12–13

 

entities. See also members

 

code values, creating automatically, 178–179

 

configuring, 178

 

creating, 192–193

 

domain-based attributes, 193–194

 

managing, 177–179

 

many-to-many mappings, 179

 

permissions on, 183–185

 

entry-point packages, 119–120

 

environment references, creating, 133–135

 

environments

 

creating, 132

 

package, 119

 

projects, connecting, 132

 

environment variables, 119, 132

 

creating, 132–133

 

error domain values, 148

 

errors, alert-processing, 249

 

events, multidimensional, 215

 

event tracing, 215

 

Excel. See Microsoft Excel

 

Excel 2010 renderer, 229–230

 

Execute Package dialog box, 136

 

Execute Package task, 102–103

 

updating, 122

 

Execute Package Task Editor, 102–103

 

execution objects, creating, 135–136

 

execution parameter values, 127

 

execution, preparing packages for, 135–136

 

exporting content

 

cleansing results, 163–164

 

matching results, 165

 

exposures reported to NIST, 10

 

Express edition of SQL Server 2012, 15

 

Expression Builder, 101–102

 

expressions, 100

 

parameter usage in, 124

 

size limitations, 115–116

 

Expression Task, 101–102

 

Extended Events, 66

 

event tracing, 215

 

management, 12

 

monitoring, 216

 

new events, 90

 

Extensible Key Management (EKM), 57

 

extract, transform, and load (ETL) operations, 111

 

F

 

failover

 

automatic, 23, 27

 

of availability databases, 25

 

manual, 23, 27

 

modes, 27

 

to single unit, 4

 

startup and recovery times, 5

 

failover clustering, 21–22. See also clustering

 

for consolidation, 39

 

flexible failover policy, 34

 

limitations, 23

 

multi-subnet, 5

 

Server Message Block support, 39

 

TempDB on local disk, 34

 

Failover Cluster Instances (FCI), 5

 

Failover Cluster Manager Snap-in, 24

 

failover policy, configuring, 34

 

failures, audit

 

, 62, 63

 

FCI (Failover Cluster Instances), 5

 

fetching, optimizing, 44

 

file attributes, updating, 186

 

file data, application compatibility with, 11

 

files

 

converting to ISPAC file, 121

 

copying to FileTable, 80

 

ragged-right delimited, 105

 

FILESTREAM, 74, 76–77

 

enabling, 78

 

file groups and database files, configuring, 79

 

FILESTREAM

 

 

 

FILESTREAM (continued)

 

FileTable, 77

 

scalability and performance, 76

 

file system, unstructured data storage in, 74–75

 

FileTable, 77–81

 

creating, 80

 

documents and files, copying to, 80

 

documents, viewing, 80–81

 

managing and securing, 81

 

prerequisites, 78–79

 

filtering, 204

 

before importing, 204

 

dates, 219–220

 

hierarchies, 216

 

row-level, 211

 

view, 244–245

 

visualization, 245

 

FireAlert event, 250

 

FireSchedule event, 250

 

Flat File source, 105–106

 

folder-level access, 139

 

Full-Text And Semantic Extractions For Search

feature, 83

 

full-text queries, performance, 81

 

full-text search, 11–12

 

scale of, 81

 

word breakers and stemmers, 82

 

G

 

GenerateAlert event, 250

 

geometry and geography data types, 86

 

new data types, 87–89

 

Getting Started window, 95–97

 

global presence, business, 5

 

grid view data viewer, 110–111

 

groups

 

data-flow, 109–110

 

default schema for, 58–59

 

guest accounts, database access with, 71

 

H

 

hardware

 

active secondaries, 32–34

 

utilization, increasing, 4

 

hardware requirements, 16

 

for migrations, 20

 

HASHBYTES function, 71

 

health rules, server, 226–227

 

hierarchies, 208–209

 

arranging members in, 177, 179, 181

 

filtering, 216

 

managing, 180

 

permissions inheritance, 184

 

in PowerPivot for Excel, 221

 

high availability (HA), 4–6, 21–40

 

active secondaries, 32–34

 

AlwaysOn, 21–23

 

AlwaysOn Availability Groups, 23–32

 

AlwaysOn Failover Cluster Instances, 34–36

 

technologies for, 21

 

HostRecordTTL, configuring, 36

 

hybrid solutions, 3

 

I

 

Image URL property, 212

 

implicit measures, 220

 

Import from PowerPivot template, 202

 

Import from Server template, 202

 

indexes

 

columnstore, 6, 41–56

 

varchar(max), nvarchar(max), and varbinary(max)

columns, 7

 

index hints, 53–54

 

initial catalog parameter, 69

 

initial load packages, change data processing, 112

 

in-memory cache, 213

 

In-Memory query mode, 214

 

In-Memory With DirectQuery query mode, 214

 

in-place upgrades, 17–19

 

Insert Snippet menu, 7

 

installation, 17

 

hardware and software requirements, 16–17

 

on Windows Server Core, 38

 

Insufficient CPU Resource Allocation rule, 226

 

Insufficient CPU Resources On The System rule, 226

 

Insufficient Disk Space rule, 226

 

Insufficient Memory Threshold rule, 226

 

Integration Services, 93–139

 

administration tasks, 135–139

 

catalog, 128–135

 

change data capture support, 111–114

 

Cleansing transformation, 171–172

 

connection managers, sharing, 98–99

 

Connections Project template, 94

 

control flow enhancements, 101–103

 

data flow enhancements, 103–111

 

data viewer, 110–111

 

FILESTREAM

 

 

 

Integration Services (continued)

 

deployment models, 116–122

 

DQS integration, 171–172

 

Execute Package Task, 102–103

 

expressions, 100

 

Expression Task, 101–102

 

Getting Started window, 95, 96–97

 

interface enhancements, 93–101

 

New Project dialog box, 93–94

 

operations dashboard, 138

 

package design, 114–116

 

package-designer interface, 95–96

 

package file format, 139

 

packages, sorting, 100

 

parameters, 122–127

 

Parameters window, 95

 

Project template, 94

 

scripting engine, 99

 

SSIS Toolbox, 95, 97–98

 

status indicators, 101

 

stored procedure, automatic execution, 128

 

templates, 93

 

transformations, 106–107, 141, 171–172

 

Undo and Redo features, 100

 

Variables button, 95

 

zoom control, 95

 

Integration Services Deployment Wizard, 120

 

Integration Services Import Project Wizard, 94, 121

 

Integration Services Package Upgrade Wizard, 121

 

Integration Services Project Conversion Wizard, 121

 

IntelliSense enhancements, 7

 

invalid domain values, 148

 

IO, reducing, 44

 

isolation, 8

 

ISPAC file extension, 119

 

ISPAC files

 

building, 119

 

converting files to, 121–122

 

deployment, 120

 

importing projects from, 121

 

K

 

Keep Unique Rows property, 212

 

Kerberos authentication, 57, 233

 

key performance indicators (KPIs), 208

 

key phrases, searching on, 82–85

 

knowledge base management, 143–161

 

domain management, 143–154

 

knowledge discovery, 154–157

 

matching, 157–161

 

source data, 154

 

knowledge bases

 

composite domains, 151–153

 

creating, 145–146

 

defined, 143

 

domain rules, 150–151

 

domains, 145–146. See also domains

 

domain values, 147

 

DQS Data, 144–145

 

knowledge discovery, 154–157

 

linked domains, 153–154

 

locking, 144, 154

 

matching policy, 157–161

 

publishing, 154, 157

 

reference data, 150

 

updating, 157

 

knowledge discovery, 154–157

 

domain rules, running, 155

 

domain value management, 156–157

 

monitoring, 167

 

pre-processing, 155

 

KPIs (key performance indicators), 208

 

L

 

Large Objects (LOBs), 45

 

LEFT function, 116

 

legacy SQL Server instances, deploying SQL Server

2012 with, 19–20

 

licensing, 15–16

 

line-of-business (LOB) re-indexing, 40

 

linked domains, 153–154

 

Load To Connection Ratio rule, 226

 

LOBs (Large Objects), 45

 

local storage, 5

 

logging

 

backups on secondaries, 33

 

package execution, 137

 

server-based, 137

 

logon dependencies, 67

 

log shipping, 22, 23

 

M

 

machine resources, vertical isolation, 8

 

manageability features, 7–10

 

managed service accounts, 72

 

MANAGE_OBJECT_PERMISSIONS permission, 139

 

manual failover, 27

 

manual failover

 

 

 

mappings

 

data source fields to domains, 154–155, 158, 172

 

input to output columns, 108–109

 

many-to-many, 179

 

master data management, 177. See also MDS (Master

Data Services)

 

bulk updates and export, 197

 

entity management, 177–179

 

in Excel, 187–196

 

integration management, 181–183

 

sharing data, 194

 

Master Data Manager, 177–185

 

collection management, 180–181

 

entity management, 177–179

 

Explorer area, 177–181

 

hierarchy management, 180

 

Integration Management area, 181–183

 

mappings, viewing, 179

 

Metadata model, 197

 

permissions assignment, 183–185

 

Version Management area, 183

 

Master Data Services (MDS). See MDS (Master Data

Services)

 

matching, 165

 

exporting results, 165

 

MDS Add-in entities, 194–196

 

MDS support, 141

 

results, viewing, 160

 

rules, 158–159

 

score, 158, 169

 

matching data quality projects, 164–166

 

monitoring, 167

 

matching policy

 

defining, 157–161

 

mapping, 158

 

matching rule creation, 158–159

 

monitoring, 167

 

results, viewing, 160

 

Similarity parameter, 159

 

testing, 159

 

Weight parameter, 159

 

Maximum Files (or Max_Files) audit option, 63

 

Maximum Number of Connections rule, 226

 

mdm.udpValidateModel procedure, 183

 

MDS Add-in for Excel, 173, 175, 187–196

 

bulk updates and export, 197

 

data quality matching, 194–196

 

filtering data, 188–190

 

installation, 187

 

link to, 177

 

loading data into worksheet, 190

 

master data management tasks, 187–192

 

MDS database connection, 187–188

 

model-building tasks, 192–194

 

shortcut query files, 194

 

MDS databases

 

connecting to, 187–188

 

upgrading, 175

 

MDS (Master Data Services), 175–198

 

administration, 198

 

bulk updates and export, 197

 

configuration, 176–177

 

Configuration Manager, 175–176

 

DQS integration, 195

 

DQS matching with, 173

 

filtering data from, 188–189

 

Master Data Manager, 177–185

 

MDS Add-in for Excel, 187–196

 

Metadata model, 197

 

model deployment, 185–187

 

new installations, 176

 

permissions assignment, 183–185

 

publication process, 191–192

 

SharePoint integration, 197

 

staging process, 181–183

 

system settings, 176–177

 

transactions, 198

 

upgrades, 175–176

 

MDSModelDeploy tool, 185–186

 

measures, 207–208

 

members

 

adding and deleting, 177–178

 

attributes, editing, 178

 

collection management, 180–181

 

hierarchy management, 180

 

loading in Excel, 188–190

 

transactions, reviewing, 191

 

memory allocation, 216

 

memory management, for merge and merge join

transformations, 107

 

memory_node_oom_ring_buffer_recorded event, 90

 

Merge Join transformation, 107

 

Merge transformation, 107

 

metadata, 108

 

managing, 108–109

 

updating, 186

 

Metadata model, 197

 

Micorsoft Word 2010 renderer, 230

 

Microsoft Excel, 143. See also worksheets, Excel

 

data quality information, exporting to, 167

 

mappings

 

 

 

Microsoft Excel (continued)

 

Excel 2010 renderer, 229–230

 

importing data from, 147

 

MDS Add-in for Excel, 173, 187–196

 

PowerPivot for Excel, 218–222

 

tabular models, analyzing in, 211–212

 

Microsoft Exchange beyond-relational functionality,

76

 

Microsoft Office Compatibility Pack for Word, Excel,

and PowerPoint, 230

 

Microsoft Outlook beyond-relational functionality,

76

 

Microsoft PowerPoint, Power View exports, 246

 

Microsoft.SqlServer.Dts.Pipeline namespace, 107

 

Microsoft trustworthy computing initiative, 10

 

middle-tier authentication, 66

 

migration strategies, 19–20

 

high-level, side-by-side, 20

 

mirroring

 

asynchronous, 21–22

 

synchronous, 22

 

Model.bim file, 203

 

model deployment, 185–186

 

Model Deployment Wizard, 185

 

model permissions object tree, 184

 

MOLAP engine, 215

 

mount points, 39

 

multidimensional events, 215

 

multidimensional models, 215

 

multidimensional server mode, 199–201

 

multisite clustering, 34

 

multisubnet clustering, 34–35

 

multiple IP addresses, 35

 

multitenancy, 8

 

N

 

NEAR operator, 12

 

network names of availability groups listeners, 28

 

New Availability Group Wizard, 29

 

Specify Replicas page, 30

 

New Project dialog box (Integration Services), 93–94

 

“New Spatial Features in SQL Server 2012”

whitepaper, 89

 

nonoverlapping clusters of records, 160, 165

 

nonshared storage, 26

 

nontransactional access, 78

 

enabling at database level, 78–79

 

normalization, 146

 

numeric data types, filtering on, 244

 

O

 

ODBC Destination, 104–105

 

ODBC Source, 104–105

 

On Audit Log Failure: Continue feature, 63

 

On Audit Log Failure: Fail Operation feature, 63

 

On Audit Shut Down Server feature, 62

 

online indexing, 7

 

online operation, 40

 

operations management, 130–131

 

operations reports, 137

 

outages

 

planned, 18

 

systemwide, 62, 68

 

overlapping clusters of records, 160, 165

 

P

 

package connections, converting connection

managers to, 99

 

package deployment model, 116–117

 

package design

 

expressions, 115–116

 

variables, 114–115

 

package execution, 136

 

capturing data during, 137

 

logs, 137

 

monitoring, 136

 

preparing for, 135–136

 

reports on, 137

 

scheduling, 136

 

package objects, expressions, 100

 

package parameters, 124

 

prefix, 125

 

packages

 

change data processing, 112

 

connection managers, sharing, 98–99

 

deploying, 120, 185–186

 

entry-point, 119–120

 

environments, 119

 

file format, 139

 

legacy, 121–122

 

location, specifying, 102

 

preparing for execution, 135–136

 

security of, 139

 

sorting by name, 100

 

storing in catalog, 119

 

validating, 126, 135

 

page_allocated event, 90

 

page_freed event, 90

 

page_freed event

 

 

 

Parameterizing the Execute SQL Task in SSIS link, 97

 

parameters, 95, 122–127

 

bindings, mapping, 102–103

 

configuration, converting to, 122

 

configuring, 122

 

environment variables, associating, 119

 

execution parameter value, 127

 

in Expression Builder, 102

 

implementing, 124–125

 

package-level, 120, 122, 124

 

post-deployment values, 125–126

 

project-level, 120, 122–123

 

properties for, 123

 

scope, 119

 

server default values, 126–127

 

parameter value, 123

 

parsing composite domains, 152

 

partitioning new data, 47–49

 

Partition Manager, 209–210

 

partitions, 209–210

 

number supported, 7

 

switching, 48–49

 

patch management, 40

 

Windows Server Core installation and, 36

 

performance

 

data warehouse workload enhancements, 6

 

new features, 6–7

 

query, 41, 44–45

 

permissions

 

on ALTER ANY SERVER ROLE, 72

 

on CREATE SERVER ROLE, 72

 

delegation, 139

 

management, 139

 

on master data, 183–185

 

on SEARCH PROPERTY LIST, 72

 

SQL Server Agent, 231

 

updating, 186

 

per-service SIDs, 72

 

perspectives, 209, 220

 

in PowerPivot for Excel, 222

 

pivot tables, 218–222, 224–227

 

numeric columns, 221

 

Pivot transformation, 106

 

planned outages for upgrades, 18

 

Policy-Based Management, 57

 

portability, database, 9

 

PowerPivot, 42

 

PowerPivot for Excel, 218–222

 

Advanced tab, 220

 

Data View and Diagram View buttons, 218

 

DAX, 222–224

 

Design tab, 219

 

installation, 218

 

Mark As Date Table button, 219

 

model enhancements, 221–222

 

Sort By Column button, 219

 

usability enhancements, 218–221

 

PowerPivot for SharePoint, 224–227

 

data refresh configuration, 227

 

disk space usage, 225

 

installation, 224

 

management, 224–227

 

server health rules, 226–227

 

workbook caching, 225

 

PowerPivot for SharePoint server mode, 199–201

 

PowerPivot workbooks

 

BI Semantic Model Connection type, adding, 233

 

as Power View data source, 233

 

Power View design environment, opening, 234

 

Power View, 229, 232–246

 

arranging items, 238

 

browser, 232

 

cards, 238

 

charts, 237

 

current view display, 236

 

data layout, 232

 

data sources, 232, 233

 

data visualization, 237–241

 

design environment, 234–237

 

display modes, 245–246

 

filtering data, 243–245

 

Filters area, 244

 

highlighting values, 242

 

layout sections, 236

 

moving items, 237

 

multiple views, 241–242

 

PowerPoint export, 246

 

printing views, 246

 

saving files, 237

 

tiles, 239

 

views, creating, 241–242

 

pretty-printing, 139

 

primary database servers, offloading tasks from, 23,

32–34

 

Process Full command, 215

 

Process permissions, 211

 

profiling notifications, 169

 

programmability enhancements, 11–12, 73–90

 

project configurations, adding parameters, 123

 

Parameterizing the Execute SQL Task in SSIS link

 

 

 

project deployment model, 117

 

catalog, project placement in, 128

 

features, 118–119

 

workflow, 119–122

 

project parameters, 123

 

prefix, 125

 

Project.params file, 98

 

projects

 

environment references, 133–135

 

environments, connecting, 132

 

importing, 121

 

previous versions, restoring, 131

 

validating, 126, 135

 

property search, 81–82

 

provisioning, security and role separation, 72

 

publication process, 191–192

 

Q

 

qualifier characters in strings, 105

 

queries

 

execution plan, 52

 

forcing columnstore index use, 53–54

 

performance, 41, 44–45

 

small look-up, 48

 

star join pattern, 47

 

tuning, 41. See also columnstore indexes

 

query modes, 214

 

query processing

 

batch-mode, 45

 

in columnstore indexes, 47

 

enhancements in, 6

 

R

 

ragged-right delimited files, 105

 

raster data, 87

 

RBS (Remote BLOB Store), 75

 

RDLX files, 232, 237

 

RDS (reference data service), 150

 

subscribing to, 167–168

 

Read And Process permissions, 211

 

reader/writer contention, eliminating, 32

 

Read permissions, 210

 

records

 

categorization, 162

 

confidence score, 150, 162, 169

 

filtering, 66–67

 

matching results, 165

 

matching rules, 158–159

 

survivorship results, 165

 

Recovery Advisor visual timeline, 6

 

recovery times, 5

 

reference data, 150

 

for composite domains, 152

 

reference data service (RDS), 150, 167–168

 

ReferenceType property, 102

 

refid attribute, 139

 

regulatory compliance, 62–67

 

relational engine, columnstore indexes, 6

 

relationships, table

 

adding, 206

 

in PowerPivot for Excel, 221

 

viewing, 205

 

relative environment references, 133–135

 

Remote BLOB Store (RBS), 75

 

renderers, 229–230

 

REPLACENULL function, 116

 

Report Builder, 232

 

Report Designer, 232

 

Reporting Services, 229–250

 

data alerts, 246–250

 

Power View, 232–246

 

renderers, 229–230

 

scalability, 231

 

SharePoint integrated mode, 229, 232

 

SharePoint shared service architecture, 230–232

 

SharePoint support, 230–231

 

Reporting Services Configuration Manager, 231

 

Reporting Services Shared Data Source (RSDS) files,

233

 

reports. See also Reporting Services

 

authentication for, 231

 

data alerts, 246–250

 

design environment, 234–237

 

DOCX files, 230

 

fields, 235–236

 

filtering data, 243–245

 

RDLX files, 232, 237

 

reporting properties, setting, 220

 

sharing, 237

 

table and column properties, 212–213

 

tables, 235

 

views, 234, 241–242

 

XLSX files, 229–230

 

report server databases, 231

 

Required parameter property, 123

 

resilience, audit policies and, 62–65

 

Resolve References editor, 108–109

 

Resource Governor, 8

 

Resource Governor

 

 

 

resource_monitor_ring_buffer_record event, 90

 

resource pool affinity, 8

 

resource pools

 

number, 8

 

schedules, 8

 

Resource Usage events, 215

 

Restore Invalid Column References editor, 108

 

restores, visual timeline, 6, 39–40

 

role-based security, 59

 

role separation during provisioning, 72

 

roles, user-defined, 60–61

 

role swapping, 26

 

rolling upgrades, 40

 

Row Count transformation, 106–107

 

row delimiters, 105

 

row groups, 45

 

Row Identifier property, 212, 213

 

row-level filters, 211

 

row store storage model, 42–43

 

row versioning, 32

 

RSDS (Reporting Services Shared Data Source) files,

233

 

RSExec role, 232

 

RsReportDesigner.config file, 230

 

RsReportServer.config file, 230

 

alerting configuration settings, 250

 

runtime, constructing variable values at, 101

 

S

 

scalability

 

new features, 6–7

 

Windows Server 2008 R2 support, 7

 

scatter charts in Power View, 239

 

schema rowsets, 216

 

schemas

 

default, 58

 

management, 58

 

SCONFIG, 37

 

scripting engine (SSIS), 99

 

search

 

across data formats, 73–74, 76

 

Customizable Near operator, 82

 

full-text, 81–82

 

of structured and unstructured data, 11

 

property, 81–82

 

semantic, 11

 

statistical semantic, 82–85

 

SEARCH PROPERTY LIST, permissions on, 72

 

secondary replicas

 

backups on, 33–34

 

connection modes, 27, 33

 

moving data to, 27

 

multiple, 23, 25, 25–26

 

read-only access, 33

 

SQL Server 2012 support for, 4

 

synchronous-commit availability mode, 26

 

securables, permissions on, 60

 

security

 

Active Directory integration, 71–72

 

auditing improvements, 62–67

 

of Contained Database Authentication, 70–71

 

cryptography, 71

 

database authentication improvements, 67–71

 

database roles, 210–211

 

manageability improvements, 58–61

 

new features, 10–11

 

during provisioning, 72

 

SharePoint integration, 71–72

 

of tabular models, 210

 

Windows Server Core, 36

 

security manageability, 58–61

 

default schema for groups, 58–59

 

user-defined server roles, 60–62

 

segments of data, 45

 

Select New Scope dialog box, 115

 

self-service options, 229

 

data alerts, 246–250

 

Power View, 232–246

 

semantickeyphrasetable function, 85

 

semantic language statistics database

 

installing, 84

 

semantic search, 11, 82–85

 

configuring, 83–84

 

key phrases, finding, 85

 

similar or related documents, finding, 85

 

semanticsimilaritydetailstable function, 85

 

semanticsimilaritytable function, 85

 

Sensitive parameter property, 123

 

separation of duties, 59–60

 

Server Audit Specification objects, 62

 

server-based logging, 137

 

Server + Client Access License, 15

 

Server Core. See Windows Server Core

 

Server Message Block (SMB), 39

 

server roles

 

creating, 60–61

 

fixed, 60

 

resource_monitor_ring_buffer_record event

 

 

 

server roles (continued)

 

membership in other server roles, 61

 

sysadmin fixed, 72

 

user-defined, 60–62

 

servers

 

Analysis Services modes, 199–201

 

configuration, 37

 

defaults, configuring, 126

 

managing, 215–216

 

server health rules, 226–227

 

shutdown on audit, 62–63

 

SHA-2, 71

 

shared data sources, for Power View, 233

 

shared service applications, 231

 

configuration, 231–232

 

shared storage, 26

 

SharePoint

 

alerting configuration settings, 250

 

backup and recovery processes, 231

 

Create Alert permission, 247

 

MDS integration, 197

 

PowerPivot for SharePoint, 224–227

 

Reporting Services feature support, 230–231

 

security modules, 71

 

service application configuration, 231–232

 

Shared Service Application pool, 231

 

shared service applications, 229

 

shared service architecture, 230–232

 

SharePoint Central Administration, 231

 

shortcut query files, 194

 

side-by-side migrations, 19–20

 

SIDs, per-service, 72

 

slicers, 243

 

sliding-window scenarios, 7

 

snapshot isolation transaction level, 32

 

snippet picket tooltip, 7

 

snippets, inserting, 7

 

software requirements, 16–17

 

Solution Explorer

 

Connections Manager folder, 98

 

packages, sorting by name, 100

 

Project.params file, 98

 

Sort By Column dialog box, 219

 

source data

 

assessing, 156

 

cleansing, 161–164

 

mapping to domains, 154–155, 158, 161–162,

164

 

matching, 164–166

 

for Power View, 233

 

spatial data, 86–90

 

circular arc segments, 87–89

 

index improvements, 89

 

leveraging, 86

 

performance improvements, 90

 

types, 86–87

 

types improvements, 89

 

sp_audit_write procedure, 63

 

Speller, 162

 

enabling, 146

 

sp_establishId procedure, 66

 

spherical circular arcs, 87–89

 

sp_migrate_user_to_contained procedure, 70

 

SQLAgentUserRole, 232

 

SQL Azure and SQL Server 2012 integration, 9

 

SQL Azure Data Sync, 9

 

SQLCAT (SQL Server Customer Advisory Team), 96

 

SQL Server 2008 R2, 3, 4

 

PowerPivot, 42

 

security capabilities, 57

 

SQL Server 2012

 

32-bit and 64-bit editions, 17

 

Analysis Services, 199–217

 

beyond-relational enhancements, 73–90

 

BI Semantic Model (BISM) schema, 217

 

capabilities, 3

 

Cloud Ready Information Platform, 3

 

data and services ecosystem, 74–75

 

Data Quality Services, 141–173

 

editions, 12–15

 

hardware requirements, 16

 

hybrid solutions, 3

 

licensing, 15–16

 

Master Data Services, 175–198

 

migration, 19–20

 

new features and enhancements, 4–12

 

PowerPivot for Excel, 218–222

 

PowerPivot for SharePoint, 224–227

 

prerequisites for Server Core, 37–38

 

programmability enhancements, 73–90

 

Reporting Services, 229–250

 

security enhancements, 57–72

 

software requirements, 16–17

 

upgrades, 17–19

 

SQL Server Agent, configuring permissions, 231

 

SQL Server Analysis Services (SSAS), 199. See

also Analysis Services

 

SQL Server Audit, 57. See also auditing

 

SQL Server Audit

 

 

 

SQL Server Configuration Manager

 

FILESTREAM, enabling, 78

 

Startup Parameters tab, 10

 

SQL Server Customer Advisory Team (SQLCAT), 96

 

SQL Server Database Engine

 

Deploy Database To SQL Azure wizard, 9

 

scalability and performance enhancements, 6–7

 

SQL Server Data Tools (SSDT). See SSDT (SQL Server

Data Tools)

 

SQL Server Extended Events, 66, 90, 215

 

SQL Server Installation Wizard, 17

 

SQL Server instances

 

hosting of, 26

 

maximum number, 39

 

replica role, displaying, 31

 

specifying, 30

 

TCP/IP, enabling, 142

 

SQL Server Integration Services Expression

Language, 101

 

new functions, 116

 

SQL Server Integration Services Product Samples

link, 97

 

SQL Server Integration Services (SSIS), 93.

 

SQL Server Management Studio (SSMS). See SSMS

(SQL Server Management Studio)

 

SQL Trace audit functionality, 62

 

SSDT (SQL Server Data Tools), 93, 201–202, 213

 

building packages, 119–120

 

deployment process, 120

 

import process, 121

 

project conversion, 121–122

 

ssis_admin role, catalog permissions, 139

 

“SSIS Logging in Denali” (Thompson), 137

 

SSIS Toolbox, 95, 97–98

 

Common category, 97–98

 

Destination Assistant, 103–104

 

Expression Task, 101–102

 

Favorites category, 97–98

 

Source Assistant, 103–104

 

SSMS (SQL Server Management Studio)

 

audit, creating, 64–65

 

availability group configuration, 24

 

built-in reports, 137

 

columnstore indexes, creating, 50

 

Contained Database Authentication, enabling,

68–69

 

converting packages, 122

 

Create A New Availability Group Wizard, 28

 

environments, creating, 132

 

FileTable documents, viewing, 80–81

 

manageability features, 7

 

nontransactional access, enabling, 78–79

 

operations reports, 137

 

Recovery Advisor, 6

 

server defaults, configuring, 126

 

server roles, creating, 60–61

 

staging process, 181–183

 

collections and, 183

 

errors, viewing, 182

 

transaction logging during, 183

 

Standard edition of SQL Server 2012, 13–14

 

startup

 

management, 10

 

times, 5

 

statistical semantic search, 82–85

 

stg.name_Consolidated table, 181

 

stg.name_Leaf table, 181, 197

 

stg.name_Relationship table, 181

 

stg.udp_name_Consolidated procedure, 182

 

stg.udp_name_Leaf procedure, 182

 

stg.udp_name_Relationship procedure, 182

 

storage

 

column store, 42. See also columnstore indexes

 

management, 73

 

row store, 42–43

 

shared vs. nonshared, 26

 

stored procedures, 182

 

stretch clustering, 34

 

string data types

 

filtering, 244

 

language, 146

 

spelling check, 146

 

storage, 215

 

structured data, 11, 73

 

management, 74

 

Structure_Type columns, 216

 

subnets, geographically dispersed, 5

 

survivorship results, 165

 

synchronization, 31

 

synchronous-commit availability mode, 26, 27

 

synchronous database mirroring, 22

 

synchronous data movement, 24

 

synchronous secondaries, 4

 

synonyms, 146–149, 156, 163, 173

 

syntax error checking, 146

 

sysadmin fixed server role, 72

 

Sysprep, 17

 

sys.principal table, 59

 

system variables, accessing, 102

 

systemwide outages, 62

 

database authentication issues, 68

 

SQL Server Configuration Manager

 

 

 

T

 

Table Detail Position property, 213

 

Table Import Wizard, 203, 204

 

tables

 

calculated columns, 207

 

columnstore indexes for, 48

 

maintenance, 47

 

partitions, 209–210

 

pivot tables, 218–222, 224–227

 

relationships, 205–206. See also relationships,

table

 

reporting properties, 212–213

 

tabular models, 203–214

 

access to, 210–211

 

analyzing in Excel, 211–212

 

calculated columns, 207

 

Conceptual Schema Definition Language

extensions, 217

 

Conceptual Schema Definition Language,

retrieving, 216

 

database roles, 210–211

 

DAX, 222–224

 

deployment, 214, 222

 

DirectQuery mode, 213

 

hierarchies, 208–209

 

importing data, 204

 

key performance indicators, 208

 

measures, 207–208

 

model designer, 205–206

 

partitions, 209–210

 

perspectives, 209

 

query modes, 214

 

relationships, 205–206

 

reporting properties, 212–213

 

schema rowsets, 216

 

Table Import Wizard, 204

 

tutorial, 203

 

workspace database, 203–204

 

tabular server mode, 199–201

 

task properties, setting directly, 125

 

TCP/IP, enabling on SQL Server instance, 142

 

TDE (Transparent Data Encryption), 57

 

TempDB database, local storage, 5, 34

 

term-based relations, 149. See also domains; domain

values

 

testing tabular models, 211–212

 

Thomson, Jamie, 137

 

thread pools, 216

 

tile containers, 243–244

 

tiles in Power View, 239

 

TOKENCOUNT function, 116

 

TOKEN function, 116

 

transaction logging

 

on secondaries, 33–34

 

during staging process, 183

 

transactions, 198

 

annotating, 198

 

reviewing, 191

 

Transact-SQL

 

audit, creating, 65

 

audit log, stack frame information, 63

 

code snippet template, 8

 

columnstore indexes, creating, 51–52

 

Contained Database Authentication, enabling, 69

 

debugger, 7

 

FILESTREAM file groups and database files,

configuring, 79

 

nontransactional access, enabling, 79

 

server roles, creating, 61

 

user-defined audit event, 65–66

 

transformations, 106–107

 

Transparent Data Encryption (TDE), 57

 

trickle-feed packages, 112

 

trustworthy computing initiative, 10

 

TXT files, extracting data, 105

 

Type columns, 216

 

U

 

Unified Dimensional Model (UDM) schema, 217

 

UNION ALL query, 49

 

unstructured data, 11, 73

 

ecosystem, 74–75

 

management, 74

 

raster data, 87

 

semantic searches, 11

 

storage on file system, 74, 76–77

 

updates, appending new data, 48

 

UpdateSchedule event, 250

 

Upgrade Database Wizard, 176

 

upgrades, 17–19

 

control of process, 18, 20

 

high-level in-place strategy, 18–19

 

rolling, 40

 

version support, 17

 

uptime. See also availability

 

maximizing, 4–6

 

Windows Server Core deployment and, 6

 

uptime

 

 

 

user accounts, default schema for, 58–59

 

user databases

 

user authentication into, 9

 

user information in, 68

 

user-defined audit events, 63, 65–66

 

user-defined roles, 60–61

 

server roles, 60–62

 

users

 

authentication, 9

 

migrating, 70

 

V

 

validation, 135

 

varbinary max columns, 74

 

variables

 

adding to packages, 115

 

dynamic values, assigning, 101–102

 

in Expression Builder, 102

 

scope assignment, 115

 

vector data types, 86–87

 

vendor-independent APIs, 75

 

vertical isolation of machine resources, 8

 

VertiPag compression algorithm, 44

 

VertiPaq mode, 199, 215

 

view filters, 244–245

 

virtual accounts, 72

 

virtual network names of availability groups, 28

 

visualization filters, 245

 

Visual Studio 2010 Tools for Office Runtime, 218

 

Visual Studio, Integration Services environment, 93

 

Visual Studio Tools for Applications (VSTA) 3.0, 99

 

vulnerabilities reported to NIST, 10

 

W

 

Web edition of SQL Server 2012, 15

 

Windows 32 Directory Hierarchy file system, 74

 

Windows Authentication, 71

 

Windows Azure Marketplace, 167–168

 

Windows file namespace support, 11

 

Windows groups, default schema for, 58–59

 

Windows PowerShell

 

MDS administration, 198

 

PowerPivot for SharePoint configuration cmdlets,

224

 

Windows Server 2008 R2, scalability, 7

 

Windows Server Core

 

prerequisites for, 37–38

 

SCONFIG, 37

 

SQL Server 2012 installation on, 5, 17, 36–38

 

SQL Server feature support, 38

 

Windows Server Failover Cluster (WSFC), 5, 26

 

deploying, 24

 

Word 2010 renderer, 230

 

word breakers and stemmers, 82

 

workbooks, Excel

 

caching, 225

 

data refreshes, 227

 

workflow, adding Expression Tasks, 101

 

workloads, read-based, 47

 

worksheets, Excel

 

creating data in, 192

 

data quality matching, 195–196

 

filtering data from MDS, 188–189

 

loading data from MDS, 188–190

 

publishing data to MDS, 191–192

 

refreshing data, 190

 

workspace databases, 203–204

 

backup settings, 203

 

hosts, 203

 

names, 203

 

properties, 203

 

query mode, 214

 

retention settings, 203

 

WSFC (Windows Server Failover Cluster), 5, 24, 26

 

X

 

XLSX files, 229–230

 

XML files

 

annotations, 139

 

attributes, 139

 

refid attribute, 139

 

XML For Analysis (XMLA) create object scripts, 215

 

Z

 

zero-data-loss protection, 23

 

user accounts, default schema for

 

 

 

About the Authors

 

ROSS MISTRY is a best-selling author, public speaker, technology evangelist,

community champion, principal enterprise architect at Microsoft, and former

SQL Server MVP.

 

Ross has been a trusted advisor and consultant for many C-level executives

and has been responsible for successfully creating technology roadmaps,

including

the design and implementation of complex technology solutions

for some of the largest companies in the world. He has taken on the lead

architect role for many Fortune 500 organizations, including Network

Appliance,

McAfee, The Sharper Image, CIBC, Wells Fargo, and Intel. He specializes in data

platform,

business productivity, unified communications, core infrastructure, and private cloud.

 

Ross is an active participant in the technology community—specifically the Silicon Valley

community.

He co-manages the SQL Server Twitter account and frequently speaks at technology

conferences around the world such as SQL Pass Community Summit, SQL Connections, and

SQL Bits. He is a series author and has written many whitepapers and articles for Microsoft,

SQL Server

Magazine, and Techtarget.com. Ross’ latest books include Windows Server 2008 R2

Unleashed

(Sams, 2010), and the forthcoming SQL Server 2012 Management and Administration

(2nd Edition) (Sams, 2012).

 

You can follow him on Twitter at @RossMistry or contact him at http://www.rossmistry.com.

 

STACIA MISNER is a consultant, educator, mentor, and author specializing

in Business Intelligence solutions since 1999. During that time, she has

authored

or co-authored multiple books about BI. Stacia provides consulting

and custom education services through Data Inspirations and speaks

frequently

at conferences serving the SQL Server community. She writes

about her experiences with BI at blog.datainspirations.com, and tweets as

@StaciaMisner.